It's made a lot easier if you use two boats, one for the stone and another with the oarsmen to tow it.
That removes loads of the objections at a stroke. It's far safer. The stone barge could be designed in just the right way. It could be covered over, so it was virtually unsinkable. It could be hauled up rivers without the need to unload and transfer the stones.