The damage was probably done during the establishment of the gulag - either institutional vandalism or for the purposes of plundering building materials. I think the reference to their removal is to do with the relatively small size and portabilty of individual stones rather than through natural weathering