On further reading, it appears that the damage was in fact done and the stones removed as a result of the presence of the gulag.

Reading Anne Applebaum's terrifying history of the camps, it is apparent that the Solovetsky islands were the first, the biggest and one of the most enduring of the locations for forced labour. Many, many people worked and died here - adding a poignant dimension to the description of the labyrinths by archeologist Nikolai Vinogradov (himself an inmate) as representations of 'cities of the dead'. Much of the work involved digging, clearing, road making and building.