Hm yes which brings up an interesting point.
A stone drift may have been seen as Sacred in the Neolithic but once its emptied and the stones reused elsewhere there is nothing for Modern man to latch onto as a Sacred place.
There are plenty of near empty stone drifts around.
Cow Down, Beckhampton valley etc.
No one goes there now but they would if there was just a single stone standing!