See things like that tend to Stick in the mind.
Silbury was white, the Stonehenge stones came from the Marlborough downs.
It makes more sense to me to look for Absense of stones, not search in a large stonedrift.
Dragging or rowing large 60 ton+ sarsens over a field of smaller stones would probably make Gordon P break down and weep at the very thought.
As Gordon said "I row the stones this way because I'm lazy"
NeoMan probably thought about the easiest way also.
Its such a shame that most of Salisbury plain is still under Army rule as there are places there that need looking at again as a source of a sarsen drift.