It kinda happened in reverse for me. I grew up in Wroughton (just down the road from Avebury) so places like Avebury and Savernake Forest were all within biking distance for our 'gang' of pre-teen boys. I later went to art school in Swindon and I'm ashamed to say that, back in the 60s, places like Avebury and West Kennet Long Barrow were frequent venues for student parties. SLAP - <i>ouch</i> (but it was over forty years ago).
Not actually sure when the 'mystery' crept up on me. I spent a long time living aboard and the memory of the sweeping Wiltshire Downs, Avebury and Silbury was all that kept me going sometimes. As moss said, there's something about, "...the magic of the landscape round Avebury..." There <i>was</i> one 'moment' though. About 25 years ago I'd come to a crossroads in my life and couldn't decide which way to go. I went up onto the Ridgeway - can't remember exactly where but there's a crossroads and a signpost up there somewhere. I sat with my back against the signpost. Warm sunny day, completely alone except for the buzz of insects and a light breeze through the grass.
After an hour or so I knew which way to go. There's something very reassuring about these ancient places - perhaps those who believe in god find the same kind of reassurance in their places of worship.