Phil Quinn (Third Stone 26) mentions that fairies used to live here at Cadbury. But they couldn't stand the noise from the new church bells - they buried their gold and left (perhaps too heavy to take with them?).
The folklore mentioned by Purejoy below is included by Ruth Tongue in her 1965 'Somerset Folklore' - she also says that the local people (unsurprisingly) called the camp 'Camelot'.
Another ‘cadbury’ and another Arthurian legend! There is rumoured to be a cavern in this hillfort where King Arthur and his knights lie sleeping, waiting to be called out to help England face her enemies. Black magic and witchcraft were said to have been practiced here, plus dancing around a thorn tree on May Eve and Midsummer Eve