This was to be the site of Christchurch priory. It was said that everytime stones were laid on the site, by morning they had moved from the hill to where the priory now sits, over 2 miles away.
A similar tales exists in the parish of Fordington in Dorchester, which involved the siting of the parish church. The twist is that an ancient stone returned to the site after having been removed.
Both these stories appear in Peter Knight’s “Ancient Stones of Dorset”