“The person who pulled down this cromlech is said to have brought a number of misfortunes about him in consequence; thus his cattle died and crops failed, which left a warning impression on the minds of his neighbours.”
From an 1861 source quoted in Grinsell’s ‘Folklore of Prehistoric Sites in Britain’.