My post-visit research at the NMRW came up with the following about this site’s demise in 1844:
The Rev. T.G. Mortimore remonstrated with the destroyer [of the stone], and with the view of inducing him to desist reminded him of the old saying that ill-luck befell those who destroyed the Druid’s altars. Some years afterwards the vandal admitted that the house he had built of the stones had not brought him good luck!
At least we got some good folklore out of this site.