The Crags are sandstone and in some parts rise as cliffs to the height of one and two hundred feet. There are great rents in these rocks and tumbled down masses, which here and there form caverns. One of these, Wedderburn's Cave, was examined; another bears the name of the Priest's Cave.
In times of disturbance and insecurity, when the borders, especially, were subject to plundering and slaughter, such caverns may have been used as hiding places, and have taken their name from the persons who found refuge in them. Some persecuted minister of religion may have found temporary safety in the Priest's Cave, and possibly a freebooting Wedderburn may have escaped death by concealment in the dark recess which bears his name.
From the History of the Berwickshire Naturalists' Club; the anniversary address delivered on 27th September 1861 (and written up by George Tate).
Wedderburn Hole is at NU077099. Macartney's Cave is at NU060093. Alison's photo on Flickr makes the former look a bit of a squeeze. But the latter looks a bit more homely.
Castle Hill is an Iron Age fort, later reused in medieval times. You can see the remains of a tower on top, and this has a legend attached which is more often associated with the siting of churches (but without the fake boar - it's usually the devil or some other supernatural interferer. Could this be a 'rationalised' version of a previous tale?):
A lord of Callaley in the days of yore commenced erecting a castle on this hill; his lady preferred a low sheltered situation in the vale. She remonstrated; but her lord was wilful, and the building continued to progress. What she could not attain by persuasion she sought to achieve by stratagem, and availed herself of the superstitious opinions and feelings of the age. One of her servants who was devoted to her interests entered into her scheme; he was dressed up like a boar, and nightly he ascended the hill and pulled down all that had been built during the day. It was soon whispered that the spiritual powers were opposed to the erection of a castle on the hill; the lord himself became alarmed, and he sent some of his retainers to watch the building during the night, and discover the cause of the destruction. Under the influence of the superstitions of the times, these retainers magnified appearances, and when the boar issued from the wood and commenced overthrowing the work of the day, they beheld a monstrous animal of enormous power. Their terror was complete when the boar, standing among the overturned stones, cried out in a loud voice--
"Callaly Castle built on the height,
Up in the day and down in the night;
Builded down in the Shepherd's Shaw,
It shall stand for aye and never fa'.
They immediately fled and informed the lord of the supernatural visitation; and regarding the rhymes as an expression of the will of heaven, he abandoned the work, and in accordance with the wish of his lady built his castle low down in the vale, where the modern mansion now stands. --George Tate, F.G.S., in Alnwick Mercury, August 1, 1862.
From the Denham Tracts, which also has a bit on weather forecasting using the site:
When the "Callaly pot is boiling" it indicates bad weather. A mist in a ferment rises straight up from the ravine between the Castle Hill and Lorbottle Moor, and clings to the top of the hill. This is a sure sign of rain, both as seen from Biddleston on the west and Shawdon on the east. The "Callaly pot" was boiled by the Clavering owners, who were a Catholic family, to provide a dinner for the poor people who on Sunday and holidays attended the services at the chapel attached to the mansion.
In the late 19th century (according to the Magic record) several Bronze Age stone coffins were discovered during quarrying on the north side of the hill. On the south side there are quite a few round cairns. One is near Macartney's Cave ('In one of the huge fantastic rocks among the heather is Macartney's Cave, a little oratory hewn out of the sandstone by a former chaplain of Callaly Castle'*), and at least five are near 'Hob's Nick' - a deep fissure in the rock.
*taken from 'Northumbria' (1920 - no author mentioned?) online at http://www.oldandsold.com/articles32n/northumbria-32.shtml
The Denham Tracts say of the waterfalls in Callaly Crags close by:
The pot-holes... are Robin Goodfellow's or Hob-Thrush's Mills, wherein he grinds his visionary grain. The mills are set going by spates, which bring down stones that rattle in the pot holes, like the grinding gear of a mill set in motion.