There is good evidence that the banks of Segsbury Castle were originally faced with sarsens. The following quotation is taken from T Hearne’s Letter containing an account of some Antiquities between Windsor and Oxford (1725) -
...Sackborough Castle (by which name they call certain strange works, or an old camp) on the South-East side of Wantage in Berks, about two miles from it. It is in a manner round, tho’ I cannot call it a perfect round. I take it, however, to be Danish. Within the Bank that lies on the Inside of this Camp, or as they vulgarly call it, Castle, they dig vast red stones, being a red flint, some of which a cart will hardly draw. They have dug up a great many loads of them, and with many of them they build. They are placed in the banks of the dike or trench in form of a wall. ... When first I walk’d in those parts, I inquired, where it was they could either dig or meet with such stones? It was answer’d that the like occur’d upon Lambourn Downs. Upon which I concluded, and afterwards found, that they grow upon those downs.
From White Horse Hill and surrounding country, by L V Grinsell