In a book on the Breckness Estate there is a quote from 1830 of “a mound of stones on the sea beach, called the Castle of Sneusgarth” which means the castle is either this mound or (more likely) the broch [unless there was a now lost mound at the taing]. The castle is a moveable feast whichever way you read things.