Follow the signs for Brinkburn Priory (E.H. site) off the B6344.
The O/S map shows the fort to be on the promontory above the priory.
The road leading from the car park to the Priory cuts across the northern steep sided slope. This would have indeed been a very steep incline before the road was built.
As for spotting any evidence of remaining defences I couldn’t see any.
Where the O/S map shows earthworks is an undulating field.
Difficult to tell (for me anyway) what was natural and what was man made.
The Priory is worth visiting but don’t go out of your way to look for anything older.
The Folklore Society's volume 2 of the Denham Tracts (1895) has a lengthy description of a legend connected with this site. The priory and fort were in a tight loop of the river and so protected on three sides by the water and steep slopes.
The story begins: "Under a grassy swell, which a stranger may know by its being surrounded with a wooden railing, on the outside of Brinkburn Priory, tradition affirms there is a subterraneous passage, of which the entrance remains as yet a secret, leading to an apartment to which access is in like manner denied;[...] it is asserted that a hunter who had in some way offended one of the priors was along with his hounds, by the aid of enchantment, condemned to perpetual slumber in that mysterious abode."
To try and summarise the rest, it seems that only once has anyone seen this underground mysteriousness. It was a shepherd and his dog - he noticed a door in the ground and walked down a dark flight of steps. Pushing through a door at the bottom there was a brightly lit room, and inside a sleeping hunter and another man, and lots of snoozing hounds. On a table were a horn and a sword, but when the shepherd picked these up, everyone started waking up. He ran to the rapidly closing door (a bit like Indiana Jones) and just made it outside, with 'a terrible voice assailing his ears pouring maledictions on him for his temerity.' His dog wasn't quite so lucky and got nipped in half by the door slamming shut. Nasty.
The themes are a bit like the story connected with Sewingshield Crags - and various other places.
Further on in the Tracts Denham mentions that "Mr. Wilson says the fairies lie buried at Brinkburn. This mortality, unheard of elsewhere, must have been attributable to the potency of the bells." The Bell Pool is a deep part of the river below, and you can read how the bells from the priory were variously accidentally lost or deliberately placed in that pool, and how 'young swimmers of the neighbourhood' still dive for them.