I visited this site last December on a gloriously sunny winter's day. I parken on the lane next to a phone box near the entrance to Perth-Y-Pia. The walk up to the hillfort is very, very steep but the views you are rewarded with make the effort worthwhile. I sat for quite a while surveying the surrounding landscape - lovely.
An alternative route from that mentioned by Gladman - visited 27.3.2010 walking up from the pleasant town of Crickhowell, via Great Oak and The Wern. Crickhowell is served by a decent bus service running between Abergavenny and Brecon (and Cardiff).
The approach to the fort is a pretty stiff climb, but as already mentioned is well worth it for the views over the Usk valley, with the Sugar Loaf an easily identifiable landmark to the south-east. The ramparts top a near-vertical rock-band and are themselves topped with the remains of a stone wall/bank (not much earth to create a bank and ditch from after all). Although it boasts a ridiculously large number of walker's shelters, most of which appear to have been robbed from the rampart, this wall is still visible all the way around the summit of the hill, with an entrance on the east side. A wide ridge links Table Mountain to the shoulder of Pen Cerrig-calch, a flat-topped 700m summit boasting some decent cairns and extensive views.
Overlooking the busy town of Crickhowell (Crug Hywel in the native Welsh) which eventually superseded it, a visit to the small hillfort of Crug Hywel is memorable, to say the least. Set upon what is known locally as Table Mountain - for obvious reasons, since the site looks as if some alien spaceship pilot had a few too many 'Romulan ales' and sliced a bit off - at 1,480ft the hillfort requires a very steep climb to reach. It is well worth it, however, if only for the views of the wonderful Usk valley alone. I'd recommend an approach from the east, via the outdoor centre at Perth-y-pia, since this is the route myself and the Mam Cymru took.
Despite being in such an exposed position the defences have survived reasonably well, all things considered, the state of preservation probably assisted by the fact that it's such bloody hard work just getting here, let alone pinching stone and carrying it back down again… I'll leave the technical stuff to Coflein:-
The fort is tear-drop shape in plan, tapering towards the northwest end, and encloses an area roughly 100 metres northwest-southeast and 50 metres northeast-southwest. The interior slopes towards the southeast end. The fort is entered through the northeast side, through a gap in the rampart and inner defensive wall roughly 8 metres wide. The inner defensive wall has collapsed to 5 metres wide and approximately 0.3 metres tall. The rampart wall is better preserved, standing to a maximum of 2 metres high, although it has become flattened on the south and southwest sides. The vertical distance between the bottom of the rampart and the top of the inner wall is roughly 10 metres. Two hut platforms and a hut circle are located against the inside face of the inner wall on the northwest side of the fort. There is also evidence of a round building to the south of the entrance.
Note that after the initial rigours of the ascent the 2,300ft summit of Pen Cerrig-Calch is a relatively easy onward walk to the approx NW and provides great panoramic views of, well, everything. If you fancy it go prepared, however. This is Wales, after all. Oh, one more thing. Don't forget that the Gwernvale chambered cairn - well the orthostats at least, since there's no 'cairn' - is a 'no-walk' bonus beside the A40 just outside of Crickhowell itself.