This is s cracking little site I visited last summer.
Follow the lane north east through Bettws Newydd up Clytha Hill. You can pak right next to the edge of the hillfort and access is very easy through a gate. The scenery is lovely and a 10 minute walk will take you right around the hillfort. Despite being so easy to get to, the site has a fairly 'remote' feel to it. This is probably why I stumbled across a 'courting couple' when I visited - sorry for the interuption!!!!
In a Welsh manuscript from 1600* there is a list of giants and their abodes. It includes
"Clidda Gawr in the parish of Bettws Newydd, and his abode in the place called Cloddeu Caer Clidda, and that land today is called Tir Clidda in the parish of Llanarth."
Coed y Bwnydd is near/on Clytha Hill, so it seems Coed y Bwnydd may well be Clidda's castle. It's now a wooded nature reserve looked after by Monmouthshire council. (There is a Clytha Castle to the north, but that's far too new for a giant).
*Owen, H., 'Peniarth Ms. 118, fos. 829-837' Y Cymmrodor, XXVII, (1917) pp.115-52. Text and translation.
cited here on the Arthurian Gwent page.