What a shame :(
I had a nice aerial view of this site during my horrible descent from Foel Fynyddau a couple of years back. Looking elsewhere, it seems your experience isn't unique, unfortunately. Others have reported furious encounters with the landowners and allegations that they were disturbing the horses and leaving rubbish. Sounds like permission is needed but it's not somewhere that you would be welcome even then.
Hope it hasn't spoiled your South Wales trip too much.