Montgomery has a castle, which is free to get into:
http://www.castles-of-britain.com/castlexw.htm

as well as a hillfort:
http://www.welshpool.org/welshpool1/ffridd.htm

and don't forget Offa's Dyke is all over the place, as is Wantyn's Dyke.

and of course (all of this stuff is 10-15 miles east of Newtown on the A489), there is always Mitchells Fold stone circle:
http://www.virtual-shropshire.co.uk/cards/1035738447.html

you can always have a walk on the Severn river path as it goes thru Newtown