That sounds fun!
You certainly get a good view of the tor from the M5, and that's low lying on the Somerset levels. So if you're raised up although further away - hey why not, it's such a distinctive shape isn't it.
The second picture on this page shows Brean Down and Steep Holm - with South Wales behind. That'd be the right direction for where you were looking from.