As always the argument falls on several sides on this one, there is no answer to the point that 'freedom is the prerogative of all' except that in a small country of 60 million people we have to have some rules....
so asking the question 'why is off-roading so destructive' the Friends of the Peak District have answered it...

http://www.friendsofthepeak.org.uk/Campaigns/Take_back_the_tracks/Our_answers_to_your_questions!/

It falls beautifully in the centre of the argument....