Speaking as someone who lives not too far away from this stone I'm really glad that they are leaving it in it's setting rather than digging it out and sticking it in some museum. If the stone was removed it would lose all of its context. Aren't there enough dusty old rocks in our museums?
Don't forget that they've laser scanned the stone and so can create extremely accurate copies both physically and electronically.
As for pill boxes, what's your problem with them? aren't examples of our wartime coastal defences worthy of preservation or are the two world wars not part of our heritage?