Personally I couldn't give two hoots that the graffiti in question is a swastika.
What irritates me is the fact that when someone chalks on a stone or leaves anything visible at a site, they chose to push their interpretation of how a site should be onto me. Now I feely admit I am doing exactly the same by returning the sites I visit to a non-graffiti-d & non offering strewn state.
I feel no qualms however, I know the sites are better not having chalk on the stones, they are better when you don't have to step over some discarded tealights, they are better when you don't have to breath in someone elses incense, when it isn't full of rotting vegetation.
And for once, the law is on my side. It is illegal to graffiti, it is illegal to litter.
Also (whilst I'm here) yes the stones may have been decorated, but with what? Is it right to decorate them with anything just because they were once possibly decorated with glyphs unknown?