It isn't the chalk but the symbolism; when most people see a sawstika they associate it with one thing only, despite its historical provenance.

Yeah, the chalk is a problem - there are all kinds of symbols/patterns that I find attractive or interesting, but it just is not appropriate or acceptable to draw them all over our commonly owned heritage.

By the way - are there any guidlines for removing chalk graffitti (I know this may have been covered here before, apoplogies if that is the case).