You're kinda both right (you and FW).

A lot depends on the support (the surface to which a pigment or paint is applied such as canvas, paper, silk etc ). Chalk on a blackboard is going to wash off fairly easily because the blackboard's relatively smooth. Chalk on a sarsen stone is going to be more difficult to remove because it's been deposited in the micro cavities of a much rougher surface.

Fortunately chalk has little or no binder (an oil or animal protein etc which is used to bond the pigment together to form a paint) so solvents would not generally be required.