Granite's quite light compared with, say, Basalt. As a result you weigh less in Cornwall than Iceland as the the amount of Earth excerting a gravitational pull is less over less dense rock.
Both have a pretty solid crystalline structure so are still fairly dense, wheras sandstone has more voids in it, ie is more porous (though not necessarily permeable). Because more of it is air it's lighter.
Most rocks are still pretty darned heavy if they're big enough though...
(That's the first time I've used anything from my Geology degree in ages!)