Well I certainly agree that if its an altar its special so is quite likely to have been brought in specially. "Right lads, the temple is finished, we just need an altar - I know, lets use this sarsen offcut" - doesn't sound right. I bet not many altars anywhere are the same stone as the buildings they're in.
So actually, I think we've just proved that contrary to Mr Pitts' opinion, if the stone is different that makes it MORE likely to have come from Stonehenge! :>)