And to throw in a cliche, absence of evidence is not evidence of absence. Especially given how long ago we're talking about.

An entire religious belief system can founded on feathered hats and big sticks. You don't need a building to have a religion.