Also take into account, you don't have to conciously imbibe psychedelics. Ergot mould can grow on crops, and some of the pigments and colours used in cave art are known to be poisonous, but the neurological by product of the poisoning can be hallucinations.
I'd like to think that Ergot infected crops would have been discarded..but i guess if the year was tough, or the crop poor or failing, then you'd have no choice other than to use the food or starve. I think i'm right in saying that cases of 'St Anthony's Fire' from the Middle Ages was attributed to ergot poisoning. For which another by-product can be blood posining and gangreen with prolonged exposure.

I used to take psychedelics many years ago as a kid, and remember one trip out in the country very vividly...i was fascinated by the wind blowing through a field of wheat, the visual hallucinations appeared to slow time down..leaving 'vapour trails' in the crop that were almost like waves on the ocean, or ripples in a pond. Also the geomtry of nature comes to the fore and is more apparent in altered states too.