Ooh! A chance to mention TC Lethbridge again!
In Boats and Boatsmen Lethbridge talks about the difference between fishing communities and farming communities. His comments were something like neither one can understand the other and neither would swap the way of life they were raised with. He was talking about 1950s Cornwall.
Fishing communities are dead now, but not so long ago they were very strong. They went and fished and let others get on with farming. They just traded fish for grain etc.
Obviously, due to transport/freshness issues sea fish was not available to people living well inland unless they were very rich. Seaside dwellers however would have had plenty of it in their diet.
I don't agree with your statement (paraphrasing now) that fish eating tapered off after the Neolithic. As the inland population grew they couldn't eat as much fish as mentioned above, but the seaside folk certainly could. The number of seaside dwellers became a smaller percentage of a growing population, so some stats will falsely show that less fish was eaten.