... The hillfort was abandoned within a generation.
What fits beautifully were if the people resting in the barrows thereabouts were known, or family, of the hillfort builders. In that scenario, a barrow within a 'full time' living defensive site may be understood, perhaps. And the site, an early one and abandoned after very little time, may have been, due to the proliferation of "non kin" barrows for want of a better expression, very unattractive to others wishing to reuse the site after its original inhabitants had gone.

All just thoughts, and we could go on all day creating plausible scenarios, but I would certainly imagine the ancestors in those mounds had a strong influence upon whoever built or used that site.