Assuming there was a fragment of bluestone present, it was only a single fragment, not a large piece or even several shards. Who's to say this solitary fragment wasn't brought along to Silbury by a neolithic man, perhaps as a tribute or a totemic symbol - a personal attachment for him back to Stonehenge. I carry a stone in my pocket as a keepsake, maybe the neolithic artisans were in the habit of doing the same ? Perhaps the bluestone was immediatly identifiable as being connected with Stonehenge and carried a certain prestige when shown - a sort of 'Access all areas' backstage pass that allowed this individual to visit the pinnacle of Silbury. If it was sharp or unworked, perhaps it slipped out of his bag or 'pocket' and became lost underfoot, maybe it was ceremoniously buried as a libation or as a 'connector' between the two sites.
Either way, it would be marvellous to discover that a dance of bluestones once topped Silbury, a stone crone on a vast head of earth.
x W