Lots of Roman things were found there (lead, a mirror, pottery, glass), which is interesting isn't it?
www.euppublishing.com/doi/pdfplus/10.3366/gas.1982.9.9.60
It says that at first the builders and the Romans must have been on good terms - maybe that the Romans let them build there but the broch owners had to keep the locals in check to get preferential treatment and products.

(it's an old article though)