I'd say go for the Tom Herman records, the 1st 3 lps
but just occurred to me that since this is cheap you migt as well go the whole hog and buy it.
http://www.amazon.co.uk/Datapanik-In-The-Year-Zero/dp/B000024N7V/ref=sr_1_1?ie=UTF8&qid=1382180411&sr=8-1&keywords=datapanik+in+the+year+zero
All the material from the first incarnation(s) of the group on a 4cd set including one live lp. It has all the Hearpen singles on the 1st cd alongside the Modern Dance lp.

I'd also look at buying these
http://www.amazon.co.uk/The-Shape-Things-Pere-Ubu/dp/B00009EBUE/ref=ntt_mus_ep_dpi_1
a live set from 1976 with several tracks in formative stages

http://www.amazon.co.uk/ONE-MAN-DRIVES-WHILEOTHER-SCREAMS/dp/B0001IW2UM/ref=ntt_mus_ep_dpi_22
I think this is mainly from '78

and the compilation of Rocket From The Tombs the band that both Pere Ubu and the Dead Boys emerged from
http://www.amazon.co.uk/Day-Earth-Met-Rocket-Tombs/dp/B005GKGBI4/ref=sr_1_1?s=music&ie=UTF8&qid=1382180710&sr=1-1&keywords=the+day+the+earth+met+rocket+from+the+tombs

There were some great live bootlegs going around of Pere Ubu from various times between 76 and probably the current day, but David Thomas has stopped them from being shared on torrent sites. & my hard drive with mine on crashed and I was unable to salvage anything. Major drag, there were some very good ones.

might also look into the later band's lps which I haven't to any great extent. I did like Cloudland, but haven't heard many of the others.

Pere ubu have an online shop here too http://ubuprojex.net/

Stevo