I'd second the recommendation for "Datapanik...". As Stevo says it has their first two (excellent) LPs in full, plus all the early singles and an extra disc featuring various offshoots and precursors (such David Thomas and Peter Laughner's brilliant pre-Ubu band Rocket From the Tombs). the second half of disc 2 and disc 3 - featuring their third, fourth and fifth Lps - isn't much cop IMO (too self-consciously arty) but for £13 you can't really go wrong.

If you like this set I'd also recommend their two late 80s albums "The Tenement Year" and "Cloudland", plus last year's storming "Lady from Shanghai". The 90s / noughites albums have their supporters two, but I can't comment on those as I haven't heard them.