Clearly what is needed is a scheme to extract that line-number information and compress it into a (much) smaller representation. This should be perfectly possible, because within one object file line numbers increment fairly smoothly and addresses do too, so the mapping between the two within each .o should be representable with a much smaller number of bits than are required for full-blown addresses.
(Ideally one would gzip the result as well, but this requires ungzipping to be working while oopsing or panicking, which may be too much to hope for.)