Squashfs submitted for the mainline

If you're doing the compressing, you ought to be able to guarantee a memory limit for decompression. Just try decompressing while you still have the original around, and tweak the compression parameters if you run out of memory. That's how MPEG encoders work - consumer products that do mpeg stream decompression have a defined amount of memory (2Mbits, if I remember correctly).

Admittedly that might be considered an expensive solution, but it proves that a solution exists.