Compress data, gain performance

Posted Feb 2, 2007 16:29 UTC (Fri) by walles (guest, #954)
Parent article: Embedded Linux: Using Compressed File Systems

It's obvious that compressing and decompressing data takes more CPU time than not doing it.

But since reading blocks from disk is *really* expensive, why wouldn't it be possible to actually *gain* performance from compressing data on disk? Assuming that fewer blocks would have to be read from disk this way?

Does anybody know about any numbers for how much it actually costs or how much it's possible to gain performance wise by compressing data on disk?

to post comments

Compress data, gain performance

Posted Feb 2, 2007 16:43 UTC (Fri) by daney (guest, #24551) [Link] (2 responses)

Like many things, the benefits of compression depend on the specific hardware in question.

* Slow CPU + very fast disk -> Compression might not be useful.

* Fast CPU + very slow disk -> Compression may be useful.

You really should test it on your specific configuration.

Compress data, gain performance

Posted Feb 5, 2007 14:30 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (1 responses)

But where's the line? Ignoring the outliers, I strongly suspect that the 97% of users running fairly close to off-the-shelf Intel/AMD boxes all fall into the same category, and I suspect that those running on other fairly modern off-the-shelf hardware also fall in the same boat. There's thousands of issues where it's possible that the choice may not be optimal for odd hardware, but it's far easier and more efficient to pick a standard choice than have every user check their system.

Compress data, gain performance

Posted Feb 9, 2007 11:54 UTC (Fri) by robbe (guest, #16131) [Link]

Remember that this article talks about "Embedded Linux", so your CPU is
typically puny in comparison with the CPUs you seem to think of.
Furthermore, the "disk" is often flash memory: slower than DRAM, but much
faster than a harddisk.

Benchmark results for various compressed vs. standard filesystems on a
laptop (i.e. slow mass media, fast CPU) was linked to in the article:
http://tree.celinuxforum.org/CelfPubWiki/SquashFsComparisons

My guess is that the dividing line lies between desktop/laptop and
embedded machines.

I suspect what is holding back compression as a standard fs feature in
desktop/laptop systems is the complexity of fragmentation-avoidance in
the write-often case, and that the really big data (images, films, music)
is often compressed anyway (and by means better than general purpose
algorithms can offer).

Compress data, gain performance

Posted Feb 6, 2007 12:44 UTC (Tue) by joern (guest, #22392) [Link]

Bandwidth != Latency

Reading blocks from disk is expensive because disks have a high latency. Their bandwidth is actually very good. If you want better performance, you have to cut down on the number of seeks, not save bandwidth. Compressing data usually causes more seeks, thereby making performance worse.

If you can come up with a new scheme to reduce the number of seeks by compressing data, the storage world would welcome you heartily.