| |||||||||||||
Google releases a better compression algorithmGoogle releases a better compression algorithmPosted Mar 1, 2013 0:40 UTC (Fri) by jke (guest, #88998)Parent article: Google releases a better compression algorithm
I'd like to see a compression/decompression library that uses more than a single thread in a meaningful way.
But then again most new compression libraries seem to all suffer from not being a "drop-in replacement" for the well rooted and entrenched.
I'd be happy even if the file format was incompatible so long as the decompresser seamlessly handles legacy formats as well.
(Log in to post comments)
Google releases a better compression algorithm Posted Mar 1, 2013 1:15 UTC (Fri) by wmf (guest, #33791) [Link]
pigz and pxz exist; I don't know if they've been librarified but it would seem the hard work has been done.
Google releases a better compression algorithm Posted Mar 1, 2013 1:45 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
Mock, the tool used by the Fedora build system as well as packagers for building packages in a chroot uses pigz based on my suggestion and it seems to work fine for our purposes.
Google releases a better compression algorithm Posted Mar 1, 2013 1:57 UTC (Fri) by keeperofdakeys (subscriber, #82635) [Link]
Pxz has its own disadvantages though. Since (from what I have read) LZMA can't be multithreaded easily, pxz compresses different parts of the file at the same time (dumping them in /tmp), and finally putting them all together at the end. For solid archives this reduces the compression ratio in exchange for a (quite significant) speedup. In most cases the ratio watsed is quite small, so the tradeoff is worth it.
Google releases a better compression algorithm Posted Mar 1, 2013 9:16 UTC (Fri) by nmav (subscriber, #34036) [Link]
Indeed. However I'd suggest looking at plzip instead of the xz tools. I was surprized by the difference in compression ratio between them, even if they are based on the same compression algorithm.
http://www.nongnu.org/lzip/lzip.html
Google releases a better compression algorithm Posted Mar 1, 2013 8:02 UTC (Fri) by jezuch (subscriber, #52988) [Link]
There's also pbzip2. Bzip2, being a BWT-type algorithm, is quite easily parallelizable with little cost to compression ratio.
Google releases a better compression algorithm Posted Mar 2, 2013 5:06 UTC (Sat) by SEJeff (subscriber, #51588) [Link]
pbzip2 is incredible for bz2 files, but it will use up 100% of your available memory when you run it on big binary files. even with ionice -c3, it can be very intrusive
|
|||||||||||||