No more .bz2 files from kernel.org [LWN.net]

No more .bz2 files from kernel.org

Posted Dec 28, 2013 17:31 UTC (Sat) by theophrastus (guest, #80847) [Link] (12 responses)

Curious as to why this was happening, i read the linked to and was still left uneducated. i suppose the answer might be 'fewer file types to deal with', but they're still allowing two. so if anyone knows the really real reason, (that an evil sinister Linus-insulting cabal is behind bzip2), please give us a hint here - thankee.

No more .bz2 files from kernel.org

Posted Dec 28, 2013 18:01 UTC (Sat) by Tara_Li (guest, #26706) [Link]

My guess is that two formats is all they really want to support - gzip for seriously old machine support, likely for machines with limited capabilities, and that in testing, xz tends to perform better than bzip2, so they go with xz for modern machines.

No more .bz2 files from kernel.org

Posted Dec 28, 2013 19:00 UTC (Sat) by mricon (subscriber, #59252) [Link]

Providing every release in .tar.gz, .tar.bz2 and .tar.xz was deemed overly redundant. The goal is to provide two formats -- .tar.gz for fast decompression and maximum compatibility between various platforms, and .tar.xz for smaller tarballs (in order to save bandwidth for people downloading releases). With xz becoming ubiquitous, continuing to provide .bz2 releases was seen as offering little ongoing value, and so was dropped.

No more .bz2 files from kernel.org

Posted Dec 28, 2013 20:00 UTC (Sat) by lambda (subscriber, #40735) [Link]

The real question is why they would keep three formats. gzip is useful as it's available pretty much everywhere in the default install, even on quite old systems, so anyone can use it. xz provides the best compression, so the smallest downloads for people who are able to use it. bzip2 was sitting in the middle not being particularly relevant; it's a bit smaller than gzip, and maybe a bit more widespread than xz, but each of the other formats already covered those use cases.

As xz has become more commonly available, the reason for keeping bzip2 around has become smaller and smaller. At some point, you just need to decide that the hassle of supporting 3 formats is not worth the value it provides.

No more .bz2 files from kernel.org

Posted Dec 28, 2013 20:13 UTC (Sat) by jhoblitt (subscriber, #77733) [Link] (7 responses)

The motivation may be save bandwidth by pushing those using bzip2 out of habit instead of necessity to lzma.

No more .bz2 files from kernel.org

Posted Dec 28, 2013 20:44 UTC (Sat) by theophrastus (guest, #80847) [Link] (6 responses)

thank you for the explanation, one and all.

(so... "sudo apt-get install lzma".

hnh, even the informational messages are 'compressed':

>unlzma -v -t fred.lzma
fred.lzma: decoded succesfully

..or just too sophisticated for spelling [wink])

No more .bz2 files from kernel.org

Posted Dec 28, 2013 20:55 UTC (Sat) by josh (subscriber, #17465) [Link] (5 responses)

You actually want xz-utils installed. Then use tar xf (without the usual z, j, or J), and tar will autodetect the right decompression tool to use.

No more .bz2 files from kernel.org

Posted Dec 29, 2013 21:17 UTC (Sun) by FranTaylor (guest, #80190) [Link] (2 responses)

This is a handy package to install because it will allow you to write install scripts that will break on every other platform.

No more .bz2 files from kernel.org

Posted Dec 29, 2013 21:47 UTC (Sun) by josh (subscriber, #17465) [Link]

Autodetection of decompressors is a built-in feature of tar; installing the package just gives tar the decompression tools it needs.

No more .bz2 files from kernel.org

Posted Jan 4, 2014 2:43 UTC (Sat) by k8to (guest, #15413) [Link]

Not really. The flags like I and j and so on only exist in gnu tar for the most part and aren't entirely standard across the history of gnu tar patches either. If you're dealing with some other tar, there's not going to be a way to ask it to unpack the .xz file for you anyway.

The only way to make slightly clearer shellscripts is to manually invoke the unpack with a pipe, and the danger to neglect to do so is opened by installing gnu tar. You could install schilly's tar if you want!

No more .bz2 files from kernel.org

Posted Dec 30, 2013 13:15 UTC (Mon) by clopez (guest, #66009) [Link] (1 responses)

You use 'a' to enable autodetection:

tar xfav file.tar.xz

No more .bz2 files from kernel.org

Posted Dec 30, 2013 19:43 UTC (Mon) by josh (subscriber, #17465) [Link]

That's only for auto-compression based on file extension. Auto-decompression requires no option:

/tmp$ tar caf foo.tar.xz foo
/tmp$ file foo.tar.xz
foo.tar.xz: XZ compressed data
/tmp$ tar xf foo.tar.xz
/tmp$ tar tf foo.tar.xz
foo/
foo/bar

No more .bz2 files from kernel.org

Posted Dec 30, 2013 14:04 UTC (Mon) by domo (guest, #14031) [Link]

xz compresses tighter than bzip2 and decompresses faster than bzip2,
so to support 2 then gz & xz seems now like the best choice.

No more .bz2 files from kernel.org

Posted Jan 2, 2014 1:13 UTC (Thu) by PaulWay (guest, #45600) [Link] (1 responses)

A side note - it would be worth using zopfli to compress the .gz files. Zopfli spends a lot more time in the compression phase getting better compression, but the output is completely compatible with gzip.

Or if speed and minimal memory usage is preferred over size, use lzo.

I've been doing a bit of research on this topic for my own library (Cfile) :-)

Have fun,

Paul

No more .bz2 files from kernel.org

Posted Jan 2, 2014 18:15 UTC (Thu) by rriggs (guest, #11598) [Link]

LZO is nice but the last time I looked into it there was only a GPL library available. Has anyone released a good non-GPL LZO library yet -- one that can be used by MIT, BSD or Apache licensed software?

No more .bz2 files from kernel.org

Posted Jan 2, 2014 18:41 UTC (Thu) by Comet (subscriber, #11646) [Link] (1 responses)

Back in '99 or so, I sent an email to the archive maintainers noting that the files were only in .gz format, showing the size benefits to using .bz2, the difference in time and thus the difference in cost when paying for dial-up modem access in the UK.

I was pleasantly shocked when the reply was an email telling me that they'd gone ahead and provided .bz2 of every file and this was now complete.

As the instigator (though not the hard worker) behind the availability of .bz2 files, I heartily endorse this change. Storage costs money too and the archive folks are generous enough already to provide two versions, to take on some more cost for themselves (and the RAM for their working sets) to benefit people retrieving content. You either grab the "portable" variant or the "best compressed" variant, there's no need for something in the middle. If download times/costs matter to you, install the xz tools.

Zsync for ubuntu images

Posted Jan 3, 2014 4:12 UTC (Fri) by gmatht (guest, #58961) [Link]

Reminds me of the days I requested Ubuntu provide zsync files for their images. It took a while, but they've added the zsync files. For those who aren't familiar with zsync, it uses a clever trick so that you can download only those parts of a gzip file that have changed.