Formats supported [LWN.net]

Formats supported

Posted Jan 5, 2008 18:10 UTC (Sat) by epa (subscriber, #39769)
Parent article: RPM 5.0 released

The RPM packages, in addition to the default Gzip and optional Bzip2 compression, now support also LZMA compression.

This sounds like a very handy feature given the long amount of time my PC spends downloading the latest 0.0.0.0.1 version update of OpenOffice from a Fedora mirror. lrzip would be even better. (Better still some way for smart, yum etc. to download only the differences between one package and the next, but I don't know if that really needs any involvement from rpm, apart from supporting packages built without compression or with gzip -0.)

Finally, support for the old RPMv3 (LSB) package format was removed to cleanup and simplify the code base.

I know a lot of people don't have much respect for the LSB, either as an idea or an implementation, but surely this is a retrograde step? No distribution could remain LSB-compliant unless they included a separate, old version of RPM just to install LSB packages or converted them with utilities like alien. This is already the situation faced by Debian, Slackware, Ubuntu and others, so perhaps it's 'fairer' that all distributions have to treat LSB packages as different from their native packaging format. RPMv3 would then be a distribution-neutral package format.

However it's not clear that Linux distributors are going to adopt the new rpm 5 (especially not Fedora and Red Hat), so it could all be academic.

(Log in to post comments)

> Better still some way for smart, yum etc. to download only the differences between one package and the next, You're wanting "Presto!" It's in Fedora 8. But the binary diff repos are not ready yet. Maybe in time for F9? I know that Suse was using some sort of binary diffs years ago, though I don't recall being impressed by their speed.

Formats supported

Posted Jan 5, 2008 20:25 UTC (Sat) by jengelh (subscriber, #33263) [Link]

There are .patch.rpms, which contain full copies of all files that changed.
Then there are .delta.rpms, which are deltas - and yes, I have to agree, applying deltarpms is
not too fast. But that is a general thing about deltaing, not limited to rpms. As soon as
transferring the full .rpm is faster than delting it up, deltarpms are useless unless you have
a strict pay-per-byte pipe.

Formats supported

Posted Jan 6, 2008 7:36 UTC (Sun) by Richard_J_Neill (subscriber, #23093) [Link]

Surely you don't have to maintain the original .rpm file to use a delta-rpm.  The original
.rpm could be re-constructed on the fly from the packages that it installed.

Formats supported

Posted Jan 6, 2008 18:56 UTC (Sun) by MattPerry (guest, #46341) [Link]

Unlikely as RPMs contain metadata and pre- and post-installation scripts that are not kept on
the filesystem after installation.

Formats supported

Posted Jan 6, 2008 19:01 UTC (Sun) by jengelh (subscriber, #33263) [Link]

Scripts are kept. Otherwise, how would you run %preun and %postun? So yes, deltarpm magically
takes both the rpmdb and the delta file and produces a full (non-delta, perhaps .patch) rpm
out of it. And that takes a whee while.

Formats supported

Posted Jan 7, 2008 15:21 UTC (Mon) by roblucid (subscriber, #48964) [Link]

Actually OpenSuSE claims to do exactly that.  The Online update takes a 
delta rpm, and constructs the new files from the old, plus what's in the 
rpm delta file.

It generally works to!

Formats supported

Posted Jan 7, 2008 10:20 UTC (Mon) by epa (subscriber, #39769) [Link]

Hmm, surely a binary diff can just be generated on demand -
diff?version_1=bash-1.rpm;version_2=bash-2.rpm - and there is no need for lots of tiny diff
files to be pushed out to mirrors.  Because the diffs will be much smaller, indeed there may
not be a need to mirror them out.

(I'm still wondering why distros don't use bittorrent for updates, the idea of mirror sites
seems so clunky nowadays.)

Impossible

Posted Jan 7, 2008 23:15 UTC (Mon) by khim (subscriber, #9252) [Link]

Genration of DIFF is quite slow and requires some software not usually present on ftp-mirror sites. Thus it makes sense to precreate diff files and push them to mirrors - or not use them at all...

Impossible

Posted Jan 9, 2008 14:12 UTC (Wed) by epa (subscriber, #39769) [Link]

My point is, don't bother with the mirror sites, just have a single site generating the diffs
on demand.

>Impossible<

Posted Jan 9, 2008 14:43 UTC (Wed) by gvy (guest, #11981) [Link]

Useless w/o caching/propagating them.

Hey, go try xdelta1/xdelta3 for yourself and then imagine doing that "on demand".

Formats supported

Posted Jan 12, 2008 1:31 UTC (Sat) by mp (subscriber, #5615) [Link]

> I'm still wondering why distros don't use bittorrent for updates,
> the idea of mirror sites seems so clunky nowadays.

Actually, they try to: http://wiki.debian.org/DebTorrent

""" However it's not clear that Linux distributors are going to adopt the new rpm 5 """ The last thing we need is to become dependent, again, upon a project headed up by a uncooperative developer. The XFree86->Xorg move has clearly been beneficial. Standardizing on the Jeff Johnson fork of RPM would be unwise, and a step backwards, in my opinion.

Just in case it might seem too beneficial, XF86 [speaking of e.g. 3.3.6, 4.3.0] was *much* more robust per se than X.org -- which is currently in top3 insane upstreams for me. At least it did work, and they didn't manage to break *intel* drivers after specs were long published! Re Jeff, I'm afraid I'd also stay away from Red Hat. While having a lot of respect for them, I don't consider them a vendor-neutral... er... vendor.

rpm5.org now has an extra mailing list: rpm-lsb . Right now the archives show just one message: http://rpm5.org/community/rpm-lsb/0000.html Jeff Johnson writes there that he wants to document the format of rpmv4 (which is the one actually used today) well enough so it could be used in the LSB specifications. He hopes "to start in mid-January, and have a completed document by the end-of-March or so." And he's not completely alone. While Mandriva has decided to use the original rpm recently, there are still a number of distros that use rpm from rpm5.org . See the following thread: http://rpm5.org/community/rpm-devel/index.html#1824 Disclaimer: I don't follow closely rpm.org or rpm5.org .

By the way, I recently learned that 7z compresses noticeably tighter than bzip2, and even than lrzip (for the data sets that I was interested in). 7z also has the advantage over lrzip (and over rzip) that it can operate in "streaming" mode, and it has the advantage over all of them (except probably gzip) of being faster/taking less CPU time. http://allmydata.org/pipermail/tahoe-dev/2007-December/00...

This discussion has been had many other places, and so far, LZMA (aka 7z) isn't unixy enough to behave like we all expect gzip and bzip2 to behave. Outputs and subarchiving, not to mention magic numbers and crc, seem to be troublesome still. It does what it does very well, sure, but it doesn't do what gzip and bzip2 do well. The other aspect is that of "sufficient improvement to outweigh transition costs". Once it behaves itself such that it can be a drop-in replacement for gzip or bzip2, the algorithm speed and CPU efficiency will be no-brainer reasons to switch. No workflow changes, IOW. As your referenced email states, it's all about the tradeoffs and the conditions of use. Look up HPA on kernel binary compression formats.

Whether removing support for RPMv3 header+payload signatures from rpm-5.0 is a retrograde step or not depends on whether you think that the additional signature on header+payload adds security or not. However, I do point you at the following links for additional information regarding LSB "standard" packaging: http://rpm5.org/community/rpm-lsb/0001.html https://lists.linux-foundation.org/pipermail/packaging/20... The original report of problems with LSB packaging failing to pass LSB's own test suite are at http://www.linux.org.ru/view-message.jsp? msgid=2391597&lastmod=1199814655335&page=2 (apologies for the URL line wrap) hth