Removing support for DeltaRPMs in Fedora

Way back in 2009, we looked at the presto plugin for yum, which added support for DeltaRPMs to Fedora. That package format allows just the binary differences (i.e. the delta) between an installed RPM and its update to be transmitted, which saves network bandwidth; the receiving system then creates the new RPM from those two pieces before installing it. Support for DeltaRPMs was eventually added to the distribution by default, though the feature has never really lived up to expectations—and hopes. Now, it would seem that Fedora is ready to, in the words of project leader Matthew Miller, "give DeltaRPMs a sad, fond farewell".

Miller raised the question of retiring DeltaRPMs in a February 21 post to the Fedora devel mailing list. He pointed to a five-year-old open bug report that described problems with retaining the .drpm files for packages due to the way the Pungi distribution composer works. Miller also noted that a thread from 2021 discussing "deltarpm usefulness?" did not come to any firm decision.

The problem is that the DeltaRPMs are created but only getting synced to the mirrors as part of the update composed on the day the DeltaRPMs was created; the next day, a new distribution update gets composed, without using the previous DeltaRPMs, and that gets pushed to the mirrors. The net effect, as Jonathan Dieter pointed out in the bug report, is that the DeltaRPMs are only available for a day; "That means that the only way to take full advantage of deltarpms in Fedora is to update every single day." Doing things that way "has very little end-user value", Miller said.

There are some benefits to using DeltaRPMs, especially for those on slow or metered connections, but generally bandwidth consumption has faded as a major problem for most people at this point. Meanwhile, as Kevin Fenzi pointed out in the bug report, there are still costs:

Sure, you save BW [bandwidth], but you spend more in CPU and disk I/O to reassemble the rpms on every client machine. You also make updates pushes slower and use more resources.

Beyond that, there are some technical hurdles in the way of better DeltaRPM retention on the Fedora mirrors. The problem has never risen to the top of anyone's priority list, which is unfortunate, but now probably is not the time to address that, Miller said. Instead, Fedora has other technologies: "We have ostree and various container-delta approaches. We should focus on those [...]".

There were few complaints in the thread about that conclusion—generally, the opposite, in fact. Stephen Smoogen wondered when the axe should fall; should DeltaRPMs be removed for Fedora 39 and beyond, after a particular date, or, perhaps, discontinued for the upcoming Fedora 38? Miller said that normally he would be suggesting starting with Fedora 39, since the deadline for changes to Fedora 38 (which is due in April) has passed, but the DeltaRPMs have largely not been available anyway. He asked the infrastructure and release-engineering folks what the ramifications of simply shutting off the creation of DeltaRPMs would be. Kevin Fenzi replied that it was an easy change to make (and to back out if need be), without much in the way of risks.

Both Gary Buhrmaster and Dennis Gilmore posted their anecdotal experiences with the availability (and savings) from DeltaRPMs. The feature did not really provide much for either of them; Buhrmaster put it this way:

While occasionally I have seen a small decrease in the size of the files transferred (which certainly can benefit some people some of the time), the total elapsed time of the transaction has always ended up being higher as the recreation of the original rpm exceeds the time that it would have taken me to just download the full new rpm (with an admittedly reasonably high speed network provider in my environment).

The savings versus cost of DeltaRPMs is not entirely easy to work out; Ben Cotton thought the tradeoff may be particularly problematic for those with lower-powered systems. "And since delta RPMs trade bandwidth for CPU, it probably makes things worse for folks in developing countries." He did wonder if the feature should only be turned off by default on Fedora installs, while the distribution continued building the .drpm files; that would "still allow people to opt-in to it" and the distribution to measure its use. But Miller said that, even with the measurement, it would still be difficult to judge whether the benefit outweighs the cost.

Several times in the recent thread, Demi Marie Obenour advocated removing DeltaRPM support in order to reduce the attack surface of the distribution. She is mostly concerned with security holes in the program that reassembles the RPM from the delta, as she described in a March 2022 post:

This assumes that deltarpm (the program) does not contain any security flaws of its own, which could allow for code execution while the deltarpm is being applied. This is a bad assumption: a cursory audit I did found that it is not designed with untrusted input in mind. The code is also quite hard to follow, which makes auditing it quite difficult. Finally, it exposes decompression libraries to untrusted input before signature verification, and it itself has at the very least several areas where a bad deltarpm could cause it to allocate gigabytes of RAM.

In the earlier thread that Miller had pointed to, there was some confusion about the integrity and authenticity checking for DeltaRPMs. Marek Marczykowski-Górecki had proposed disabling DeltaRPMs in that earlier thread as well, but thought another reason to do so was security-related because DeltaRPMs are "processed before checking the package signature, which exposes rather big attack surface [...]". But Fenzi described the process for assembling and checking a DeltaRPM, which is just as secure as the checking for a regular RPM, he said. Obenour's concerns about the reassembly process are still valid, of course, but:

drpms work by downloading the delta, then using it + the version you have installed to recreate the signed rpm (just like you downloaded the full signed update) and then the gpg signature is checked of that full rpm, just like one you downloaded. If the drpm is tampered with it won't reassemble and it will fall back to the full signed rpm.

The security concerns seem like they could be addressed, as could the build-process issues that drastically reduce the availability of DeltaRPMs, but there is no large impetus for the feature these days. The tradeoff is potentially valuable in some scenarios, but the cost has been deemed too high by many. Given that users have been largely living without even the limited benefits that the feature provides—for a fair number of years and Fedora releases—the time has come to let it go. That will not happen without some action by the Fedora Engineering Steering Committee (FESCo), but with essentially zero opposition to removing DeltaRPMs, agreement seems clear.