LWN: Comments on "Removing support for DeltaRPMs in Fedora"

Removing support for DeltaRPMs in Fedora

rahulsundaram — Fri, 17 Mar 2023 16:27:23 +0000

> I can't speak for the fedora world, but in the Debian world the "unit of maintinance" is the source package, so if anything in a source package needs to be changed all vinary packages built from that source package get an update.

Unit of maintenance and binary but yeah this is true for Fedora and likely every distribution that builds from source.

Removing support for DeltaRPMs in Fedora

plugwash — Fri, 17 Mar 2023 14:54:49 +0000

I can't speak for the fedora world, but in the Debian world the "unit of maintinance" is the source package, so if anything in a source package needs to be changed all vinary packages built from that source package get an update.

Removing support for DeltaRPMs in Fedora

paulj — Mon, 13 Mar 2023 13:39:04 +0000

At least one of the gigantic (in terms of DC space owned at least) tech companies uses a software update system that is based on bittorrent for distribution.

Removing support for DeltaRPMs in Fedora

timon — Mon, 13 Mar 2023 11:20:03 +0000

The man-db triggers for apt/dpkg really annoy me, good thing you can just go without the man cache. I usually do something like this on any Debian-like machine:

# rm -r /var/lib/man-db

From now on, apt/dpkg will tell me something like:

> Processing triggers for man-db (2.10.2-1) ...
> Not building database; man-db/auto-update is not 'true'.

You might also want to clean up after man-db now, removing the actual cache in /var/cache/man, disabling or deleting cronjobs and systemd timers, ...

Removing support for DeltaRPMs in Fedora

cozzyd — Mon, 13 Mar 2023 03:33:06 +0000

In my experience, it's often the SELinux policy updates that are the slowest, as well as other ancillary things like updating mandb and such. Why they're so slow, I'm not sure...

Removing support for DeltaRPMs in Fedora

rwmj — Sun, 12 Mar 2023 12:45:27 +0000

I wanted to make the point that we could do this with a small change to existing proxies and an additional HTTP header, rather than boiling the ocean with a whole new protocol.

Removing support for DeltaRPMs in Fedora

pabs — Sun, 12 Mar 2023 12:40:36 +0000

That sounds like a bit like IPFS or the BitTorrent DHT. IIRC apt-p2p/debtorrent used something more like BitTorrent.

Removing support for DeltaRPMs in Fedora

rwmj — Sun, 12 Mar 2023 11:22:11 +0000

It would be nice if there existed a content-addressable CDN, ie. you'd request the hash of the data you want. The distro would then just publish the current hash of the top level index file, and everything would be hashes from there. I came up with a proposal for this a while back: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-con...

Removing support for DeltaRPMs in Fedora

gdt — Sat, 11 Mar 2023 07:06:25 +0000

The need for mirrors comes from two sources. (1) The decreased performance of TCP over networks with a high round-trip time; you'll get better performance on a short fast network. (2) The high cost of off-shore bandwith, be that undersea or satellite.

Sites like Google, Facebook and Netflix build out content distribution networks to solve these issues as well as to spread load. Attempts to build a free CDN have not been nearly as successful as mirrors. My guess is that's because mirrors have more ability for the server owner to apply policy; for example, a mirror might limit itself to Linux software, or to retrogaming, etc. A Linux distribution could use a commercial CDN, but for a network which wants more rapid access to FOSS software, that general-purpose CDN is even more problematic (eg, that network could install, say, an Akamai cache but still not be assured that FOSS software would be cached on-shore).

Slowness of RPM package updates

mbunkus — Sat, 11 Mar 2023 00:10:41 +0000

Ah yes, all the fine assumptions about timezones and how they're really not that difficult 😁 I highly suggest the classic Computerphile video about it[1]; Tom Scott is both very funny and very right in this one.

[1] https://www.youtube.com/watch?v=-5wpm-gesOY

Slowness of RPM package updates

WolfWings — Fri, 10 Mar 2023 22:24:18 +0000

Australian Central Western Standard and Nepal's timezone would like to remind you they exist on a 15-minute increment, and while ACWS covers barely a hundred people Nepal has 30-million-ish living at GMT+5:45. :)

Removing support for DeltaRPMs in Fedora

pabs — Fri, 10 Mar 2023 03:25:13 +0000

Debian has debdelta, which does something very similar to DeltaRPM, but there is also a plan for "delta debs", where the end-user system applies the delta patches directly to the filesystem, instead of generating .deb files that then get installed as per normal.

http://debdelta.debian.net/
https://debdeltas.debian.net/
https://wiki.debian.org/Teams/Dpkg/Spec/DeltaDebs

Removing support for DeltaRPMs in Fedora

excors — Thu, 09 Mar 2023 23:40:13 +0000

> We have to basically keep one per package as far back as want to get back to. Many packages get updates daily but people in low bandwidth areas are also usually only updating any where from 1/week to 1/month. That means we need to keep at least a month worth of every package around so that the delta between A-one-month-ago.rpm and A-today.rpm can be calculated.

Probably not really relevant to Fedora but for general interest in this topic: Windows has tried a few different approaches for efficient updates, described in https://devblogs.microsoft.com/oldnewthing/20200213-00/?p... (and earlier posts linked from there).

"Express" updates sound closest to DeltaRPMs: they generate a patch from every previous release to the current release, and do that every time there's a new release, so you end up generating O(n^2) patches after n releases. That's quite expensive on the server side and not great for caches.

"Quality" updates start with a single baseline release. For every new release, they generate a patch from the baseline to the new release, plus a reverse patch from the new release back to the baseline. Then a client running any arbitrary version can apply the reverse patch for their version, followed by the forwards patch for the latest version. That means you end up with O(n) patches in total. The process is not as bandwidth-efficient as express updates, but it sounds like it simplifies a lot of the update infrastructure.

MAC address is not a UUID

jreiser — Thu, 09 Mar 2023 21:36:55 +0000

Many MAC manufactured in the last 15 to 20 years or so have a writable address that gets auto-initialized from a ROM at power-up. The software driver can change the MAC address, but doing so is rare because it would break higher level auto-configuration of local networks via DHCP. The next alternative is for the driver to choose a random IPv4 or IPv6 numerical address (request that the DHCP server assign a specific IP address chosen by the driver, instead of using the previous IP address that is remembered by the DHCP server looking up the MAC address in the server's persistent storage). Some Apple and Android kit have done this for years.

Removing support for DeltaRPMs in Fedora

Cyberax — Thu, 09 Mar 2023 20:54:00 +0000

> There's a privacy risk on doing the same for a Linux distribution.

What is the scenario? A hotel network setting up a rogue mDNS-based mirror?

I guess it's possible, but what would be the purpose? They already can use your MAC address to identify your machine.

Removing support for DeltaRPMs in Fedora

smoogen — Thu, 09 Mar 2023 20:16:51 +0000

CDNs work ok if the content does not change often. Fedora has used several different CDN's to try and help with things, but the amount of churn in updates, rawhide and other parts was constantly causing cache issues, download issues, and storage issues on the part of the CDN in different locations. A common problem was that one set of people would complain about broken RPMs but others in the same area would not. Each was going to the same 'IP' address but were getting two different backing stores which weren't getting synced out. Where we could have the mirror manager software say 'this mirror is not in sync, take it out until it is' the CDN would just show up as 'all-good' or 'all-bad' while users had completely different views.

Working with one of the CDN's we tried a method to keep more data longer in the CDN but ran over what they were wanting to give us in a week or so. I believe currently we only use the CDN's in specific targeted regions and solutions (aka if you have an Amazon cloud instance you will go through their CDN).

The problem is increased by the number of delta-rpms we need to keep. We have to basically keep one per package as far back as want to get back to. Many packages get updates daily but people in low bandwidth areas are also usually only updating any where from 1/week to 1/month. That means we need to keep at least a month worth of every package around so that the delta between A-one-month-ago.rpm and A-today.rpm can be calculated. Each one of those takes anywhere from a few seconds to a minute to calculate. We normally update a couple hundred packages to a thousand packages a day. Doing just the previous delta makes it so that the time to calculate deltas can be done in an hour or so for a compose to get out the door. We would need to extend that by about 7 to 30 times that to make it really useful for people who need delta rpms.

Removing support for DeltaRPMs in Fedora

cesarb — Thu, 09 Mar 2023 19:25:14 +0000

> Even *Windows* just does p2p update distribution without manual setup.

There's a privacy risk on doing the same for a Linux distribution. On Windows, you could assume that everyone who is on the same Windows version has the same set of Windows components. On a Linux distribution, the set of installed packages is usually distinct enough that it could even be used to fingerprint a single device. And that's just fingerprinting, the presence or absence of a package could be something you might not want to announce to anyone who happens to be on the same local network as your laptop (for instance, you might not want your peers to know that you have installed tor-browser, or emacs).

Of course, for desktop computers which never leave a trusted local network, that's not much of an issue, so it would be great if enabling a "p2p update distribution" mode were as simple as setting a configuration flag or installing a single package (with no further manual configuration needed).

Removing support for DeltaRPMs in Fedora

flussence — Thu, 09 Mar 2023 18:10:11 +0000

Local mirrors of entire files would work out better than deltas as soon as you have a few computers running the same OS.

But I'd also like to point out the experience of trying to do so in Linux utterly sucks. Even *Windows* just does p2p update distribution without manual setup.

We have nearly ubiquitous mDNS, everything is cryptographically signed, and the status quo is that if you want to do something as simple as (e.g.) getting a few Debian/Ubuntu boxes to share .deb files locally, you've got to ssh into every box individually to configure apt-cacher-ng by hand. I imagine Fedora's situation isn't much nicer, or else they would've done this sooner. My own Gentoo+NFS setup is functional but the less said about its specifics the better.

Removing support for DeltaRPMs in Fedora

rahulsundaram — Thu, 09 Mar 2023 14:38:17 +0000

> They might do better to provide a simple "black box" method for locally cacheing rpms site-wide, so that folks with two or more similar installations would only have to download an rpm once on their broadband

You might be looking for https://dnf-plugins-core.readthedocs.io/en/latest/local.html

quick-fedora-mirror at https://fedoraproject.org/wiki/Infrastructure/Mirroring#M... if you want something more full fledged.

Removing support for DeltaRPMs in Fedora

pabs — Thu, 09 Mar 2023 14:02:16 +0000

Distros often get CDNs for free, I note Debian does for eg.

Removing support for DeltaRPMs in Fedora

eru — Thu, 09 Mar 2023 13:40:51 +0000

I haven't seen deltarpm:s save anything, but then, I live in a land of fast net connections.
What I have wondered for along time, though, is why many updates seem to update RPM's that probably have had no other change than the version number. For example, any Wine update reinstalls a ton of font files. I guess it is a simplification for the project maintainers: Easier to label all components as Vn+1 even though many are same as in Vn or even Vn-1, than keep track of the real changes.

Removing support for DeltaRPMs in Fedora

epa — Thu, 09 Mar 2023 13:13:29 +0000

When I said proxy server I did not mean something you have to set in your web browser config. I meant a simple caching service where http://mycache/foo/bar gives you the content of http://download.fedoraproject.org/foo/bar and it will perhaps cache it to make later requests faster. That seems a lower maintenance alternative to the traditional mirror that has to copy everything in advance and may become outdated (as happened here with delta rpms).

Removing support for DeltaRPMs in Fedora

LtWorf — Thu, 09 Mar 2023 12:00:44 +0000

For mirror owners it's quite a good tradeoff to use algorithms with maximum compression.

Removing support for DeltaRPMs in Fedora

hailfinger — Thu, 09 Mar 2023 11:07:39 +0000

CDNs are expensive, and mirrors are free (the owner of the mirror takes care of bandwidth costs).

And for some larger organizations (universities, internet providers, ...) it makes sense to provide their own mirror for often-downloaded software to reduce the amount of external bandwidth needed. Faster downloading for their own users is a nice side effect.

Removing support for DeltaRPMs in Fedora

NRArnot — Thu, 09 Mar 2023 10:36:09 +0000

I won't miss them.

They might do better to provide a simple "black box" method for locally cacheing rpms site-wide, so that folks with two or more similar installations would only have to download an rpm once on their broadband. How about a live USB rpm-cacher that one could just plug into any old PC with a decent amount of space to spare on its hard disk, and a one-line mod to the .repo files?

The last time I had to upgrade an entire site (about 20 systems) from one Fedora release to another, I hacked together a special-purpose Squid proxy. But it had a few gremlins, and I had to disable it rather than leave it in place for day-to-day upgrades. It wasn't worth my time to work out what I was doing wrong and it had served its main purpose (of removing limited broadband bandwidth as an hours-at-work factor).

Removing support for DeltaRPMs in Fedora

gtirloni — Thu, 09 Mar 2023 10:02:23 +0000

I usually disable deltas because, more often than not, dnf gets stuck on them for reasons I never cared to investigate.

It was pretty cool when I was in a 256Kbps connection, but now that everyone I know has >100Mbps, it's much less useful.

So long, and thanks for all the fish, DeltaRPM :wave:

Removing support for DeltaRPMs in Fedora

Sesse — Thu, 09 Mar 2023 09:02:11 +0000

dpkg is slow because of two reasons:

1. It is very conservative in fsync; if e.g. the power goes out, the package manager needs to be able to rollback to a safe state.
2. It has a home-grown database and transaction system based on flat files, which means there can be no WAL or similar techniques to batch states, and every large action involves dozens of files. (For instance, when you start up dpkg -i, it reads a couple thousand of these flat files into RAM.)

#2 makes #1 _much_ worse. I'm fairly certain that if you switched to something like SQLite and redesigned the rollback system around that (not literally reusing database transactions for filesystem transactions, but sticking your intents into the database and then persisting them all to disk in a single commit), you could get package installation down to <100 ms on an SSD, cold-cache.

Removing support for DeltaRPMs in Fedora

Sesse — Thu, 09 Mar 2023 08:54:36 +0000

Actually, apt update takes minutes, due to pdiffs.

Removing support for DeltaRPMs in Fedora

pabs — Thu, 09 Mar 2023 08:13:55 +0000

I was talking about end users not about Fedora infrastructure, which definitely does not use delta stuff.

Removing support for DeltaRPMs in Fedora

anton — Thu, 09 Mar 2023 08:11:56 +0000

It seems to me that content delivery networks (CDNs) are the (mostly) user-transparent modern version of mirrors, and are extensively used by companies with lots of traffic for the same files. It's interesting that they have not replaced mirrors in those areas where mirrors are established; I guess the CDNs take more money. As with mirrors, sometimes (rarely) there are problems. Concerning proxy servers, they went out of favour a long time ago (even before https became common) for some reason; in a way CDNs deliver the same service, but the server side arranges them rather than the client side.

According to the article Ben Cotton thinks that CPU is more of a problem in developing countries than bandwidth. I rally doubt that. Bandwidth can be a problem even in not-so-central areas in developed countries and can differ by several orders of magnitude, while the CPU performance for single-threaded latency-sensitive code differs only by about an order of magnitude, even for older and weaker CPUs.

AFAIK git uses diffs over the wire (and only over the wire), and I think that's still a good approach.

Removing support for DeltaRPMs in Fedora

epa — Thu, 09 Mar 2023 06:52:54 +0000

It surprises me that there is still a need for mirrors. As they say bandwidth is much higher now. If the problem is with syncing files to the mirrors, maybe just stop using mirrors? Or replace the traditional ftp mirroring with a simple caching proxy server which fetches a file on demand and saves it for the next request.

Removing support for DeltaRPMs in Fedora

pabs — Thu, 09 Mar 2023 05:26:24 +0000

Unfortunately bandwidth quotas and slow networks *aren't* a thing of the past, so I feel sorry for the Fedora users out there who rely on those sorts of networks.

Slowness of RPM package updates

rahulsundaram — Thu, 09 Mar 2023 01:46:51 +0000

> This might be a worthwhile tradeoff, but it would be hard to be sure until someone has actually written an alternative and seen how fast it goes.

RPM is largely written in C and parts of it is now using Rust.

https://fedoraproject.org/wiki/Changes/RpmSequoia

If the reference to Python is about yum/dnf, then dnf5 is written in C++ (with Python bindings) and should improve speed (some numbers in the change proposal)

https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5

Slowness of RPM package updates

rgmoore — Thu, 09 Mar 2023 01:12:45 +0000

To put it another way, the designers chose to prioritize simplicity and maintainability over performance. It might be possible to rewrite the code in a better performing language than Python, cache the timezone data, and so forth, but that would come at the cost of making the code more complex and thus harder to maintain. This might be a worthwhile tradeoff, but it would be hard to be sure until someone has actually written an alternative and seen how fast it goes.

Removing support for DeltaRPMs in Fedora

josh — Wed, 08 Mar 2023 23:21:06 +0000

Yeah, it seems like pdiff was designed for a different era of tradeoffs. It was valuable when `apt update` took minutes; now, `apt update` takes seconds.

Slowness of RPM package updates

jreiser — Wed, 08 Mar 2023 21:14:08 +0000

Part of the reason is that every action generates several time-stamped lines in a log file. The time stamp is in local time, and it's written by Python code which keeps no local storage. So every line must open the timezone file, read and decode it, convert time_t to local time in ASCII, write the line, and close the timezone file. There can be many actions per second of real time, and the Python code does not realize that the the ASCII representation changes only once per second, and the timezone can change only on 30-minute boundaries. Tens of thousands of open+close of the timezone file add up.

Removing support for DeltaRPMs in Fedora

bartoc — Wed, 08 Mar 2023 21:08:29 +0000

I suspect much of it is down to scriptlets and perhaps somewhat outdated choices of compression codecs that save bandwidth but take a long time to extract. Also it's not uncommon to get a crap mirror, esp since which mirror you choose rotates.

Removing support for DeltaRPMs in Fedora

Cyberax — Wed, 08 Mar 2023 20:44:05 +0000

One thing I personally don't understand, is why reassembly is so slow?

Actually, why everything related to RPM or DEB packages is so slow? Some time ago I tried https://distr1.org/ and I was amazed by package installation speed (which is kinda its goal). APK in Alpine is another really fast package manager.

Removing support for DeltaRPMs in Fedora

Sesse — Wed, 08 Mar 2023 20:17:33 +0000

Same thing with apt's pdiffs…

Removing support for DeltaRPMs in Fedora

hikingpete — Wed, 08 Mar 2023 19:13:58 +0000

I welcome this change. It burns a little everytime I wait 30s for a download, 30s for a delta rpm reassembly, and then see that the 30s of reassembly saved only 2% on the download.