Relocating Fedora's RPM database

The deadlines for various kinds of Fedora 36 change proposals have mostly passed at this point, which led to something of a flurry of postings to the distribution's devel mailing list over the last month. One of those, for a seemingly fairly innocuous relocation of the RPM database from /var to /usr, came in right at the buzzer for system-wide changes on December 29. There were, of course, other things going on around that time, holidays, vacations, and so forth, so the discussion was relatively muted until recently. Proponents have a number of reasons why they would like to see the move, but there is resistance, as well, that is due, at least in part, to the longstanding "tradition" of the location for the database.

To /usr

As is normal for Fedora change proposals, program manager Ben Cotton posted the proposal to the mailing list on behalf of its owners: Chris Murphy, Michel Alexandre Salim, and Neal Gompa. It can also be found on the Fedora wiki. In a nutshell, it would move the database maintained by the RPM package manager from its current location, /var/lib/rpm, to /usr/lib/sysimage/rpm; the former would be replaced with a symbolic link to the latter. The DNF package-management tool would not be switching locations for its database, at least yet; the RPM developers have mixed feelings about the change, but are not standing in its way.

The RPM database tracks which packages have been installed and their metadata. In addition, it tracks all of the files that get installed to the system, their locations, and which package is responsible for them. RPM uses it to clean up packages when they are removed; users can query the database, as well, to answer various questions about their packages and files.

There are several benefits for Fedora listed in the proposal, but the main driving force seems to be support for rolling back failed or undesired updates of the operating system. The RPM database describes what is in /usr for the most part, so having it stored in the same directory hierarchy means that /usr can be rolled back as a unit, without needing to change /var. The switch is also in keeping with what the Fedora variants based on rpm-ostree (CoreOS, IoT, Sliverblue, and Kinoite) already do; it is also aligned with another RPM-based distribution ecosystem, the SUSE distributions (SUSE Linux Enterprise, openSUSE, and Tumbleweed).

Much of the early discussion revolved around the Filesystem Hierarchy Standard (FHS), which Fedora packaging explicitly follows—with a few exceptions. Tom Hughes pointed out that the FHS descriptions for /usr ("shareable, read-only data") and for /var ("variable data files") seem somewhat at odds with the proposal. In particular, he said, the /var description "appears to exactly describe the RPM database". Thus the changes are not FHS compliant, nor do they follow the packaging guidelines, he said.

Stephen John Smoogen agreed and suggested that the change needed to be made in the FHS first, then it could be done for the distribution. Fedora project leader Matthew Miller did not think that was likely to happen, since the most recent release (FHS 3.0) was in 2015 "and the whole thing has been effectively dead since". On the other hand, Miller was in favor of reviving the FHS effort, especially with buy-in from other distributions.

But, as Murphy (and others) said, no matter what the FHS says, /usr cannot be more than "mostly read-only" or the system can never be updated:

In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed.

RPM developer Panu Matilainen agreed that the database has generally only been changed during update operations, "but it doesn't mean it will stay that way forever more". There are unimplemented features that might change that situation; beyond that, there are already situations where changes are made completely outside of /usr but that require an update to the RPM database:

There seems to be this strange underlying assumption that all packaged content lives in /usr when that's not at all the case. To install a kernel, or a config-only package (under etc), or 3rd party software putting stuff under /opt, or... you need a writable rpmdb. Ditto for 'rpm --import'.

Vitaly Zaitsev also pointed to the FHS and its "read-only" language for /usr, but as Gordon Messmer said: "If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally)." Gompa concurred with that, and expanded on the advantages of moving the database out of /var:

The bigger problem is that having the RPM database in /var makes it much harder to correctly implement a boot-to-snapshot scheme for Fedora Linux that reasonably maintains system state properly once /var is carved out. The reason that /var *isn't* carved out by default with our Btrfs configuration is because of the RPM database. Once the RPM database is moved, it becomes possible to split /var into its own subvolume and make it trivial to configure a full boot-to-snapshot system leveraging the technologies we have available to us.

Miller suggested that the "Benefit to Fedora" section of the change proposal add more information about that "pretty compelling benefit". Murphy made that change, but did caution: "There's more hurdles to jump, just so no one thinks snapshots and rollbacks are showing up in Fedora 36."

/state

Things had mostly quieted down in the discussion after the new year, at least until David Cantrell threw a bit of a curve ball on January 9. As with others in the thread, he was uncomfortable with moving the database to /usr, "but we should move it to gain the improvements as noted in the feature proposal". In his lengthy message, he suggested some other, novel ways to look at /usr and /var:

It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or [expectation] now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place.

[...] Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably.

With that in mind, he suggested moving "the RPM database and other variable and stateful data" to a new top-level directory called /state. The FHS does not prevent the addition of new top-level directories, Cantrell said. Michael Catanzaro thought that adding a top-level directory was "a pretty big hammer" and that perhaps it was easier to just support two separate locations for the RPM database.

But rpm-ostree developer Colin Walters took exception to Cantrell's "unnecessary and sort of out of place" characterization. "Multiple independent groups who *actually work* on image based updates and/or client side snapshots all generally agree that the rpmdb should be in /usr." Murphy said that /state does not really solve the problem, it is "just rearranging the chairs". The RPM database holds information about multiple locations, so it needs to stay in sync with the rest of the system, no matter where it lives:

If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable.

If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help.

Cantrell is not alone in feeling that /usr is a bad location, however; Matilainen said that putting the RPM database there "just *feels so wrong*". There is other data like the RPM database, he said, so having a separate location for it all makes a lot of sense:

For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs.

Just FWIW, I would quit my whining about this right there if it went to a new toplevel directory instead because it just *feels* right unlike /usr.

Further pieces

Walters said that problems, such as PGP keys being added via rpm --import, need to be addressed as part of the adoption process:

The TL;DR for me is: I think everyone agrees that moving the rpmdb as it is today to /usr is not 100% a perfect fit. But it's a ~90% fit - almost all the raw data is just headers which are clearly immutable data generated elsewhere. And by making this change, we're basically saying we'll fix the remaining 10% of cases.

[...] But, I hope we can get agreement about something like having `rpm --import` write to `/etc/pki/rpm-gpg` and dropping gpg keys from the rpmdb.

There is, it seems, an effort toward regularizing package installation in various ways with an end goal that is clear to some, but perhaps not to others. Zbigniew Jędrzejewski-Szmek described where he sees things heading:

Traditionally, packages installed all kinds of files all over the place. But we're slowly and painfully moving towards the model where:

1. packages are only allowed to install under /usr, /var, and /etc. (Or under /opt, but I'd want to move that to /usr/opt…)
2. packages must support /var/cache being wiped at any time, and most packages support anything under /var being wiped at any time.
3. systemd and other projects are trying to only use /etc for local admin state, and support "factory reset" by wiping /etc and /var.

Based on that, he said that it makes sense to put the database under /usr somewhere, and the exact location is only a matter of convention, "so obviously we want to follow what opensuse and others are already using". But Matilainen was concerned that there was something of a "hidden agenda" behind the change proposal. He thought the goals should be more clearly stated:

I'm not saying these are necessary bad goals at all, it's just that there's a huge disconnect between reality and the above model on which this change seems based on, and not a single mention about these goals and changes needed to get there. I mean, I totally get that you can't change everything at once, but if there's a plan this big behind something then maybe it should be brought up front, no?

Matilainen said that "nearly all packages put something in /etc", for example, which is something that is being ignored, but Walters said that the handling of /etc is something that is being improved in the image-based update world:

rpm-ostree uses ostree, which introduces /usr/etc which are the pristine default config files. /etc is 3-way merged by ostree. One of the major benefits of this that I really love is `ostree admin config-diff` - at any point we can show you machine-local changes from the default, and it's trivial to reset back to defaults without redownloading a whole RPM.

[...] There's no hidden agenda - the goal is to support image based updates as well as client side snapshots, factory reset, etc. And we're shipping today versions of Fedora that do a lot of this, and we want to continue to improve it.

The discussion is still ongoing as of this writing, which likely means that the Fedora Engineering Steering Committee (FESCo) will not decide on the proposal right away. Several members are favorably disposed to it, as can be seen in the FESCo issue entry. Matilainen has said that he is "not going to stand in front of the Change truck", even though he has reservations about the proposal. While it may "feel wrong" to do so, at least for some, it seems that the writing may already be on the wall for this particular change. Whether that larger agenda (hidden or not) is adopted will play out over the next few releases.