Addressing UID/GID drift in rpm-ostree and bootc
The Fedora Project is looking for solutions to an interesting problem with its image-based editions and spins, such as the Atomic Desktops or CoreOS, that are created with rpm-ostree or bootc. If a package that is part of a image-based version has a user or group created dynamically on installation, and it owns files installed on the system, the system may be subject to user ID (UID) and group ID (GID) "drift" on updates. This "UID/GID drift" may come about when a new image with updates is generated, and therefore files may have the wrong ownership. This can have side-effects ranging from mildly inconvenient to serious. No solutions have been adopted just yet, but there are a few ideas on how to deal with the problem.
UID/GID Drifting
In a traditional package-based install of Fedora Linux, a package would be installed once and upgraded thereafter. If part of its installation includes dynamically allocating a user and/or group, the UID and GID will be assigned once, and it should not be changed by upgrades to the package later on. However, things work differently for image-based versions of Fedora. Instead of installing packages once and then updating them as needed, the images created by rpm-ostree and bootc are generated from packages each time an update is created. (LWN covered rpm-ostree's underlying technology OSTree in 2014, and looked at bootc in 2024.)
Even though image-based distributions are often described as "immutable", that is not exactly the case. As the documentation for Fedora Silverblue explains, only some of the filesystem and directories are immutable or read-only. The /etc and /var directories store configuration files, runtime state, and data that is writable and persists between updates.
Most of the time UID and GID assignments will remain static between updates. As Zbigniew Jędrzejewski-Szmek pointed out on the fedora-devel mailing list when raising the UID/GID drift problem, as long as the list of packages remains the same, the installation order of the packages is stable and will always get the same UIDs/GIDs.
But we can get different assignments if the package installation order changes, for example because of some dependency change, or more/less packages are installed, or when one of the packages defines a new user/group. In general, we can expect the uid/gid assignments to change regularly.
He goes into detail using the opencryptoki package as an example. That package creates the pkcs11 group when installed, and its GID is selected at that time. The package has a file in its payload, /etc/opencryptoki/strength.conf, that is owned by the pkcs11 group. An rpm-ostree or bootc image is built that includes that package, and it receives the GID 999 (for example) from the range of GID numbers that are available to system accounts. Users deploy that image to their systems. The /etc/opencryptoki/strength.conf file is now owned by GID 999 because filesystems store ownership as numbers and not user or group names.
Later, when a new image is created from package updates, something changes in the package order. This means that the pkcs11 group gets GID 998 this time. Note that files in /etc (or /var) are not necessarily overwritten when there is a new image, so some of the files that have group ownership stored as GID 999, such as /etc/opencryptoki/strength.conf, may persist even though the pkcs11 group is now GID 998. So when users update to the new image, some files are owned by a GID that either maps to a different group in /etc/group or to no group at all.
The consequences of such wrong ownership can vary: in the most benign case this is just a cosmetic issue in 'ls' output. The case that is actually reported by users is when the service fails to start because it cannot access some file. But then there's also some silent group of cases where the wrong ownership creates a security vulnerability or an information leak.
Jędrzejewski-Szmek noted that the way that users and groups are created changed in Fedora 42. Instead of packages having scripts (referred to as "scriptlets") that call utilities like useradd and groupadd, the package should have a sysusers.d file which is processed by systemd-sysusers. The change proposal explains this in more detail. However, despite the change in mechanism in assigning users and groups, it does not have any impact on UID/GID drift. Users and groups are still assigned dynamically out of the same range of UID and GID numbers. There are currently 372 unique names for users or groups in the sysusers files for Fedora Rawhide, he said.
Fedora is the only major distribution using rpm-ostree and bootc to create image-based releases. The openSUSE Project uses Btrfs for its image-based versions such as Aeon and MicroOS. Ubuntu uses ubuntu-image to create its Ubuntu Core distribution. Those approaches do not seem to have have the UID/GID drift problem.
Proposed solutions
He suggests three possible solutions. The first approach, which he recommends, would be to simply not have any files owned by system users other than root. That would align with the "hermetic /usr" approach for immutable, image-based systems, as described by Lennart Poettering in a blog post from 2022:
In my model a hermetic OS is hence comprehensively defined within /usr/: combine the /usr/ tree with an empty, otherwise unpopulated root file system, and it will boot up successfully, automatically adding the strictly necessary files, and resources that are necessary to boot up.
Another approach would be to keep the files unowned—actually,
owned by the root user or group—as part of the package payload
but change ownership after installation by specifying the desired
ownership in tempfiles.d
configuration files. That avoids incorrect UID/GID assignments, but at
the cost of having to create and maintain the necessary tmpfiles.d
configuration. Jędrzejewski-Szmek noted that it can be
implemented easily, but "it's actually quite a bit of work and
bother to do correctly
".
The third option he proposed is soft-static
assignment of UIDs and GIDs. That requires packagers to file a
ticket with Fedora's packaging committee and justify the need for a
static assignment. In addition to the bureaucracy and additional work
for packagers, it would also create potential conflicts with
third-party packages. Jędrzejewski-Szmek mused that the project
could consider bigger changes, such as not using numeric UIDs/GIDs for
bootc images and only assigning those during installation. But he
offered no way to achieve that. He said that combining the first two approaches,
"minimizing the number of 'owned files' and judiciously using
tmpfiles.d to adjust ownership at runtime
", would be a relatively
painless option and called for comments or other ideas.
The creator of rpm-ostree and bootc, Colin Walters, replied
that there was a proposal for
bootc to automatically reset file ownership in the event of UID/GID
drift that he hoped to put on the near-term roadmap. He said that it
would take bootc a bit away from being image-based "but only
slightly and would greatly help the problem
".
Neal Gompa did not like the idea of ownerless files or having to fiddle with tmpfiles to assign ownership. He complained that the project had just introduced support for sysusers to make things easier for packagers, and now it looked like things could become more complex than they were before.
I don't think that will fly, at least it doesn't with me.
Having packages create a user, pre-own directories and files, and install them with that ownership is a rather common pattern that is essentially unavoidable. I do not see a scalable solution to work around that.
Jędrzejewski-Szmek said that he used to think that way as well, but he had found other reasons to avoid owned files, such as using bootc to create system images as an unprivileged user. He also said that there were maybe one-hundred packages that needed to be changed, and that they could be changed if ownerless was the best approach.
At the moment, the project is still mulling its options, and there has not been an abundance of ideas or objections offered to date. UID/GID drift is a real, if infrequent, problem. I've been using various image-based Fedora versions off and on for more than a decade, and haven't run into it on any of my systems. But a problem doesn't need to be common to be worth solving.
It will be interesting to see what solution Fedora develops—and what new problems the solutions create. Sometimes one gets the impression that every time a problem is solved in distributing and installing open-source software, two more spring up hydra-like in its wake.
Posted Apr 23, 2025 17:17 UTC (Wed)
by zyga (subscriber, #81533)
[Link]
The notes are from 2017 but a lot of the observations remain valid to this day.
Posted Apr 23, 2025 17:25 UTC (Wed)
by rcampos (subscriber, #59737)
[Link] (3 responses)
Does anyone know?
Posted Apr 23, 2025 18:36 UTC (Wed)
by zyga (subscriber, #81533)
[Link]
Ordinary application squashfs cannot use users or groups ot eh we than root. This is sometimes problematic and requires patching applications to compensate. The reason for this is that one cannot allocate such IDs at runtime and some mimetic value must be baked into the signed disk image.
The second problem is that the same base squashfs snap is used on Ubuntu Core distribution, classic Ubuntu, Debian, Fedora and any other. In all the cases the user and group database is provided by the host and, outside of core itself, may not line up exactly as one would want.
Posted Apr 24, 2025 6:24 UTC (Thu)
by danieldk (subscriber, #27876)
[Link]
Disclaimer: I don't use SUSE, this is what I found based on the documentation.
Posted Apr 25, 2025 8:00 UTC (Fri)
by kukuk (subscriber, #39885)
[Link]
SUSE does not ship /etc/passwd or /etc/group but generate that at install time, we do that at least for MicroOS and derivatives, and add new users via sysusers.d at update time.
The /etc directory of most Linux distributions contains four kind of files:
If you update that with a package manager, the package manager will try to merge that for you. If you have an image based system, that's not possible. That's why I don't understand why some image based distributions could come to the idea to ship /etc with their image. It's a big mess if you update the image.
The mid- to long term plan for SUSE is to have a hermetic-usr system. For many core package we have that already, but it's still a long way to go, especially as many upstream projects are not interested in this and reject patches.
Posted Apr 23, 2025 18:23 UTC (Wed)
by dbnichol (subscriber, #39622)
[Link] (1 responses)
We really would have preferred to make this semi-dynamic using systemd-sysusers. What I thought was the only reasonable solution was to finalize the passwd and group files during the ostree build process. I think this is basically the soft-static proposal + sysusers but without negotiations. Essentially:
1. After the ostree is completely assembled, parse /etc/passwd and /etc/group to find any users and groups defined by the build.
At runtime systemd-sysusers fills in the users and groups, and some of them will have fixed IDs per the above process. This is idealistic, though, as it does nothing to address systems that have already allocated a user or group and the existing ID doesn't match the fixed ID in the sysusers entry. In other words, the drift discussed here. Also, I never actually got around to implementing it.
Posted Apr 24, 2025 14:30 UTC (Thu)
by MrWim (subscriber, #47432)
[Link]
You'd map the ostree's UIDs and GIDs to usernames/group names according to /etc/passwd and /etc/group inside the ostree tree, and then map back again according to the host's /etc/passwd and /etc/group.
Obviously it's still better to avoid that complexity if you can.
Posted Apr 24, 2025 1:04 UTC (Thu)
by geofft (subscriber, #59789)
[Link] (20 responses)
Specifically, it seems like the most underlying problem is that filesystems store UIDs and GIDs, not user and group names (or some other larger namespace that more precisely identifies the semantics of ownership; you could imagine UUIDs or URIs or something even wackier, as long as it's not small integers from a range of a couple hundred). If you could store the names directly the problem wouldn't arise.
So, you store files with numbers, as you have to, but at the root of the filesystem in question (the /etc or /var partition), make a file that records what the meaning of the UIDs and GIDs used on this filesystem are. This could be literally in the format of the passwd and group files, stored as /.fspasswd and /.fsgroup or something. (These files themselves should be owned by, say, UID/GID 0, along with a convention that 0 always maps to root and is never remapped, since presumably root is not going to drift.)
Then, when you mount the filesystem, first mount it in a temporary location. Read in the /mnt/.fspasswd and /mnt/.fsgroup files, and compare them with /etc/passwd and /etc/group (or more precisely NSS output) on the running system. If there are discrepancies, construct an ID-mapping to fix them, such as "gid:999:998:1" for the example in the article (if I have the direction right). If there are any relevant accounts (probably just accounts managed by sysusers) that are not present in the .fspasswd and .fsgroup files, add them. Then create an idmapped mount in the right place and unmount the temporary mount.
This way, the filesystem stores, inside itself, the actual meaning of all the UIDs and GIDs in use on that filesystem, so in effect you can treat the filesystem as if it stored files with user and group names instead of numeric IDs. This is compatible with other approaches to try to address drift, in that obviously things are simpler if there is no idmap, but it does address drift when it happens. Would this be workable for Fedora?
Posted Apr 24, 2025 2:18 UTC (Thu)
by champtar (subscriber, #128673)
[Link]
Make me think of this bug https://issues.redhat.com/browse/RHEL-72572 where systemd-sysusers creates root group (in the initramfs) with GID 999 :)
Posted Apr 24, 2025 4:08 UTC (Thu)
by NYKevin (subscriber, #129325)
[Link] (11 responses)
In the "old world" (fs stores numbers, names are mostly decorative), this either causes a minor cosmetic issue where ls output and the like still uses the old name, or else the package manager updates /etc/passwd with the new name and everything Just Works. As such, developers are likely to assume that this is a reasonably safe thing to do, even if a package has already been widely deployed and regularly owns a ton of files.
In the "new world" (fs stores both numbers and a name-number mapping), this becomes a bit of a mess. Since we're recreating the image from scratch, we're going to end up with the new names in /etc/passwd, whether we want them or not. The old names will still be in the filesystem root mapping, because the whole point of said mapping is to remain stable even when the image is rebuilt. It's a question of whether your UID remapping logic is smart enough to do an automated migration at the next opportunity, or if it just fails to map the old names to the new UIDs.
Posted Apr 24, 2025 19:32 UTC (Thu)
by geofft (subscriber, #59789)
[Link] (1 responses)
I guess one problem is that sysusers files do not even have a spot for extensible metadata.
Posted Apr 28, 2025 2:27 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link]
* Whether the data is readily available at the appropriate time (i.e. everything is mounted or about to be mounted and we can actually do a UID/username migration right now).
Posted Apr 24, 2025 20:08 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link]
Posted Apr 25, 2025 4:51 UTC (Fri)
by DemiMarie (subscriber, #164188)
[Link] (7 responses)
Posted Apr 25, 2025 11:40 UTC (Fri)
by SLi (subscriber, #53131)
[Link] (2 responses)
Posted Apr 25, 2025 14:18 UTC (Fri)
by intelfx (subscriber, #130118)
[Link] (1 responses)
I was practically screaming "UUIDs" (or, hell, SIDs) at the screen the entire time I was reading the article and the comments. But that would be too sane. So we're stuck with hacks on top of hacks on top of hacks on top of hacks.
Posted Apr 25, 2025 17:09 UTC (Fri)
by DemiMarie (subscriber, #164188)
[Link]
Posted Apr 25, 2025 17:41 UTC (Fri)
by geofft (subscriber, #59789)
[Link] (3 responses)
I was also surprised to find that, even with SIDs, Active Directory accounts also have a GUID, and SIDs can change but GUIDs are permanent. There is an "SIDHistory" field in the AD object that lists all previous SIDs of the account. https://learn.microsoft.com/en-us/windows-server/identity...
I was amused but not actually surprised to find that NTFS-3G Linux driver supports mapping SIDs to UIDs/GIDs via a text file in a hidden directory in the filesystem root. :) https://github.com/tuxera/ntfs-3g/wiki/File-Ownership-and...
Posted Apr 28, 2025 2:16 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link] (2 responses)
To the best of my understanding, the Microsoft answer to that problem is "don't pervasively hang permissions on every single object in the entire filesystem." Instead, permissions are inherited from the parent directory by default (and from its parent recursively, all the way up to the root or until you hit a directory with non-inherited permissions). So you don't have to do N checks for a path consisting of N components - you just do one or two checks for the high-level directories that actually have permissions on them, and everything else is cheap lookups of cached ACLs.
> I was also surprised to find that, even with SIDs, Active Directory accounts also have a GUID, and SIDs can change but GUIDs are permanent.
As a rule of thumb, in the Microsoft universe, everything has a GUID, or else it must not be very important.
Posted May 2, 2025 17:19 UTC (Fri)
by Conan_Kudo (subscriber, #103240)
[Link] (1 responses)
Posted May 3, 2025 8:44 UTC (Sat)
by Wol (subscriber, #4433)
[Link]
The directory system is independent (which is what gives us hard links). So the mechanism that works everywhere else - "traverse back up the file tree until you hit an ACL" - doesn't work in Posix.
Cheers,
Posted Apr 24, 2025 10:26 UTC (Thu)
by taladar (subscriber, #68407)
[Link] (1 responses)
Posted Apr 24, 2025 19:42 UTC (Thu)
by geofft (subscriber, #59789)
[Link]
You probably want to mount removable media nosuid and perhaps noexec and nodev unless explicitly told otherwise by the user, regardless of whether you are remapping UIDs on the filesystem.
Posted Apr 24, 2025 18:20 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (1 responses)
At which point, any sane dba will have his head in his hands, weeping and crying "when will they learn?!". NAMES ARE A DAFT IDEA!
Names are not unique. Names change. Names screw you over. Real case study, a friend of mine wrote a medical database keyed on surname ... at which point I got involved. And gently pointed out that a lady doctor getting married would corrupt the database. A new doctor joining with the same surname would corrupt the database. Etc etc.
It's normal practice to reserve some space for system uids/gids - under 500, or under 1000. The obvious solution (yes I know, "simple, obvious, and wrong") would be to reserve the top - lets say 200 - of those entries and pre-allocate them. Okay, we run the risk of running out of pre-allocations, but at least that's a problem we can catch at build time, not when an image goes live and screws the user over.
I'm all for real-world primary keys, I'd much rather use an SSN or NI number or VIN number than some random uid that changes between databases, but as soon as we get into this problem space, we need a primary key that uniquely identifies the item of interest. And names AREN'T IT. Sendmail, anyone?
Cheers,
Posted Apr 24, 2025 19:24 UTC (Thu)
by geofft (subscriber, #59789)
[Link]
(I actually wonder what a good solution to this would be in the general case. In practice, if you have an employee who has changed their name and whose username represents their old name, the most reliable solution seems to be to give them an entirely new user account, grant them continued access to the old one, and ask them to migrate things as they go. And heavens help you if you are sysadminning a merger between two companies with traditional UNIX account systems like LDAP; username conflicts are just as much of a pain as user ID conflicts.)
But yes, that's why I mentioned UUIDs or URIs in the sentence you quoted, which avoid the problems you describe. You need a way to associate the UID/GID on the filesystem with a persistent, unambiguous identifier for an account; you can then figure out what username and UID that account has on the current running system, both of which could have changed since the files were written, and you can handle them both correctly.
My guess is that the entire reason Fedora is having this discussion is that there are not, in fact, enough UIDs in the <1000 range for every use case in Fedora for a system user to pre-allocate one. This is the "soft-static assignment" approach proposed in the article; the link takes you to a sentence that explains that a ticket is needed precisely because the available space is limited.
Posted Apr 24, 2025 20:04 UTC (Thu)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
Posted Apr 24, 2025 21:32 UTC (Thu)
by geofft (subscriber, #59789)
[Link] (1 responses)
I guess you could teach the kernel to create these xattrs. Maybe an LSM can do it? Or maybe you could make a FUSE filesystem that does it, though overhead is likely unacceptably high.
Posted Apr 25, 2025 6:50 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
Posted Apr 24, 2025 2:24 UTC (Thu)
by champtar (subscriber, #128673)
[Link]
> systemd-sysusers
Posted Apr 24, 2025 6:44 UTC (Thu)
by spacefrogg (subscriber, #119608)
[Link] (3 responses)
Posted Apr 24, 2025 13:50 UTC (Thu)
by mattdm (subscriber, #18)
[Link] (1 responses)
This is not a viable approach for a general-purpose distribution like Fedora Linux.
Posted Apr 25, 2025 5:47 UTC (Fri)
by rgb (subscriber, #57129)
[Link]
1) spec matches /usr; missmatch between /usr and /home
Posted Apr 25, 2025 5:29 UTC (Fri)
by rgb (subscriber, #57129)
[Link]
Do you have an example when this might happen? I can only think of a package that switches from dynamic to static user/group creation on update, but that should be very rare?
Posted Apr 25, 2025 13:27 UTC (Fri)
by alexl (subscriber, #19068)
[Link] (1 responses)
This is like a micro-version of GUIDs, but without changing the format.
Posted Apr 26, 2025 12:52 UTC (Sat)
by smcv (subscriber, #53363)
[Link]
I think Debian is fairly typical here: it puts dynamically-allocated system users (the ones being discussed here) in the range 100-999, so there are only 900 available. (As per https://www.debian.org/doc/debian-policy/ch-opersys.html#... 90% of the 16-bit range is made available to sysadmins for local allocation of humans' user accounts.)
Posted Apr 28, 2025 6:02 UTC (Mon)
by pothos (subscriber, #116075)
[Link]
Ubuntu Core explored this topic as well
what does suse/Ubuntu do different?
what does suse/Ubuntu do different?
what does suse/Ubuntu do different?
what does suse/Ubuntu do different?
* host specific files generated at install time or first boot, like /etc/machine-id, /etc/passwd, /etc/group, ...
* distribution provided configuration files, which admins and distributors can change
* admin made changes
* stuff which does not belong to /etc, but where upstream projects are reluctant to fix that. Why do we need shell scripts for grub in /etc?
But that's the only clean solution for the current /etc mess in update case, even for traditional distributions.
Tough problem
2. Parse all the conf files in /usr/lib/sysusers.d to find any users and groups defined by that method.
3. Verify that any non-root user or group has an accompanying sysusers entry.
4. Walk the entire assembled tree looking for paths that are not owned by root. Note that /var is deleted at this point, which is one of the main sources of non-root owned files. Verify that any non-root owned path has a fixed ID in its sysusers entry.
5. Truncate the user/group DBs (/etc/{passwd,shadow,group,gshadow}) so they just contain the root entry. We have verified that any other required user or group will be created by systemd-sysusers at runtime.
Tough problem
idmapped mounts
idmapped mounts
idmapped mounts
idmapped mounts
idmapped mounts
* Whether the tooling is smart enough to do it automatically, or if that functionality still needs to be built out.
idmapped mounts
idmapped mounts
idmapped mounts
idmapped mounts
User UUIDs
idmapped mounts
idmapped mounts
Yep. And personally, I think it's a smarter model.
idmapped mounts
idmapped mounts
Wol
idmapped mounts
idmapped mounts
idmapped mounts
Wol
idmapped mounts
idmapped mounts
idmapped mounts
idmapped mounts
Static passwd/group + checks
so we ended up having static passwd/group + the following in postprocess-script:
> # fail if extra user / group have been created
> tail -n1 /etc/passwd | grep -q nobody || exit 42
> tail -n1 /etc/group | grep -q nobody || exit 42
NixOS also deals with that
NixOS also deals with that
NixOS also deals with that
2) /usr matches /home; missmatch between /usr and spec
NixOS also deals with that
Allocate uids based on username?
Allocate uids based on username?
systemd-userdb
Ideally the package is the source of this but one can also generate it at image build time.