LWN: Comments on "Addressing UID/GID drift in rpm-ostree and bootc"

idmapped mounts

Wol — Sat, 03 May 2025 08:44:36 +0000

The problem is that - in the standard Unix filesystem model - we have a flat root-level namespace. All files are uniquely identified by their i-node.

The directory system is independent (which is what gives us hard links). So the mechanism that works everywhere else - "traverse back up the file tree until you hit an ACL" - doesn't work in Posix.

Cheers,
Wol

idmapped mounts

Conan_Kudo — Fri, 02 May 2025 17:19:17 +0000

Yep. And personally, I think it's a smarter model.

systemd-userdb

pothos — Mon, 28 Apr 2025 06:02:48 +0000

I think static assignment makes most sense for image based distros. With systemd-userdb a JSON record can be shipped as part of /usr and gets integrated with the system throught nss.
Ideally the package is the source of this but one can also generate it at image build time.

idmapped mounts

NYKevin — Mon, 28 Apr 2025 02:27:41 +0000

I'm sure the data needed to do this exists in some form. My question was mostly centered on:

* Whether the data is readily available at the appropriate time (i.e. everything is mounted or about to be mounted and we can actually do a UID/username migration right now).
* Whether the tooling is smart enough to do it automatically, or if that functionality still needs to be built out.

idmapped mounts

NYKevin — Mon, 28 Apr 2025 02:16:32 +0000

> I suppose the NT kernel is designed sufficiently differently from UNIX that they don't have the same model of numeric user IDs and access checks are performed with tokens/handles, so you can put the SID behind a pointer and deal with the indirection and memory allocation rarely. But it feels like this has to be high-overhead for file access checks on NTFS files, which store ownership by SID.

To the best of my understanding, the Microsoft answer to that problem is "don't pervasively hang permissions on every single object in the entire filesystem." Instead, permissions are inherited from the parent directory by default (and from its parent recursively, all the way up to the root or until you hit a directory with non-inherited permissions). So you don't have to do N checks for a path consisting of N components - you just do one or two checks for the high-level directories that actually have permissions on them, and everything else is cheap lookups of cached ACLs.

> I was also surprised to find that, even with SIDs, Active Directory accounts also have a GUID, and SIDs can change but GUIDs are permanent.

As a rule of thumb, in the Microsoft universe, everything has a GUID, or else it must not be very important.

Allocate uids based on username?

smcv — Sat, 26 Apr 2025 12:52:16 +0000

If you want your OS to be runnable in a rootless container (like the typical use of Podman) then you can only rely on having a 16-bit range available (for everyone - system users as well as humans' user accounts), so there aren't really enough uids available for this sort of sparse allocation to avoid collisions reliably.

I think Debian is fairly typical here: it puts dynamically-allocated system users (the ones being discussed here) in the range 100-999, so there are only 900 available. (As per https://www.debian.org/doc/debian-policy/ch-opersys.html#... 90% of the 16-bit range is made available to sysadmins for local allocation of humans' user accounts.)

idmapped mounts

geofft — Fri, 25 Apr 2025 17:41:21 +0000

I was surprised to find that Windows SIDs are arbitrary-length. I suppose the NT kernel is designed sufficiently differently from UNIX that they don't have the same model of numeric user IDs and access checks are performed with tokens/handles, so you can put the SID behind a pointer and deal with the indirection and memory allocation rarely. But it feels like this has to be high-overhead for file access checks on NTFS files, which store ownership by SID.

I was also surprised to find that, even with SIDs, Active Directory accounts also have a GUID, and SIDs can change but GUIDs are permanent. There is an "SIDHistory" field in the AD object that lists all previous SIDs of the account. https://learn.microsoft.com/en-us/windows-server/identity...

I was amused but not actually surprised to find that NTFS-3G Linux driver supports mapping SIDs to UIDs/GIDs via a text file in a hidden directory in the filesystem root. :) https://github.com/tuxera/ntfs-3g/wiki/File-Ownership-and...

User UUIDs

DemiMarie — Fri, 25 Apr 2025 17:09:19 +0000

Those would allow idmapped users to work in a decent way.

idmapped mounts

intelfx — Fri, 25 Apr 2025 14:18:32 +0000

Yup.

I was practically screaming "UUIDs" (or, hell, SIDs) at the screen the entire time I was reading the article and the comments. But that would be too sane. So we're stuck with hacks on top of hacks on top of hacks on top of hacks.

Allocate uids based on username?

alexl — Fri, 25 Apr 2025 13:27:02 +0000

Can we maybe allocate dynamic uid:s not sequentially, but based on the user name. Pick a large range, say 24bits (with the upper 8 bits as a tag), and then use "hash(username) & 0xffffff" to allocate the id (handling conflicts like a hash table). This would make it quite unlikely to have conflicts.

This is like a micro-version of GUIDs, but without changing the format.

idmapped mounts

SLi — Fri, 25 Apr 2025 11:40:29 +0000

Yeah, it would make sense to have a single thing that nobody has any reason to ever change, an "identity", which is a UUID. All other things like the username should be just properties attached to this. Perhaps even the transient things like uid.

what does suse/Ubuntu do different?

kukuk — Fri, 25 Apr 2025 08:00:11 +0000

> I wonder what Ubuntu and suse do different in their immutable versions to avoid this issue.

SUSE does not ship /etc/passwd or /etc/group but generate that at install time, we do that at least for MicroOS and derivatives, and add new users via sysusers.d at update time.

The /etc directory of most Linux distributions contains four kind of files:
* host specific files generated at install time or first boot, like /etc/machine-id, /etc/passwd, /etc/group, ...
* distribution provided configuration files, which admins and distributors can change
* admin made changes
* stuff which does not belong to /etc, but where upstream projects are reluctant to fix that. Why do we need shell scripts for grub in /etc?

If you update that with a package manager, the package manager will try to merge that for you. If you have an image based system, that's not possible. That's why I don't understand why some image based distributions could come to the idea to ship /etc with their image. It's a big mess if you update the image.

The mid- to long term plan for SUSE is to have a hermetic-usr system. For many core package we have that already, but it's still a long way to go, especially as many upstream projects are not interested in this and reject patches.
But that's the only clean solution for the current /etc mess in update case, even for traditional distributions.

idmapped mounts

mathstuf — Fri, 25 Apr 2025 06:50:13 +0000

Ah right… Perhaps it could be a periodic process that goes and adds the xattr to any files missing them in `/etc` and `/var` (and some tool to detect missed xattrs on startup)?

NixOS also deals with that

rgb — Fri, 25 Apr 2025 05:47:07 +0000

But this UID/GID mismatch is exactly what you want? It is just a missmatch between the current system and the package spec. There are only two options (if you don't want to re-assign every file in /home etc):

1) spec matches /usr; missmatch between /usr and /home
2) /usr matches /home; missmatch between /usr and spec

NixOS also deals with that

rgb — Fri, 25 Apr 2025 05:29:45 +0000

> even if the configuration demands it

Do you have an example when this might happen? I can only think of a package that switches from dynamic to static user/group creation on update, but that should be very rare?

idmapped mounts

DemiMarie — Fri, 25 Apr 2025 04:51:01 +0000

I think the best solution is to use something like GUIDs or Windows’s SIDs.

idmapped mounts

geofft — Thu, 24 Apr 2025 21:32:31 +0000

The complication with that approach is you need something to store the xattr on every file creation and update it on every chown; your average program that creates a config file or a state file won't have logic to do this, so you'll end up with a file with a non-root UID and no xattr explaining what that UID means. (Creating the file in advance isn't always an option - consider things like log rotation where the program renames the old file and creates a new one.) ID-mapping is a feature that's already in the kernel that takes place in the path between the program creating a file and the bits ending up on disk, so if you set up the ID map, then any unmodified userspace program using the mount gets the right behavior behind the scenes.

I guess you could teach the kernel to create these xattrs. Maybe an LSM can do it? Or maybe you could make a FUSE filesystem that does it, though overhead is likely unacceptably high.

idmapped mounts

mathstuf — Thu, 24 Apr 2025 20:08:11 +0000

I may have been playing with fire at the time, but you can have two users that share a uid in `/etc/passwd`. I did it to shove Steam into a different `$HOME` before flatpak was a thing to avoid it dumping things into there (IIRC, trying to use symlinks to move where it actually did stuff broke things at the time).

idmapped mounts

mathstuf — Thu, 24 Apr 2025 20:04:55 +0000

Alternatively, could the intended uname/gname be stored as xattrs? Then scan `/etc` for non-root files, check that the names and uid/gid match and if not, look up the new ids and update them directly. Would also work for rollbacks, no?

idmapped mounts

geofft — Thu, 24 Apr 2025 19:42:52 +0000

I was originally going to say that this mechanism should only be used for trusted filesystems like /etc or /var (where being able to make a file with name and contents of your choice lets you pwn the system, even without particular control of that file's ownership). But thinking about this more, I do not think that remapping UIDs increases the danger of untrusted filesystems. An untrusted filesystem can already make whatever claims about UIDs and file permissions that it wants, i.e., it can contain setuid binaries for any possible UID. Giving it the ability to target a specific user account by name isn't actually giving it more power. It's maybe making an attack logistically easier, but notably it doesn't make things easier for attacking root, which is always UID 0, or any other accounts with fixed UIDs. (An attacker can also just have a directory with setuid binaries for every possible UID, and a launcher that looks up the account it wants to target and picks the right setuid binary.)

You probably want to mount removable media nosuid and perhaps noexec and nodev unless explicitly told otherwise by the user, regardless of whether you are remapping UIDs on the filesystem.

idmapped mounts

geofft — Thu, 24 Apr 2025 19:32:41 +0000

Is the package able to retain the former names of the system users? If you have that available in some form of metadata, then you can easily update the .fspasswd file to the current canonical name as part of the process of matching up its accounts to the current system's accounts. I would assume that, for non-image-based systems, the package's pre/post scriptlets need to rename the user anyway, so that information is already included in the package in some form.

I guess one problem is that sysusers files do not even have a spot for extensible metadata.

idmapped mounts

geofft — Thu, 24 Apr 2025 19:24:50 +0000

Well, these are usernames, not names in the human sense. We already have the problem that usernames are hard to change. Plenty of other things on the system will reference accounts by username instead of by UID - home directory, sudo rules, /etc/shadow, PAM, etc. etc. Correctly and fully changing the username of an account on an existing traditional-UNIX system is a problem that's as hard as changing the UID.

(I actually wonder what a good solution to this would be in the general case. In practice, if you have an employee who has changed their name and whose username represents their old name, the most reliable solution seems to be to give them an entirely new user account, grant them continued access to the old one, and ask them to migrate things as they go. And heavens help you if you are sysadminning a merger between two companies with traditional UNIX account systems like LDAP; username conflicts are just as much of a pain as user ID conflicts.)

But yes, that's why I mentioned UUIDs or URIs in the sentence you quoted, which avoid the problems you describe. You need a way to associate the UID/GID on the filesystem with a persistent, unambiguous identifier for an account; you can then figure out what username and UID that account has on the current running system, both of which could have changed since the files were written, and you can handle them both correctly.

My guess is that the entire reason Fedora is having this discussion is that there are not, in fact, enough UIDs in the <1000 range for every use case in Fedora for a system user to pre-allocate one. This is the "soft-static assignment" approach proposed in the article; the link takes you to a sentence that explains that a ticket is needed precisely because the available space is limited.

idmapped mounts

Wol — Thu, 24 Apr 2025 18:20:44 +0000

> Specifically, it seems like the most underlying problem is that filesystems store UIDs and GIDs, not user and group names (or some other larger namespace that more precisely identifies the semantics of ownership; you could imagine UUIDs or URIs or something even wackier, as long as it's not small integers from a range of a couple hundred). If you could store the names directly the problem wouldn't arise.

At which point, any sane dba will have his head in his hands, weeping and crying "when will they learn?!". NAMES ARE A DAFT IDEA!

Names are not unique. Names change. Names screw you over. Real case study, a friend of mine wrote a medical database keyed on surname ... at which point I got involved. And gently pointed out that a lady doctor getting married would corrupt the database. A new doctor joining with the same surname would corrupt the database. Etc etc.

It's normal practice to reserve some space for system uids/gids - under 500, or under 1000. The obvious solution (yes I know, "simple, obvious, and wrong") would be to reserve the top - lets say 200 - of those entries and pre-allocate them. Okay, we run the risk of running out of pre-allocations, but at least that's a problem we can catch at build time, not when an image goes live and screws the user over.

I'm all for real-world primary keys, I'd much rather use an SSN or NI number or VIN number than some random uid that changes between databases, but as soon as we get into this problem space, we need a primary key that uniquely identifies the item of interest. And names AREN'T IT. Sendmail, anyone?

Cheers,
Wol

Tough problem

MrWim — Thu, 24 Apr 2025 14:30:57 +0000

It probably wouldn't be that hard to modify ostree's checkout logic to map UIDs and GIDs when doing a checkout. For /usr you'd still require everything to be root:root, which would be necessary for the hard-linking it does, but for /etc it has to copy anyway, so there wouldn't be any issues.

You'd map the ostree's UIDs and GIDs to usernames/group names according to /etc/passwd and /etc/group inside the ostree tree, and then map back again according to the host's /etc/passwd and /etc/group.

Obviously it's still better to avoid that complexity if you can.

NixOS also deals with that

mattdm — Thu, 24 Apr 2025 13:50:36 +0000

> On each system update the admin gets notified about the UID/GID mismatch such that they can clean up.

This is not a viable approach for a general-purpose distribution like Fedora Linux.

idmapped mounts

taladar — Thu, 24 Apr 2025 10:26:46 +0000

It might be necessary to consider the security implications here, especially in combination with files getting privileges related to their owner/group (setuid is one example but there might be others). At the very least you probably want to avoid mounting removable media with this type of mapping and without the noexec mount option at the same time.

NixOS also deals with that

spacefrogg — Thu, 24 Apr 2025 06:44:20 +0000

Being an image-based distribution in that particular regard, NixOS also has to take care of UID drift. There, the OS maintains a passwd/groups database in /var, such that a UID/GID once created is not changed later in a particular installation, even if the configuration demands it. On each system update the admin gets notified about the UID/GID mismatch such that they can clean up.

what does suse/Ubuntu do different?

danieldk — Thu, 24 Apr 2025 06:24:26 +0000

OSTree/bootc build an image. Every update is basically a fresh image with the packages installed from scratch into the image, which leads to the ordering issue. As far as I understand with the SUSE atomic versions they use traditional system updates combined with btrfs snapshots. So when an update is done, a new snapshot is made, the snapshot is mounted read-write and various directories are mounted on top of it. Then the read-write snapshot is updated using Zypper as a normal, mutable SUSE system. If the update failed, the snapshot is discarded. If the update was successful, the snapshot is unmounted and then mounted as the read-only root during the next boot. It doesn't have the UID/GID issue, because aside from the tricks to make root read-only, it updates in the same way is a non-atomic Linux system.

Disclaimer: I don't use SUSE, this is what I found based on the documentation.

idmapped mounts

NYKevin — Thu, 24 Apr 2025 04:08:05 +0000

Overall, that sounds like a perfectly reasonable approach, but I can think of one minor-but-possibly-annoying corner case: What happens if a package renames its system user?

In the "old world" (fs stores numbers, names are mostly decorative), this either causes a minor cosmetic issue where ls output and the like still uses the old name, or else the package manager updates /etc/passwd with the new name and everything Just Works. As such, developers are likely to assume that this is a reasonably safe thing to do, even if a package has already been widely deployed and regularly owns a ton of files.

In the "new world" (fs stores both numbers and a name-number mapping), this becomes a bit of a mess. Since we're recreating the image from scratch, we're going to end up with the new names in /etc/passwd, whether we want them or not. The old names will still be in the filesystem root mapping, because the whole point of said mapping is to remain stable even when the image is rebuilt. It's a question of whether your UID remapping logic is smart enough to do an automated migration at the next opportunity, or if it just fails to map the old names to the new UIDs.

Static passwd/group + checks

champtar — Thu, 24 Apr 2025 02:24:07 +0000

At work we use rpm-ostree but rarely overlay anything that we don't control,
so we ended up having static passwd/group + the following in postprocess-script:

> systemd-sysusers
> # fail if extra user / group have been created
> tail -n1 /etc/passwd | grep -q nobody || exit 42
> tail -n1 /etc/group | grep -q nobody || exit 42

idmapped mounts

champtar — Thu, 24 Apr 2025 02:18:00 +0000

> since presumably root is not going to drift.

Make me think of this bug https://issues.redhat.com/browse/RHEL-72572 where systemd-sysusers creates root group (in the initramfs) with GID 999 :)

idmapped mounts

geofft — Thu, 24 Apr 2025 01:04:17 +0000

It seems like ID-mapped mounts (https://lwn.net/Articles/896255/, https://github.com/brauner/mount-idmapped) would address this use case, at the cost of a tiny bit of kernel overhead.

Specifically, it seems like the most underlying problem is that filesystems store UIDs and GIDs, not user and group names (or some other larger namespace that more precisely identifies the semantics of ownership; you could imagine UUIDs or URIs or something even wackier, as long as it's not small integers from a range of a couple hundred). If you could store the names directly the problem wouldn't arise.

So, you store files with numbers, as you have to, but at the root of the filesystem in question (the /etc or /var partition), make a file that records what the meaning of the UIDs and GIDs used on this filesystem are. This could be literally in the format of the passwd and group files, stored as /.fspasswd and /.fsgroup or something. (These files themselves should be owned by, say, UID/GID 0, along with a convention that 0 always maps to root and is never remapped, since presumably root is not going to drift.)

Then, when you mount the filesystem, first mount it in a temporary location. Read in the /mnt/.fspasswd and /mnt/.fsgroup files, and compare them with /etc/passwd and /etc/group (or more precisely NSS output) on the running system. If there are discrepancies, construct an ID-mapping to fix them, such as "gid:999:998:1" for the example in the article (if I have the direction right). If there are any relevant accounts (probably just accounts managed by sysusers) that are not present in the .fspasswd and .fsgroup files, add them. Then create an idmapped mount in the right place and unmount the temporary mount.

This way, the filesystem stores, inside itself, the actual meaning of all the UIDs and GIDs in use on that filesystem, so in effect you can treat the filesystem as if it stored files with user and group names instead of numeric IDs. This is compatible with other approaches to try to address drift, in that obviously things are simpler if there is no idmap, but it does address drift when it happens. Would this be workable for Fedora?

what does suse/Ubuntu do different?

zyga — Wed, 23 Apr 2025 18:36:16 +0000

I cannot speak for SUSE but Ubuntu Core experiences this problem in two different ways:

Ordinary application squashfs cannot use users or groups ot eh we than root. This is sometimes problematic and requires patching applications to compensate. The reason for this is that one cannot allocate such IDs at runtime and some mimetic value must be baked into the signed disk image.

The second problem is that the same base squashfs snap is used on Ubuntu Core distribution, classic Ubuntu, Debian, Fedora and any other. In all the cases the user and group database is provided by the host and, outside of core itself, may not line up exactly as one would want.

Tough problem

dbnichol — Wed, 23 Apr 2025 18:23:51 +0000

Endless OS is based on ostree like this. We've avoided this there by nss-altfiles with static passwd and group files in /lib containing every possible user and group with fixed IDs. It's ugly, all the developers hate it, and it only works because Endless doesn't allow system overlays like rpm-ostree does. Our ostree build process is also a little simplified compared to rpm-ostree in that we install Debian packages in a root, let them run the maintainer scripts as they want, and then do any necessary cleanup before the commit.

We really would have preferred to make this semi-dynamic using systemd-sysusers. What I thought was the only reasonable solution was to finalize the passwd and group files during the ostree build process. I think this is basically the soft-static proposal + sysusers but without negotiations. Essentially:

1. After the ostree is completely assembled, parse /etc/passwd and /etc/group to find any users and groups defined by the build.
2. Parse all the conf files in /usr/lib/sysusers.d to find any users and groups defined by that method.
3. Verify that any non-root user or group has an accompanying sysusers entry.
4. Walk the entire assembled tree looking for paths that are not owned by root. Note that /var is deleted at this point, which is one of the main sources of non-root owned files. Verify that any non-root owned path has a fixed ID in its sysusers entry.
5. Truncate the user/group DBs (/etc/{passwd,shadow,group,gshadow}) so they just contain the root entry. We have verified that any other required user or group will be created by systemd-sysusers at runtime.

At runtime systemd-sysusers fills in the users and groups, and some of them will have fixed IDs per the above process. This is idealistic, though, as it does nothing to address systems that have already allocated a user or group and the existing ID doesn't match the fixed ID in the sysusers entry. In other words, the drift discussed here. Also, I never actually got around to implementing it.

what does suse/Ubuntu do different?

rcampos — Wed, 23 Apr 2025 17:25:06 +0000

I wonder what Ubuntu and suse do different in their immutable versions to avoid this issue.

Does anyone know?

Ubuntu Core explored this topic as well

zyga — Wed, 23 Apr 2025 17:17:18 +0000

Ubuntu Core, a special flavour of Ubuntu based on immutable squashfs filesystems explored this as well. Quite a bit of discussion and analysis is still on the Snapcraft forum, in this thread: https://forum.snapcraft.io/t/multiple-users-and-groups-in...

The notes are from 2017 but a lot of the observations remain valid to this day.