LWN: Comments on "An operation for filesystem tucking"

An operation for filesystem tucking

smurf — Sun, 09 Apr 2023 10:46:02 +0000

If you flip that symlink while a program is starting up it'll get some libraries from A and others from B, which may cause it to crash. So in that sense it's definitely not atomic. The same problem applies to a mount-on-top operation.

On the other hand if you have a tuck-mount you can do an atomic unmount afterwards, but only if there's no current user. That guarantees that you don't run into inconsistent libraries et al. even when you can't control exactly when a program (re)starts.

An operation for filesystem tucking

NYKevin — Sun, 09 Apr 2023 07:49:37 +0000

To be fair, the classic (pre-container) way to do this is to flip a symlink, and that really is perfectly atomic.

One wonders why userspace container systems don't just put a symlink inside of the filesystem image. You could have, for example, two mount points A and B, and a symlink to whichever mount point is currently in use. When you want to upgrade to a new version, you mount it on the other mount point, and then flip the symlink. Perhaps the container software doesn't want to dictate the use of such symlinks? But that seems like a rather... insubstantial reason to make this the kernel's problem, IMHO.

An operation for filesystem tucking

bbbush — Fri, 07 Apr 2023 18:25:31 +0000

in rpm-ostree one can replace a package with another. It sounds like the same is applied to filesystems, so that the overall status is good. When replacing the package, the status of the tree. When replacing the filesystems, the status of the tree as well? Then it does not matter if any file is open, because the operation is going to be followed by a "systemctl reboot"?

An operation for filesystem tucking

nim-nim — Tue, 04 Apr 2023 13:38:28 +0000

> I wonder, why not mount the new overlay on top of the old overlay, and then unmount the old overlay from beneath it, instead?

IMHO the article does not state clearly that local changes (for example /etc) are likely to exist in a RW layer mounted over the old overlay, so you still need to tuck under it. Nobody really succeeded in strict separation between RO and RW directories for complex setups, that‘s an ideal people are still chasing.

Also since I suppose unmounting the old overlay is faster than mounting the new one, that means you can wait for the new overlay to be mounted under the old one in all affected containers in the system, and them pop the old one in one go.

An operation for filesystem tucking

smurf — Tue, 04 Apr 2023 07:38:43 +0000

Of course it's still fragile, but it's less fragile than a mount-on-top with some way to unmount the lower-layered file system later.

An operation for filesystem tucking

stefanha — Mon, 03 Apr 2023 22:13:32 +0000

It's still not atomic. A new file dropped into a .d directory can easily cause problems since the old mount is still in place with the majority of the set of files.

This seems fragile to me, but then, in place updates are always tricky.

An operation for filesystem tucking

NYKevin — Mon, 03 Apr 2023 17:49:25 +0000

I think the simplest way to handle that is to make it userspace's problem: If you want to tuck a new overlay that overrides files in the base overlay, you first tuck a copy of the old overlay that also contains all of the files you are going to override. Any reasonable container system should be keeping track of what files are supposed to exist in each layer, and so it should be able to detect situations like this.

An operation for filesystem tucking

mgedmin — Mon, 03 Apr 2023 12:57:51 +0000

The file in the new overlay might be overriding an existing file in the base overlay that was not being overridden by the old overlay.

An operation for filesystem tucking

NYKevin — Sat, 01 Apr 2023 20:13:07 +0000

According to inotify(7): You get an IN_UNMOUNT event, followed by an IN_IGNORED (which indicates that the watch has been invalidated). You do not explicitly pass either of those events to inotify_add_watch(2) (the kernel just sends them to you automatically), so the only way to mess this up is to fail to read inotify(7) and miss a case.

An operation for filesystem tucking

NYKevin — Sat, 01 Apr 2023 20:03:30 +0000

IMHO this is less likely to cause problems, because it wouldn't be seen (by the processes inside the container) as an existing file getting abruptly replaced or deleted. While it is certainly possible for software to enumerate the contents of a directory and (maybe) get upset when a new file shows up unexpectedly, in practice this usually only happens at startup or on an explicit signal such as SIGHUP or SIGUSR1/2 (at least for server-like things that you would put in a container in the first place - obviously something such as GNOME is monitoring ~/Desktop all the time so that it can draw icons etc., but running GNOME in a container is probably not the target use case here).

An operation for filesystem tucking

smurf — Sat, 01 Apr 2023 12:56:11 +0000

With the "mount underneath" method you can simply unmount the top when there are no users. Thus you're guaranteed to have a consistent state at that point.

When mounting on top there probably are open files on the overlaid file system. You now have a mix of open old and unopened new content, which is a surefire recipe for eventual disaster.

An operation for filesystem tucking

Karellen — Sat, 01 Apr 2023 10:49:15 +0000

On the other hand, if the new overlay provides a file not in the old overlay, in the tuck scenario it could "peek through" before the old overlay is removed.

An operation for filesystem tucking

tchernobog — Sat, 01 Apr 2023 07:17:41 +0000

Does anybody know how this might tie in with the inotify mechanics? Is the inotify fd just invalidated upon unmounting of the topmost overlay, and the application has to reopen it?

An operation for filesystem tucking

epa — Sat, 01 Apr 2023 06:41:34 +0000

What if the old overlay contained a file for /etc/aliases but the new overlay does not because you decided, after all, not to override /etc/aliases but stay with the base system version.

An operation for filesystem tucking

roc — Fri, 31 Mar 2023 21:43:17 +0000

To me, mount semantics seem like a complete nightmare of overlapping, piled-up complexity. If I were to go looking for critical kernel bugs that's where I'd be looking.

An operation for filesystem tucking

NYKevin — Fri, 31 Mar 2023 21:14:59 +0000

The new overlay could contain a whiteout, if necessary. AFAICT, the real problem is that "lazy" umount(2) exists, but "lazy" mount(2) does not exist.

An operation for filesystem tucking

epa — Fri, 31 Mar 2023 20:59:09 +0000

I guess not quite, if the old overlay provided a file which is not present in the new overlay.

An operation for filesystem tucking

NYKevin — Fri, 31 Mar 2023 20:24:23 +0000

> For example, if the unmount of the old overlay occurs while rtld, the run-time dynamic linker, is loading a list of shared libraries into an address space, then it may be easy to trigger an incompatibility between libraries at the beginning of the list versus libraries at the end.

As far as I can tell, this is simply illegal. umount(8) says this:

> Note that a filesystem cannot be unmounted when it is 'busy' - for example, when there are open files on it[...]

As long as the loader has at least one file in the mount open, it cannot be unmounted, at least according to this man page.

This is probably also why they want to tuck the new mount underneath the old mount, instead of putting it on top - umount(2) already provides flags for doing a "lazy" unmount that avoids busyness problems, but there is no such thing as a "lazy" mount(2).

An operation for filesystem tucking

Karellen — Fri, 31 Mar 2023 17:00:03 +0000

I wonder, why not mount the new overlay on top of the old overlay, and then unmount the old overlay from beneath it, instead?

That should provide the same atomic, no possibility of missing overlay, semantics, but without the extra flags and options, shouldn't it?

If unmounting covered filesystems is not currently allowed, then some new code may be needed to allow that, but it's not immediately obvious to me why that approach would be longer or more complex than the proposed "tucking" mechanism?

An operation for filesystem tucking

jengelh — Fri, 31 Mar 2023 16:56:03 +0000

While the process is running, /usr/bin/whatever is in use (memory mappings), and /etc/passwd might be open (file descriptor), both blocking unmounting. Does overlayfs even support modifying the stack of mounts already? And how does that affect open fds?

An operation for filesystem tucking

jreiser — Fri, 31 Mar 2023 16:21:18 +0000

From the viewpoint of an application, there may be more races. The new overlay should be at least API (application programming interface) compatible with the old, and ABI (application binary interface) compatibility also may be required in some cases. For example, if the unmount of the old overlay occurs while rtld, the run-time dynamic linker, is loading a list of shared libraries into an address space, then it may be easy to trigger an incompatibility between libraries at the beginning of the list versus libraries at the end. Of course such a race is present already with any overlay, but now there will be more races, even (especially) during security updates that use such tucking. Checking the fsuid of loaded pieces may become even more important. Pausing for an overlay update may be the only general solution.

An operation for filesystem tucking

bluca — Fri, 31 Mar 2023 14:53:08 +0000

Friar Tuck's new favourite syscall!