Ext4 data corruption trouble [LWN.net]

Ext4 data corruption trouble

Posted Oct 24, 2012 21:02 UTC (Wed) by nix (subscriber, #2304) [Link]

Can systemd cope with the case this hack was handling, to wit a filesystem hierarchy like so:

/local-foo/loopback-nfs-mounted-foo/local-foo

In this situation, you cannot unmount local filesystems until you've killed everything that may have them as current directories -- but you can't kill local processes either, because that'll render your loopback-NFS mount, and everything underneath it including the local mount, inaccessible, and you don't learn this until your umount on reboot stalls indefinitely, which is tough if you're hundreds of miles away and working remotely and this is your main fileserver.

So I resorted to lazy-unmounting everything. But this unfortunately means that umount returns before the *successful* umounts are complete. So I sleep for a bit after umounting... but not necessarily long enough. This is all really gross and unclean and disgusting, but it's been working for many years so I forgot it existed and never tried to make it less gross.

(Aside: this may well go wrong even if I sleep for ages, in which case all that is needed to trigger this is a non-unmounted fs, not an fs halfway through unmounting. I'll be testing that next.)

Ext4 data corruption trouble

Posted Oct 24, 2012 21:26 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

As I understand, systemd tries killing/unmounting in the topoligical order. So if there's an order in which your processes can be killed and filesystems unmounted - it can do this.

Ext4 data corruption trouble

Posted Oct 24, 2012 21:36 UTC (Wed) by nix (subscriber, #2304) [Link]

Topological order won't help, alas, not unless it identifies processes which have files open on filesystems or current directories on such filesystems and toposorts *them* (note further that an unambiguous toposort in this case many not be possible, e.g. if you had a weird userspace fileserver serving /foo and /foo/bar, and that fileserver had a current directory set to /foo/bar...)

Raw umount(8) does a toposort unmount as well. It is not enough.

Ext4 data corruption trouble

Posted Oct 24, 2012 22:00 UTC (Wed) by tomegun (subscriber, #56697) [Link]

If you use "the right" initramfs (e.g. dracut or Arch's mkinitramfs) with systemd, it might work better.

In that case systemd will jump back to the initramfs on shutdown, and the initramfs will then try to kill/unmount whatever processe/mounts remains in the rootfs.

Ext4 data corruption trouble

Posted Oct 25, 2012 11:16 UTC (Thu) by nix (subscriber, #2304) [Link]

Sure! But... what if you have a mount point causing stalls (perhaps relating to an inaccessible NFS server), with mounted local filesystems buried beyond it? If you do a umount, rather than a umount -l, your shutdown will lock up forever as soon as it hits that mount point.

Worse yet, what if you have processes in other PID namespaces, holding open filesystems in other filesystem namespaces? The initramfs can't even see them! *No* umount loop can fix that. I hate adding new syscalls, but I really do think we need a new 'unmount the world' syscall which can cross such boundaries :(

Ext4 data corruption trouble

Posted Oct 25, 2012 12:50 UTC (Thu) by rleigh (subscriber, #14622) [Link]

It would be nice for such a system call to also work for selected mount namespaces, so that you can be sure everything is consistent after the last process in the namespace exits. (I assume once the namespace no longer has any users, the mounts are automatically umounted?)

Ext4 data corruption trouble

Posted Oct 25, 2012 13:31 UTC (Thu) by nix (subscriber, #2304) [Link]

I think that's so, yes: that's how MNT_DETACH and MNT_EXPIRE are implemented (and, thus, umount -l).