LWN: Comments on "Mounting images inside a user namespace"

Mounting images inside a user namespace

tao — Tue, 01 Aug 2023 11:24:19 +0000

Ah, sorry, my apologies for not being able to recognise that you were joking. Keep up the good work on systemd!

Mounting images inside a user namespace

nye — Wed, 19 Jul 2023 09:36:39 +0000

I know this is a month old but your behaviour across these comments is so shocking that I just can't stay quiet. You should be ashamed.

Mounting images inside a user namespace

mathstuf — Sun, 02 Jul 2023 11:48:31 +0000

> I should point out, as a heavy user of udisks2 and Lubuntu -- **udisks** is not the one triggering the automount. In my LXQT desktop; the file manager has a checkbox for enabling automounting (which defaults to on).

Yes, `udiskie` needs to be running for `udisks2` to do anything on my machines. One can also set up automounting behaviors via udev properties (I use automount for many things, so a lot of udiskie's feature set is asking for LUKS passwords and then mediating locking again).

Mounting images inside a user namespace

Kamilion — Sat, 24 Jun 2023 00:08:39 +0000

I should point out, as a heavy user of udisks2 and Lubuntu -- **udisks** is not the one triggering the automount. In my LXQT desktop; the file manager has a checkbox for enabling automounting (which defaults to on).

One of my long-time modifications while rolling my own livecd has been disabling it, as one of my primary dayjob tasks in ewaste happens to be erasing media, and it's incredibly annoying to have something trying to automount a disk you're just about to scramble.

"oooh, I see a lvm signature! please may i?"
"No, I took the LVM tools away from you for a reason, don't touch it."
"awwww."

(I do, in fact support the idea of mounting VFAT through fuse for additional safety though. I *would* however, appreciate someone finally getting around to a sane way to inform graphic console users something has gone wrong with storage processes. There's been many times I've briefly plugged a device in and unplugged it, and this has angered the kernel or userspace on many occasions, with no recourse but to stare at dmesg for a while. "oh, kernel was still retry looping on link rate for 18 seconds before giving up on the SATA endpoint or noticing a serial change")

Surely you need to handle corrupt data anyway

pizza — Mon, 19 Jun 2023 12:26:29 +0000

> I don't even understand why an automount of a USB stick happens with udisk if it's not in VFAT format. (Those eccentric people who want to mount ext4 USB sticks -- and I belong to them! -- are not worth to make that the default behaviour.)

For Fedora/GNOME, EXT4 and XFS filesystems are only sometimes automounted. SD cards seem to be, but "real" external drives required me to explicitly mount the drive, either manually or by clicking on it in GNOME's file browser. Though now that I think about it, it's possible there are LVM interactions involved here too. (yay, another layer of things to go very wrong..)

(I don't recall if NTFS volumes behave the same way -- I have a 1TB SSD set up that way, but it's not stashed somewhere convenient)

Surely you need to handle corrupt data anyway

farnz — Mon, 19 Jun 2023 11:32:43 +0000

One thing that's not inherent to C versus Rust, but is a cultural difference that tends to go in Rust's favour, is Rust developers' preference for parsing over validating, and for abstractions over raw integers that prevent you making the security errors to begin with. This isn't a guarantee (unlike memory safety in safe Rust), but it does tend to result in fewer security issues.

This, in turn, leads to bounds on the danger level of accessing a filesystem; if the worst case possible by supplying a corrupt filesystem to a driver is that it'll read parts of the block device it wouldn't have read without the corruption, then you have a very different issue to one where a corrupt filesystem will cause the driver to read from an entirely different partition.

Surely you need to handle corrupt data anyway

jschrod — Mon, 19 Jun 2023 01:27:43 +0000

I'm aware of Asahi Lina's work and think it's great showpiece that writing kernel code in Rust is a big step forward to more secure code.

OK, Rust gives you memory-safe code. Well, since decades most of my programming is done in programming languages with managed memory. There the kind of problems that you cite "compared to doing the same thing in C" don't happen.

But: Do you really think that these systems are free of security bugs? I have CVEs for code, written by me in Lisp, that proves this wrong. In this thread the focus is on the obvious problems with memory management and access. But there is a huge domain of bugs *in the logic of our programs*. This logic errors are all memory safe and won't prevented by Rust, but they bite you dearly. If the logic of an interpretation of a binary structure on disk is wrong, you're out. No safe filesystem any more.

And that is, what this article and the discussion is about: are we're willing to allow random people mounting filesystems with kernel code in user space? The proposal of "md" about a `safe file-system' that someone else shall write is contra-productive, IMNSHO. "md" argues that it's possible by principle, but no one wants to do that. I have raised my voice in an opposing argument: it's not even possible to do by principle. Murphy bites, and he does it all the time.

Lennart and others made, in this article discussion, a compelling argument for using FUSE. I don't even understand why an automount of a USB stick happens with udisk if it's not in VFAT format. (Those eccentric people who want to mount ext4 USB sticks -- and I belong to them! -- are not worth to make that the default behaviour.) IMHO, these are the directions one has to follow to constrain that attack vector. Gambling on a "safe filesystem" is not an option, neither technically nor socially.

FWIW, my background: I have never been involved in filesystem development. But for the last 40(!) years, I wrote printer and graphics drivers. From that background of hardware-near software development, I know that Rust will be a *VERY BIG* help in better security, but it will not be a panacea that guarentees us to be safe of any security problem.

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 20:08:24 +0000

>Part of the point of Rust is that the unsafe part can be contained and scrutinized for that kind of errors

Exactly right.

And - what lots of people get wrong all the time - using the unsafe keyword in Rust does not mean that the program loses memory safety.
Quite contrary. The unsafe block have to commit to memory safety. In unsafe blocks it's also not allowed to write memory-unsafe code.

unsafe blocks are all around in Rust. All low level primitives are unsafe, basically.
But safe wrappers are built around them.

Therefore, of course a file system can be implemented in safe rust. Of course there is a low level data access part where raw bits have to be transmuted to safe structures. But that's a small part that can be audited for memory safety much more easily than a C-only program.

That said, it's also possible to write a safe C program, of course. It's just very much harder.
And that is the real reason why we have unsafe fs implementations. Saying that some "experts" evaluate it as being impossible is false. No real expert says that.

Surely you need to handle corrupt data anyway

smurf — Sun, 18 Jun 2023 19:45:19 +0000

> Rust doesn't have any magic pixie dust

Well … if you read Asahi Lina's (and her friends') accounts of how absurdly easy(*) it is for them to write an entire class of graphics drivers in Rust, you might change your opinion.

(*) compared to doing the same thing in C and then spending months searching for the inevitable memory reference count errors, fencepost bugs, and related issues

Rust has plenty of magic pixie dust. The point is, to apply it you need a competent magician (or three). And somebody needs to pre-mix the glue that holds the magic without disrupting it. And so on.

> All other security-relevant error causes are still there

Possibly true, but at least they don't crash the kernel any more; also, when you have code whose data structures can no longer get corrupted you can spend more time on higher-level issues. Like, ensuring that no data block points to metadata.

> It doesn't even make memory errors impossible -- since a filesystem has to handle arbitrary binary data on devices, some parts of that fs implementation would be in unsafe Rust.

Part of the point of Rust is that the unsafe part can be contained and scrutinized for that kind of errors, while the compiler handles safeguarding the rest of the code. In C there is no "rest of the code", it's *all* unsafe.

Surely you need to handle corrupt data anyway

smurf — Sun, 18 Jun 2023 19:28:22 +0000

> Saying that we can't have safe fs in the kernel "because the expert say so" has just been proven wrong.

The experts said no such thing. They said that the *current* crop of FSes are inherently unsafe and unlikely to be "fixed".

Thus if somebody wants one they get to write one themselves.

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 18:38:41 +0000

>Providing memory safety doesn't result in an implementation that can be used to
>"mount an arbitrary image inside a user namespace"

If you would have read what I had written before, then you would have noticed that I never ever made that claim.

The rest of what you wrote can be summarized in your own words:

>absurd

There is nothing but personal attacks.

Surely you need to handle corrupt data anyway

mgb — Sun, 18 Jun 2023 18:00:13 +0000

There are shades of gray between black and white.

I haven't used Rust yet - today I use mostly C++ and Perl - but I am trying to get an idea as to the tradeoffs Rust offers between programmer effort, runtime performance, and bug prevention.

Surely you need to handle corrupt data anyway

jschrod — Sun, 18 Jun 2023 17:35:13 +0000

So you think that writing a file system in Rust would result in a safe filesystem. That is so absurd, it's at the level of being comedy. Rust doesn't have any magic pixie dust that is automatically sprinkled over a program written in it and makes it safe at a level that is the point of discussion in this article. Everybody who thinks so, is, IMNSHO, not a professional programmer.

Providing memory safety doesn't result in an implementation that can be used to "mount an arbitrary image inside a user namespace" (what this article is about). It just makes it harder to get *one* sub-class of errors, caused by memory allocation and access. All other security-relevant error causes are still there. It doesn't even make memory errors impossible -- since a filesystem has to handle arbitrary binary data on devices, some parts of that fs implementation would be in unsafe Rust.

> Ok, so I prove my point

Well, actually you prove my point: You seem to have no clue at all what it means to write any safe program, even less so a program that realizes a file system. And, as I wrote, that's why your opinion on that topic is not trusted by the participants in that thread.

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 17:06:38 +0000

> I suggest you look up the definition of "cost-benefit analysis".

Ah. So now we get to a common understanding of the situation.
So it is indeed possible to have safe fs code in the kernel.

It's just that the "experts" say that a kernel memory error induced by plugging a disk is not worth fixing.

As I said, I beg to differ. But I'm fine, if you have a different opinion.
Just please be honest and say "it costs too much", or "we don't care about users" or anything but "it's not possible". Because that's false.

Surely you need to handle corrupt data anyway

pizza — Sun, 18 Jun 2023 14:56:33 +0000

> So if it might be hard, then better give up?
> Not my way of thinking.

I suggest you look up the definition of "cost-benefit analysis".

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 13:51:55 +0000

>You do realize that it's actually almost impossible (or at least used to be)
>to write even a linked list, [...] in safe rust?

You do realize that saying it's *almost* impossible means that it actually is possible?
But besides that: It's not true. A linked list can easily be implemented in safe Rust. It just has a small runtime cost. If you don't want to pay that cost, you can use unsafe Rust and audit it for memory safety problems.
Both versions would be fine.
Remember what I said: You can also use C to implement safe fs drivers in the kernel. It's just much harder.

>Some things may just not be possible in it.

So if it might be hard, then better give up?
Not my way of thinking.

Surely you need to handle corrupt data anyway

SLi — Sun, 18 Jun 2023 12:40:08 +0000

You do realize that it's actually almost impossible (or at least used to be) to write even a linked list, let alone some more complicated structure, in safe rust? Some things may just not be possible in it.

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 12:33:07 +0000

>So which FS are you going to rewrite in Rust?

I did never say that I was rewriting any fs in Rust.

I just said that if one would rewrite an fs in Rust, it would automatically be memory safe. Therefore, the claim that it was not possible to have safe in-kernel fs implementations must be wrong.
Of course, you can also have a reasonably safe implementation in C. The developer just has to commit to that goal. But for whatever reason "the experts" don't want to.

Saying that we can't have safe fs in the kernel "because the expert say so" has just been proven wrong.
We should get over that false claim.

Surely you need to handle corrupt data anyway

smurf — Sun, 18 Jun 2023 11:13:34 +0000

So which FS are you going to rewrite in Rust?

And how do you propose we should handle all the others?

Surely you need to handle corrupt data anyway

mb — Sun, 18 Jun 2023 10:34:15 +0000

Ok, so I prove my point: Rewrite the fs in Rust. Then it immediately becomes memory safe. Therefore, the claim that an fs can't be safe in the kernel is wrong.

Mounting images inside a user namespace

smurf — Sun, 18 Jun 2023 06:58:22 +0000

> Are you aware that "bluca" is an important systemd developer?

Apparently not.

I wasn't either until Lennart told us, but so what, snarky comments are par for the course around here; we've heard far worse during the systemd wars of yesteryear.

Surely you need to handle corrupt data anyway

jschrod — Sat, 17 Jun 2023 22:11:17 +0000

> >according to the experts, it's not safe to do so.

> Sure. And I doubt it. What's your point?

I think his point is clear:
If the subject matter experts tell you that it's not safe and you disagree - then *you* have to prove your point.

Because the rest of the Linux developers will trust their file system developers more than you, without decades of track record in that area.

Mounting images inside a user namespace

jschrod — Sat, 17 Jun 2023 22:00:02 +0000

Are you aware that "bluca" is an important systemd developer?

The beauty of LWN.net is that the developers are directly involved in the discussions about their work. For that, we bystanders have to do our own work to decipher their in-jokes. The former is more important than the latter.

Please deal with it.

Surely you need to handle corrupt data anyway

mb — Thu, 15 Jun 2023 21:19:49 +0000

>according to the experts, it's not safe to do so.

Sure. And I doubt it. What's your point?

>if you believe you can build a perfectly safe filesystem

Where did I say that?

Surely you need to handle corrupt data anyway

bluca — Thu, 15 Jun 2023 20:20:04 +0000

We don't agree at all - it is a technical problem. The technical problem is that, according to the experts, it's not safe to do so. I mean it's open source, so if you believe you can build a perfectly safe filesystem that can resist to attacks via malicious images, by all means do so and publish the patches.

Surely you need to handle corrupt data anyway

mb — Thu, 15 Jun 2023 19:38:58 +0000

So we agree, that this is not a technical problem. That was my whole point.

We should refrain from reproducing the same "we can't have safe fs in the kernel" and instead say "we don't want to have safe fs in the kernel, because it's too much work, etc...".
Because that's actually what currently happens.

Developing a technical workaround for that is the *wrong* decision.

Surely you need to handle corrupt data anyway

bluca — Thu, 15 Jun 2023 18:32:58 +0000

Well, no, it's because those that would have to make such commitment don't want to... if you watch the recording of the session it is quite clear where the wind blows

Surely you need to handle corrupt data anyway

mb — Thu, 15 Jun 2023 17:36:16 +0000

We can't have a commitment, because there is no commitment?

Mounting images inside a user namespace

karim — Thu, 15 Jun 2023 13:43:38 +0000

Thanks for sharing this. This is useful.

Mounting images inside a user namespace

farnz — Thu, 15 Jun 2023 13:40:34 +0000

For starters, UML is currently x86-only; so using UML as your filesystem handler means porting UML to the other architectures, and taking on the security and code maintenance burden that represents.

Then you need a trustworthy IPC mechanism with the host kernel; something that allows the UML kernel to read/write the block devices it's managing, and a trustworthy filesystem protocol to share the FS from the UML kernel to the host kernel. Again, both of these add to the security and maintenance burdens.

Alternatively, you could go the ChromeOS route - the only filesystem used on external devices is FUSE from the kernel point of view, and you run userspace filesystem implementations that have been deliberately and intentionally hardened, and can be sandboxed heavily - you can design the userspace component to accept the FS as one file descriptor, and the FUSE interface as the other, and sandbox it such that all you have is anonymous memory plus accesses to those two file descriptors. This has the advantage over UML that the process that mounts the filesystem is designed to only read that one FS, and no others; so there's a much smaller attack surface to begin with.

Mounting images inside a user namespace

geert — Thu, 15 Jun 2023 12:53:39 +0000

So there will be one UML kernel for x86, one for x86_64, and all other architectures are blocked on gaining UML support first...

Mounting images inside a user namespace

karim — Thu, 15 Jun 2023 12:19:46 +0000

So what's the best tradeoff? Adding to the release testing burden or adding to code maintenance and security burden? I get that this isn't "free" for release management. But, personally, I'd rather trust the code that's already there and has a lot of road mileage than devise new code that will only exceptionally be used just to avoid burdening the release loop.

Surely you need to handle corrupt data anyway

bluca — Thu, 15 Jun 2023 11:43:40 +0000

Well, what pretty much all the actual filesystem developers seem to be saying is that there is no such "simple commitment", so...

Surely you need to handle corrupt data anyway

bluca — Thu, 15 Jun 2023 11:42:46 +0000

Private keys are in the same family as those used for secure boot, in fact the primary means to use it is via MOK and the machine keyring. There are revocations and blocklisting and rotation to deal with that, it is already covered in the existing threat model.

Surely you need to handle^W -correct- corrupt data anyway

dgc — Thu, 15 Jun 2023 11:15:46 +0000

> You make a good point that if the data is protected by a CRC, then it cannot
> be corrupted by hardware failure without that being obvious. However, it's
> still possible for the metadata to be corrupted by a software bug.

Yes. We've thought about that, too. After all, I'm writing some of the code and I don't trust myself to write bug free code. :)

To mitigate such problems, XFS also runs metadata structure verification functions on the structures it is about to write to disk. It does this before it recalculates the CRCs to ensure what we are sealing with the CRC and writing to disk has not been corrupted by some in-memory issue. We've caught all sorts of kernel memory corruption bugs, hardware memory corruption bugs (i.e. buggy CPUs - we've caught more than one) and other software (XFS) bugs with this pre-write IO verification architecture.

> So I would still tend to err on the side of "anything that can go wrong,
> will go wrong"

It should be clear by now that we already make this assumption, and that we assume really bad things *will* happen.

If you want to learn more about the metadata verification architecture in XFS, read this design document that was added to the 3.10 kernel (yes, a decade ago):

https://docs.kernel.org/filesystems/xfs-self-describing-m...

If that doesn't scare you off, then try this one on for size, the online repair design document goes into much more detail about how this metadata structure is leveraged for total structure verification and repair:

https://docs.kernel.org/filesystems/xfs-online-fsck-desig...

The complexity of the verification and repair problem might also give you some insight into why most filesystem engineers don't think robust runtime defense against malicious attack is possible for anything other than the simplest of filesystems. It should also go some way to explain why we also think that fsck doesn't provide any guarantee that a filesystem image is "safe"...

-Dave.

Mounting images inside a user namespace

bluca — Thu, 15 Jun 2023 10:36:44 +0000

Sure thing, have fun!

Mounting images inside a user namespace

SLi — Thu, 15 Jun 2023 10:14:35 +0000

Thanks, this was insightful! First I was baffled about why you want what you want, but I think you make an important point. It's kind of a mixed social (or standardization) and technical problem, and it makes sense to say that there is a de facto base standard (the kernel ABI), whether it's good or not.

Mounting images inside a user namespace

gray_-_wolf — Thu, 15 Jun 2023 09:14:37 +0000

Ignoring the previous comment (I think it was somewhat funny, but that is a matter of taste, and I have a poor one), I have to admit I am not sure why this is required. You mention AppImage and Flatpak, but why do they need disk images? Rootless podman works on my machine today, so I am not sure why they cannot use the same mechanism for distributing the files they need. Why does it need to be a disk image?

Mounting images inside a user namespace

farnz — Thu, 15 Jun 2023 08:51:12 +0000

In most distros today, you build one kernel per CPU architecture you support - so there's one AMD64 kernel, one AArch64 kernel, one RISC-V 64GC kernel etc.

You'd need to repeat that with UML for each CPU architecture you support, thus doubling the number of kernels you build - instead of building one kernel per supported architecture, you'd build two.

Mounting images inside a user namespace

smurf — Thu, 15 Jun 2023 08:07:11 +0000

The writable partition is usually encrypted. This means that any offline change either results in random data (which are typically not useful) OR requires the encryption key.

However, your ability to access that key implies that the device is already rooted, thus you don't need to fudge with the file system in the first place.