Local root vulnerability in snap-confine [LWN.net]

Local root vulnerability in snap-confine

Posted Feb 17, 2022 20:06 UTC (Thu) by jra (subscriber, #55261)
Parent article: Local root vulnerability in snap-confine

Oh look, *ANOTHER* symlink vulnerability. This time, in a defensively programmed, security hardened binary. (The text below even remarks on how hard this was):

https://www.qualys.com/2022/02/17/cve-2021-44731/oh-snap-...

"Discovering and exploiting a vulnerability in snap-confine has been
extremely challenging (especially in a default installation of Ubuntu),
because snap-confine uses a very defensive programming style, AppArmor
profiles, seccomp filters, mount namespaces, and two Go helper programs.
...
setup_private_mount() is programmed very defensively (f*() and *at() syscalls, O_DIRECTORY and O_NOFOLLOW flags) to avoid race conditions:"

But that wasn't enough (it never is). Symlinks to the rescue for the malware community !

More evidence (if such were needed) that it is *IMPOSSIBLE* for even security engineers to write symlink-safe code (I sometimes wake up in a cold sweat thinking I missed something in the Samba symlink fixes. Worst of all, I probably did). When will our engineering community finally have enough of this and work to excise symlinks from our file systems ?

to post comments

Local root vulnerability in snap-confine

Posted Feb 17, 2022 20:15 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

To be totally fair, it also has a hardlink and mount vulnerability. Should we ban them as well?

IMO, it's POSIX API that we should stop deifying. It's a mess that evolved without much coherent thought.

Local root vulnerability in snap-confine

Posted Feb 17, 2022 20:21 UTC (Thu) by jra (subscriber, #55261) [Link] (4 responses)

Yes, the symlink problem isn't the only one. But vulnerabilities around hardlink and mount are so rare they are worth note.

Symlinks are the most common security problem with Linux applications. I agree the POSIX filesystem API is a horrible mess, but it's symlinks that have made it so. The worst part of it is that what you're calling the "POSIX API" is actually the thin layer on top of the Linux system call interface (which because of symlinks is also a horrible mess).

Banning symlinks would remove a large number (I did the stats for my SambaXP talk) of security holes in applications on Linux. Think of the ones we've yet to find.

Local root vulnerability in snap-confine

Posted Feb 17, 2022 20:28 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> Yes, the symlink problem isn't the only one. But vulnerabilities around hardlink and mount are so rare they are worth note.
But they're not!

Any way that can mutate the filesystem structure causes exactly the same problem, and this most definitely includes mounts/overlays. And overlays are becoming critical for containers that we would want to launch without admin permissions.

So mutability of the filesystem is not going away and apps will have to deal with it, symlinks or not.

> I agree the POSIX filesystem API is a horrible mess, but it's symlinks that have made it so.
Not really? It's mostly the lack of handle-based interfaces that is the issue, both for filesystem APIs and for the process/signals API.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 20:41 UTC (Fri) by developer122 (guest, #152928) [Link]

Side question: would it have done anything if it was originally required that the user have permissions for the destination if they wished to create a simlink? Is this feasible? (invalid symlinks might be a tricky edgecase, especially if they're not invalidated when their target is removed)

Local root vulnerability in snap-confine

Posted Feb 18, 2022 0:55 UTC (Fri) by Karellen (subscriber, #67644) [Link]

The worst part of it is that what you're calling the "POSIX API" is actually the thin layer on top of the Linux system call interface

That feels backwards to me.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 1:15 UTC (Fri) by aabc (subscriber, #55202) [Link]

/proc/sys/fs/protected_symlinks?

Local root vulnerability in snap-confine

Posted Feb 18, 2022 8:22 UTC (Fri) by mfuzzey (subscriber, #57966) [Link] (4 responses)

But symlinks are also extremely useful and many things would beak without them (shared library links, /etc/alternatives are the first things that spring to mind).

Removing them in the name of security would be like unplugging the network cable to fix network security issues.

Maybe the API could be improved by phasing out the path based interface over the fd based interface but given the quantity of code using it that would be hard.

I don't think removing features or even *forcing* code changes in the name of security is a good idea. By all means we should provide more secure ways of doing things but security is not the top requirement in all use cases. By "forcing" above I mean things like changing the kernel so the old insecure app wouldn't work anymore (which isn't going to fly with Linus anyway)..

Local root vulnerability in snap-confine

Posted Feb 18, 2022 8:50 UTC (Fri) by taladar (subscriber, #68407) [Link]

I don't think forcing code changes should be ruled out as an option but it can only ever be a later step where the first step is to actually provide at least one secure alternative.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 18:21 UTC (Fri) by jra (subscriber, #55261) [Link] (2 responses)

One of the things I'd like to see is wide knowledge and availability of MNT_NOSYMFOLLOW use, and making it available as a normal mount flag.

Then applications can state:

"This application is only known to be secure when used on a filesystem mounted with the MNT_NOSYMFOLLOW option. Use on filesystems allowing symlinks can lead to race conditions and security vulnerabilities."

Let admins know how to protect themselves from this misfeature.

Local root vulnerability in snap-confine

Posted Feb 19, 2022 6:23 UTC (Sat) by intelfx (subscriber, #130118) [Link] (1 responses)

> "This application is only known to be secure when used on a filesystem mounted with the MNT_NOSYMFOLLOW option. Use on filesystems allowing symlinks can lead to race conditions and security vulnerabilities."

I wouldn't use such software.

Local root vulnerability in snap-confine

Posted Feb 19, 2022 23:21 UTC (Sat) by jra (subscriber, #55261) [Link]

If I can get this mount flag to be easily used I will certainly add it to the recommended settings for a Samba server.

I'm sick of symlink insanity.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 8:59 UTC (Fri) by epa (subscriber, #39769) [Link] (6 responses)

Would it be fair to say that most of these security holes are not caused by symlinks alone, but by race conditions with symlinks?

The legitimate uses of symlinks are mostly static -- a shared library linked to a different name, an old user who no longer exists. They don't change much. Can we somehow restrict the times at which symbolic links can be created or changed? Perhaps a process should only be able to see the symlinks that existed when that process started, unless it calls a refresh_symlinks() system call. That would avoid path traversal race conditions while still letting us have the uses of symlinks we've come to depend on.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 11:51 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (4 responses)

That is a lot of state that needs to be tracked. It also raises the question of "how do I debug this path (name) not resolving in this daemon" with fun "oh, it hasn't updated its symlink view" kinds of answers.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 12:46 UTC (Fri) by epa (subscriber, #39769) [Link] (3 responses)

Maybe if a symbolic link has appeared or changed since you last called refresh(), then path lookup operations traversing that link would fail with EREFRESH. This would be fail-safe for older code, and newer code might be able to handle it sanely (it certainly sounds easier than meticulously rewriting all your code with the 'at' system calls). At worst, you just have to restart your daemon. Again, symlinks don't change often in practice so this doesn't seem too high a price.

It would have to be based on real world time, not just whatever mtime is in the file system. A fully snapshottable filesystem would be impossible to graft on to current POSIX semantics, but if we're just making a sticking plaster for race conditions with symlinks, some kind of per-process view of the world seems possible. You're not trying to snapshot exactly what symlinks existed at a point of time, but only to note whether one has changed in any way, and if so fail when it's used.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 14:57 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

Yup. Don't snapshot your symlinks per process, but make it so the first access caches it - REMEMBERING THE INODE - and any further attempts to access the symlink get the same inode until the cache is actively flushed.

Cheers,
Wol

Local root vulnerability in snap-confine

Posted Feb 18, 2022 20:50 UTC (Fri) by developer122 (guest, #152928) [Link]

A surprisingly elegant solution, imo.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 17:43 UTC (Fri) by nix (subscriber, #2304) [Link]

> This would be fail-safe for older code

What? It would cause new failures for older code, i.e. it would introduce countless DoS vectors, only a microscopic proportion of which correspond to actual attacks, but all of which would annoy users (where "annoy users" spans the entire spectrum from "slightly annoying" through "bug we can't track down that goes away on restarting" through to "oops now the system is unbootable because of an unexpected and undiagnosed EREFRESH while installing something boot-critical").

Hell no.

"owning" a symlink?

Posted Feb 21, 2022 10:14 UTC (Mon) by kronat (guest, #117266) [Link]

> The legitimate uses of symlinks are mostly static [...]
Based on the above, isn't it possible to "block" any modification to the symlink until the application(s) that "locked" the symlink by opening it for reading is (are) done (doing a close on it)?

Local root vulnerability in snap-confine

Posted Feb 18, 2022 11:48 UTC (Fri) by auerswal (subscriber, #119876) [Link] (2 responses)

> Oh look, *ANOTHER* symlink vulnerability.

Please read the report, there are no symlink vulnerabilities mentioned at all.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 12:49 UTC (Fri) by epa (subscriber, #39769) [Link] (1 responses)

One of the vulnerabilities, CVE-2021-44731, is a symlink attack.

Local root vulnerability in snap-confine

Posted Feb 19, 2022 18:02 UTC (Sat) by spigot (subscriber, #50709) [Link]

Amusingly, the Acknowledgments section of the write-up has this dedication:

This advisory is dedicated to 8lgm -- followers of symbolic links, overflowers of stack buffers, and dereferencers of NULL pointers: [...]

Local root vulnerability in snap-confine

Posted Feb 18, 2022 15:35 UTC (Fri) by intgr (subscriber, #39733) [Link] (19 responses)

> https://www.qualys.com/2022/02/17/cve-2021-44731/oh-snap-...

Thanks for the link, this report is pure gold and worth reading in full.

Truly, props to snap-confine developers for implementing mitigations to make exploitation so complicated.

As for the other vulnerabilities they found......... *sigh*

> CVE-2021-3995: Unauthorized unmount in util-linux's libmount

> only the first "sz" characters of the
> current user's uid are compared to the filesystem's "user_id" option (sz
> is user_id's length). This second vulnerability allows an unprivileged
> user to unmount the FUSE filesystems that belong to certain other users;
> for example, if our own uid is 1000, then we can unmount the FUSE
> filesystems of the users whose uid is 100, 10, or 1

This is hilarious and sad at the same time. It would be obvious to blame the C language. But I think it's time to start putting blame on developers as well, when instead of using proper primitives like "compare string A and B and tell me if they're equal", they insist on using low-level primitives with pointer arithmetic and buffer manipulation. This is the same mindset that led to the recent "pkexec" vulnerability.

This way of parsing and handling strings should be unacceptable in a culture that values security. Please use (create) a proper C library of high-level string manipulation functions, no shortcuts!

> CVE-2021-3996: Unauthorized unmount in util-linux's libmount

> when parsing /proc/self/mountinfo, the libmount blindly
> removes any " (deleted)" suffix from the mountpoint pathnames

> For example, on Fedora, /tmp is a tmpfs, so we can mount a basic FUSE
> filesystem named "/tmp/ (deleted)" (with FUSE's "hello world" program,
> ./hello) and unmount /tmp itself (a denial of service)

Hilarious. This, again, is a culture problem. We should be migrating to proper serialization formats when communicating complex data structures to between kernel and userland, instead of this stringly typed complex to parse hodgepodge that is "mountinfo".

The "deleted" flag should be a boolean or flags field, not part of the path name string.

> CVE-2021-3997: Uncontrolled recursion in systemd's systemd-tmpfiles

> if we create thousands
> of nested directories in /tmp, then "systemd-tmpfiles --remove" (when
> executed as root at boot time) will call its rm_rf_children() function
> recursively (on each nested directory) and will exhaust its stack and crash

> - for example, on Ubuntu 21.04, systemd-tmpfiles fails to create the
> directory /run/lock/subsys; but because /run/lock is world-writable,
> attackers can create their own /run/lock/subsys; and because various
> legacy packages and daemons write into /run/lock/subsys as root, the
> attackers may create arbitrary files via symlinks in /run/lock/subsys.

Mind=blown. One would expect the crash of a startup service to "fail safe" and possibly disable part of the system. But the auditors managed to turn it into a privilege escalation vulnerability.

Not sure where to put the blame, I could see myself making this mistake just as well.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 16:20 UTC (Fri) by epa (subscriber, #39769) [Link] (12 responses)

I think you can also blame whoever decided that ( ) and space should be legal characters in a mount point path name. It serves no purpose to allow them. But this is a cultural hot button issue, since the original Unix decision to allow any byte except / and NUL in a filename has become somehow sacred. (I'm not even talking about filenames in general, just the name of mount points!)

You can also blame whoever came up with the mountinfo format, but that may be a little unfair as I'm sure it has simply evolved rather than being designed. With hindsight, if you are pasting together strings in a formatted text output you should think in advance of what to do if one of the formatting characters (like, I don't know, newline) appears in a string. But again, there's no cultural consensus on what to do, and any scheme you invented would probably be parsed wrongly by lots of userland code, since it's an uncommon case.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 18:33 UTC (Fri) by mpr22 (subscriber, #60784) [Link] (2 responses)

It's "sacred" because getting any traction for moving away from such permissiveness means an unknown amount of work for an unknown number of people who don't even know you're planning to impose that work on them.

Local root vulnerability in snap-confine

Posted Feb 18, 2022 21:11 UTC (Fri) by ms-tg (subscriber, #89231) [Link] (1 responses)

> It's "sacred" because getting any traction for moving away from such permissiveness means an unknown amount of work for an unknown number of people who don't even know you're planning to impose that work on them.

I wonder if there are steps that, over some years, could help make the unknown number of people visible? For example, what if Linux, either in the file systems or more likely in distro-level utilities, simply detected filenames that violated certain rules, and let the user know that these existed and might need to be renamed to follow a proposed RFC naming convention in the future if it becomes accepted?

Perhaps users could opt-in to also reporting counts back to a central server?

I'm even picturing a nice web page, the way big security vulnerabilities have these days? Maybe 'linux-filenames-rfc.io'? Which could include the latest text of the proposal, links to Github where changes can be proposed, mailing lists where it is discussed, etc?

Using these sorts of community consensus approaches, might it be possible to get more of a defined sense of how often how big a real-world impact would be felt by any proposed reduction in Linux filename permissiveness?

Local root vulnerability in snap-confine

Posted Feb 24, 2022 11:07 UTC (Thu) by Wol (subscriber, #4433) [Link]

What you need is something that OPTIONALLY enforces all this stuff. Like David Wheeler's comments on filenames for example - a *create*time flag for a filesystem that says "enforce these safety rules".

Sure, all hell could then break loose, as old apps assume they can write garbage, etc etc, but it gives *developers* the opportunity to have a flag day and clean up their own stuff. Then they recommend that users enable this flag, which can't be enabled on existing systems to avoid breaking them, and all this goodness is enforced going forwards.

Then you reverse the default state of the flag so all of a sudden, people with old apps have to ask for the old behaviour ... But the breakage is minimised by making it easy for those in the know to be early adopters and test their own systems. You only need something like SUSE's Open Build System flipping their default setting, as developer's build systems start breaking and a distro could then rapidly switch their own default knowing all the apps they've built are compliant ...

Cheers,
Wol

Local root vulnerability in snap-confine

Posted Feb 19, 2022 9:40 UTC (Sat) by adobriyan (subscriber, #30858) [Link]

The technical bug is that (deleted) is appended, not prepended so it is impossible to disambiguate just by looking at the output.

Local root vulnerability in snap-confine

Posted Feb 19, 2022 10:26 UTC (Sat) by intgr (subscriber, #39733) [Link]

> whoever came up with the mountinfo format, but that may be a little unfair as I'm sure it has simply evolved rather than being designed

I don't really blame whoever came up with it, but I blame kernel developers for not offering better alternatives for new use cases and migrating old kludges over.

Kernel developers should decide on a new unambiguous and extensible serialization format for complex nested data structures. In 1990s these lessons hadn't really been learned yet, but in 2022 we have protobuf, Avro, Arrow, ASN.1, JSON and plenty more. I'm sure we can find at least one that suits the kernel use case.

> blame whoever decided that ( ) and space should be legal characters in a mount point path name

This wouldn't even help that much. Restricting characters in path names would solve this particular issue for paths, but there are still plenty of other strings with arbitrary content that may end up being communicated to user space. So would you propose blocking a set of characters in *all* kernel inputs?

I feel like a broken record at this point, but again: if you're having trouble transmitting certain characters in your serialization format, it's the serialization format that sucks.

Local root vulnerability in snap-confine

Posted Feb 20, 2022 3:03 UTC (Sun) by foom (subscriber, #14868) [Link] (6 responses)

Eh? No use for spaces and parentheses? Both of those are incredibly useful in filenames for humans! Characters like newlines or tabs, sure -- those have no place in filenames. And you could arguably exclude double quotes as well, but a filename like "Aaron's Report: Cats (2010)" is completely reasonable.

If there's blame to be had, it'd be for the Unix shell language which treats strings in a variable unsafely by default!

Local root vulnerability in snap-confine

Posted Feb 21, 2022 7:01 UTC (Mon) by epa (subscriber, #39769) [Link] (5 responses)

I meant mount points should be restricted to alphanumeric and -_. characters. For filenames in general I agree that humans expect to use ().

On some platforms space is not a legal character in filenames but you can use non-breaking space, which isn’t treated specially by the shell. I’m not sure whether that is better or worse.

Local root vulnerability in snap-confine

Posted Feb 21, 2022 12:43 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (4 responses)

Even "-" character should not be allowed as the first character of a filename. There are many gotchas with various characters. David A. Wheeler summarized them neatly in https://dwheeler.com/essays/fixing-unix-linux-filenames.html .

Local root vulnerability in snap-confine

Posted Feb 21, 2022 13:37 UTC (Mon) by adobriyan (subscriber, #30858) [Link] (3 responses)

> Even "-" character should not be allowed as the first character of a filename.

'-' is allowed in URLs. Any site issuing randomised names (like: YT) to user submitted content will create a file starting with minus easily.

Local root vulnerability in snap-confine

Posted Feb 21, 2022 14:15 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (2 responses)

Everything except NULL and / is allowed in filenames. But it doesn't mean it's a good idea to use it. Please take time to read the essay, it is eye-opening.

And you do not need a website. Just touch -- --foot-shooter (or touch -- -rf\ . if you like).

Local root vulnerability in snap-confine

Posted Feb 21, 2022 17:33 UTC (Mon) by adobriyan (subscriber, #30858) [Link] (1 responses)

David Wheeler's proposal would ban every ~1/(26+26+10+4) = 1/66-th video downloaded by youtube-dl. It is too late to ban anything.

GNU ls(1) started to quote filenames with spaces at some point, it was huge step in right direction because copy paste suddenly started working "out of the box". More features lke this are necessary.

Local root vulnerability in snap-confine

Posted Feb 21, 2022 18:04 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> David Wheeler's proposal would ban every ~1/(26+26+10+4) = 1/66-th video downloaded by youtube-dl. It is too late to ban anything.

Just add automatic escaping for such names.

Local root vulnerability in snap-confine

Posted Feb 19, 2022 8:55 UTC (Sat) by roc (subscriber, #30627) [Link] (3 responses)

> Please use (create) a proper C library of high-level string manipulation functions, no shortcuts!

The problem is that C doesn't provide enough language features to make a library string type as convenient and performant to use as char*. (Language features such as destructors.) Therefore C programmers will be disinclined to use it. It's not an accident that after all these years no such library is in broad use.

Local root vulnerability in snap-confine

Posted Feb 20, 2022 16:59 UTC (Sun) by epa (subscriber, #39769) [Link]

I meant the names of mount points, not file names in general.

Local root vulnerability in snap-confine

Posted Feb 21, 2022 15:20 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I think your comment gets very nicely at the root of our problems. We will never get security right as long as it is given lower priority than convenience and performance.

Local root vulnerability in snap-confine

Posted Feb 22, 2022 2:29 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

Heh. The C string library functions routinely shows as "hot" when I benchmark certain programs because there's no way to use a view reliably (C++ helps out here, but if it were C…there wouldn't even be the ability to factor out that `strlen` from `if/else` trees at least using static strings). To use a substring, you have to allocate and copy it to new memory just to put a NUL at the end and that adds up over time. So I don't think performance is really the thing keeping C back from a new string library because it's frankly just not that great for string manipulation-heavy tasks to begin with IME.

As for convenience, sure it's handy to be able to pass a "string" around in a register because it's "just" the size of a pointer, but weighing that against the inconvenience of having to debug mishandling of these APIs, they are by *far* in the red in my book. I mean, we have how many flavors of "append a string"? `strcat`, `strncat`, and `strlcat`? Toss `snprintf` in there too if you want to be fancy. And there's not even *consensus* on this set. Reviewing code dense with use of the C string library requires notes because I need to track where lengths are coming from, are the right APIs being used, and "oh, that's just a poorly coded `strdup`" recognitions. I don't find it useful.

So given that it scores (IMO) a 2/5 on performance (sure, registers get you something I guess), 1/5 in (overall) convenience, and 0/5 on security because the APIs just don't help out half the time, why, specifically, would roc's "attitude" not be warranted?

Local root vulnerability in snap-confine

Posted Feb 20, 2022 23:42 UTC (Sun) by judas_iscariote (guest, #47386) [Link] (1 responses)

> proper serialization formats when communicating complex data structures to between kernel and userland, instead of this stringly typed complex to parse hodgepodge that > is "mountinfo".

What is actually wanted is system calls, that return unambigous, well defined data.. same on all archictectures or kernel config options.. no new proc files to parse or to write to...

Local root vulnerability in snap-confine

Posted Feb 21, 2022 16:37 UTC (Mon) by nybble41 (subscriber, #55106) [Link]

> What is actually wanted is system calls, that return unambigous, well defined data..

We have the system calls (open, read, write). What is missing is the "unambiguous, well-defined data". Given an appropriate serialization format there is nothing wrong with communicating the data through /proc. I see no need to restrict access to languages with support for making special-purpose system calls.