Rust 1.58.1 released

[Posted January 21, 2022 by corbet]

Anybody who upgraded to the recent Rust 1.58.0 release will probably want to move on to Rust 1.58.1; among other things it contains a fix for a security vulnerability in the standard library. "We recommend all users to update their toolchain immediately and rebuild their programs with the updated compiler".

to post comments

Rust 1.58.1 released

Posted Jan 21, 2022 16:57 UTC (Fri) by compenguy (guest, #25359) [Link] (4 responses)

IIRC, this vulnerability affects pretty much all extant versions of rust, but is limited to code that uses std::fs::remove_dir_all (either directly or indirectly), as there's a race condition that could allow a malicious unprivileged user to coax a privileged program that calls remove_dir_all() on a directory chain where the attacker has access to use symlinks to delete locations that the attacker wouldn't otherwise have access to.

Limited to code that uses std::fs::remove_dir_all

Posted Jan 21, 2022 17:59 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] (3 responses)

This vulnerability is limited to only code that uses std::fs::remove_dir_all (directly or indirectly) *and* is run on a platform where an attacker can create a symlink. Do we have any idea how many programs this really affects? I'm sure this affects some applications, but I would expect it to be relatively few.

Limited to code that uses std::fs::remove_dir_all

Posted Jan 21, 2022 20:24 UTC (Fri) by yodermk (subscriber, #3803) [Link] (2 responses)

*and* the program is suid

Yeah some no doubt. But not most.

Limited to code that uses std::fs::remove_dir_all

Posted Jan 21, 2022 22:05 UTC (Fri) by plugwash (subscriber, #29694) [Link] (1 responses)

As I understand it, it applies to any cases where a program deletes a directory tree that a potentially malicious user can write to. Especially if the deletion is regularly scheduled or if the malicious user can trigger it somehow.

The difficult thing is auditing where that is or is not the case. With a vulnerability in a shared library you just patch it, deploy the patched library and move on, with a vulnerability in a static library you either rebuild/redeploy everything that depends on said static library or you spend a bunch of time trying to audit/risk assess each program, or you just ignore the issue.

Limited to code that uses std::fs::remove_dir_all

Posted Jan 23, 2022 8:53 UTC (Sun) by tialaramex (subscriber, #21167) [Link]

In this particular case it's probably easier for entities to rule themselves out because they don't meet the preconditions to be attacked.

None of the systems I'm looking after at work run Rust, but even if they all did, none of them have hostile users who can make symlinks. Our hostile users (well, students, much the same thing) can write arbitrary files to disk, although they don't get to decide the filenames, in some of our applications, but they can't write symlinks.

Rust 1.58.1 released

Posted Jan 21, 2022 19:33 UTC (Fri) by epa (subscriber, #39769) [Link] (5 responses)

And you have to rebuild all executables because there is no dynamic linking?

Rust 1.58.1 released

Posted Jan 21, 2022 19:59 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

Yes, but given that there is still not an ABI rich enough to allow Rust <-> Rust communication over a dynamic library boundary yet, it would even if dynamic linking were (trivially) supported as any symbols exported would be mangled with some kind of compiler id embedded somehow anyways.

Basically, Rust has not yet nailed down "this struct declaration has this ABI layout" in a way that it guarantees yet, so any such requests are not really all that meaningful right now anyways. FWIW, this allows further experimentation with optimizations related to niches and enum discriminant representations (which continues today) that saying "this is the one and only ABI" would block off.

Rust 1.58.1 released

Posted Jan 21, 2022 21:28 UTC (Fri) by plugwash (subscriber, #29694) [Link]

And even if/when you do nail down the mapping from source constructs to binary constructs you still have a couple of other problems remaining.

1. How do you handle stability of library ABIs (including but not limited to the standard library), a library can easily have a stable API without having a stable ABI even if the compiler ABI is stable.
2. How do you handle generics/templates which can't be compiled until the element type is known?

And if your worldview is "a few big applications for which we have the source" I can see it's easy to regard dynamic linking of libaries as not very important, you are probablly rebuildling your application quite frequently anyway.

Will rust find a good answer to dynamic linking that fits it's general philosophy? (dynamic linking in C++ is anything but safe........) or will it be stuck with static linking forever? only time will tell......

Rust 1.58.1 released

Posted Jan 21, 2022 20:54 UTC (Fri) by atnot (guest, #124910) [Link] (2 responses)

Even if there was, it wouldn't help much here because this function is generic over string/path types. So every program would have it's own copy of it regardless.

Rust 1.58.1 released

Posted Jan 22, 2022 10:03 UTC (Sat) by epa (subscriber, #39769) [Link] (1 responses)

That is true, if it were like C++ where generic code has to be recompiled for each type. It is possible to have generic library routines, dynamically linked — as in Java and .NET. But then of course the library and executable are bytecode, not native code, and the runtime has much more overhead than Rust.

It might be neat to have ‘fake shared libraries’ in a central location. Rustlib-1.50.fake would be essentially just a couple of lines of text. A Rust program built against the standard library checks for its existence on startup. Then if a vulnerability is found, you delete the fake library file. That forces you to rebuild anything that uses it.

Rust 1.58.1 released

Posted Jan 22, 2022 10:25 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

> It is possible to have generic library routines, dynamically linked — as in Java and .NET.

.NET basically does what Rust does, but in runtime. It monomorphises the generic types on the fly as they are instantiated with concrete types.

Java and Swift are fully dynamic, on the other hand.

Rust 1.58.1 released

Posted Jan 21, 2022 20:54 UTC (Fri) by jra (subscriber, #55261) [Link] (78 responses)

Symlinks *again*. Sigh. The poisonous gift that keeps on giving. Took us over 2.5 years to fix this in Samba, ending up in a complete VFS layer rewrite.

https://www.samba.org/samba/security/CVE-2021-20316.html

The POSIX filesystem API is truly broken beyond repair I'm afraid.

Rust 1.58.1 released

Posted Jan 21, 2022 23:03 UTC (Fri) by NYKevin (subscriber, #129325) [Link] (77 responses)

Symlinks are useful because they fool oblivious programs into following them automatically. Symlinks are dangerous because they fool oblivious programs into following them automatically.

Rust 1.58.1 released

Posted Jan 22, 2022 2:20 UTC (Sat) by jra (subscriber, #55261) [Link] (76 responses)

At this point I think we have enough evidence to say the dangers *way* outweigh the benefits. Symlinks have to go.

Rust 1.58.1 released

Posted Jan 22, 2022 3:00 UTC (Sat) by JoeBuck (guest, #2330) [Link] (26 responses)

They just have to be used correctly. This function should never have followed symbolic links, it should remove the link but not what it links to.

Rust 1.58.1 released

Posted Jan 22, 2022 4:30 UTC (Sat) by jra (subscriber, #55261) [Link] (25 responses)

It is impossible for an application developer to avoid all symlink pitfalls and race conditions without an insane focus on the details that is beyond superhuman.

I've come to the conclusion that symlink race conditions are impossible to avoid, and trying to push the problem of avoiding them onto application developers has poisoned the once-clean POSIX filesystem API beyond repair.

More details in my forthcoming SambaXP talk later this year.

Rust 1.58.1 released

Posted Jan 22, 2022 5:46 UTC (Sat) by willy (subscriber, #9762) [Link] (17 responses)

Maybe I should wait for the talk, but symlinks were introduced in 4.2BSD (1983). Have we made things worse since then?

Rust 1.58.1 released

Posted Jan 22, 2022 7:04 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (14 responses)

Yes, we started using code and/or inputs which we don't trust to be non-malicious. In 1983 that was barely a thing, and "well don't do that then" was a valid response to this sort of issue.

Rust 1.58.1 released

Posted Jan 22, 2022 14:29 UTC (Sat) by willy (subscriber, #9762) [Link] (13 responses)

You're missing a great deal of the context between jra and I that makes this response somewhat less than useful.

Linux has added many extensions to the file name API over the years. openat() (and the rest of the *at() family). Then openat2() with its RESOLVE flags (see https://man7.org/linux/man-pages/man2/openat2.2.html ). Much of this was done in consultation with the Samba team, so I was under the impression we had finally solved name resolution in the presence of symlinks and mounts and ... to their satisfaction.

Maybe I should have taken this question to linux-fsdevel to avoid randos misunderstanding.

Rust 1.58.1 released

Posted Jan 22, 2022 17:00 UTC (Sat) by jra (subscriber, #55261) [Link] (9 responses)

Yes, all these API extensions to fix a horrible mistake (symlinks) that should be restricted, not accommodated (IMHO of course). How many real applications actually use these correctly ? How many actually know what these flags really do ? How many libraries that do filename manipulation are careful enough to avoid the problems ? Yes, theoretically it is now safe to write a symlink-safe application (not just with the new API's, there are Linux-specific tricks with /proc that must be done also - more details in my SambaXP talk :-).

There are even *security* libraries that don't correctly cope with symlinks. The complexity is too much. Complexity is the enemy of security.

Symlinks "Have.To.Go." (tm) :-).

Rust 1.58.1 released

Posted Jan 22, 2022 17:12 UTC (Sat) by willy (subscriber, #9762) [Link] (3 responses)

Maybe we could add a prctl() to indicate the process wishes to see ENOENT when trying to resolve a pathname that includes a symlink? Or ELOOP if you'd rather have it behave like RESOLVE_NO_SYMLINKS

Rust 1.58.1 released

Posted Jan 22, 2022 21:53 UTC (Sat) by jra (subscriber, #55261) [Link]

Excellent suggestions, thanks ! I'll think about possible fixes and make some suggestions in my talk.

Rust 1.58.1 released

Posted Jan 23, 2022 15:02 UTC (Sun) by ibukanov (subscriber, #3942) [Link] (1 responses)

It will be even more useful to restrict this to symlinks to files while allowing to resolve symlinks to directories. Most of the problems with symlinks come from opening files that are symlinks, not symlinks to directories.

Rust 1.58.1 released

Posted Feb 10, 2022 18:38 UTC (Thu) by nix (subscriber, #2304) [Link]

Quite. And symlinks to directories are critically useful to administrators, far more so than symlinks to files: they give you a large portion of what hardlinks to directories would (at least until you try to rm them). Bind mounts do the same, but bind mounts don't persist across reboots without entries for each one in fstab or some other more or less manual management scheme, and frankly to this day have often bizarre and unclear semantics if you try to export them over a network.

Rust 1.58.1 released

Posted Jan 22, 2022 23:06 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Nah. The complexity stems from allowing mixed-trust code on the system. This nonsense has to go, it's a relic of the era when Unix was used on multi-user shared machines.

Rust 1.58.1 released

Posted Jan 23, 2022 13:37 UTC (Sun) by khim (subscriber, #9252) [Link]

> This nonsense has to go, it's a relic of the era when Unix was used on multi-user shared machines.

Nonsense is the idea that one user should only have one level of trust. This leads to the infamous if someone steals my laptop while I'm logged in, they can read my email, take myh money, and impersonate me to my friends… but at least they can't install drivers without my permission.

But we still have no idea what's the best way to resolve that dilemma. Although many are trying, but problem is far from being fully resolved.

In particular most developer tools are designed for the paradigm where “one user = one level of trust”.

Rust 1.58.1 released

Posted Jan 22, 2022 23:17 UTC (Sat) by khim (subscriber, #9252) [Link] (1 responses)

> Yes, theoretically it is now safe to write a symlink-safe application (not just with the new API's, there are Linux-specific tricks with /proc that must be done also - more details in my SambaXP talk :-).

The question is whether we can provide library with somewhat sane higher-level API which people may actually use to create symlink-safe applications. Is it actually doable or are needs of different apps are so diverse than it's just impossible to do?

> Symlinks "Have.To.Go." (tm) :-).

Maybe Windows solution™? There symlinks require admin privileges (unless you turn on developer's mode which makes it possible to create them with normal privilege levels).

Rust 1.58.1 released

Posted Jan 23, 2022 20:42 UTC (Sun) by jra (subscriber, #55261) [Link]

Yes, this is the solution I'm leaning towards too. Not on by default, but maybe on a per-filesystem basis.

Rust 1.58.1 released

Posted Jan 23, 2022 10:34 UTC (Sun) by cyperpunks (subscriber, #39406) [Link]

Agree.

Having fixed several security issues due to symlinks I have no hope developers will ever write code that is safe regarding race conditions of symlinks. The only way is to remove support for symlinks.

Rust 1.58.1 released

Posted Jan 23, 2022 15:26 UTC (Sun) by walters (subscriber, #7396) [Link] (2 responses)

I think in the context of this article, it's important to note that all that was needed was openat() to do the fix. The same as coreutils `rm` and other tools have been doing for years.

What openat2 adds is the ability to follow symlinks in a constrained manner (e.g. in a particular root), but one doesn't need that in order to implement `rm -rf`.

I think not may userspace code tools actually want to follow symlinks inside a particular root; it's mostly things like container build tooling and runtimes, and userspace file servers.

I am very curious what Samba needs that isn't covered by the existing openat2() APIs.

For anyone who is writing one of those things though, I would recommend looking at e.g. https://crates.io/crates/cap-std (and the dependency chain) - it's high quality code, and I'm in the middle of porting some of our projects to use it.

Rust 1.58.1 released

Posted Jan 23, 2022 20:43 UTC (Sun) by jra (subscriber, #55261) [Link] (1 responses)

openat() is not enough. There are many details this doesn't solve.

Rust 1.58.1 released

Posted Jan 23, 2022 20:46 UTC (Sun) by jra (subscriber, #55261) [Link]

Leave me some details for my talk :-) :-).

Rust 1.58.1 released

Posted Jan 22, 2022 18:16 UTC (Sat) by epa (subscriber, #39769) [Link] (1 responses)

Symlinks were removed again in Plan 9, weren’t they?

Rust 1.58.1 released

Posted Jan 22, 2022 18:20 UTC (Sat) by willy (subscriber, #9762) [Link]

They were never in Research Unix as far as I know. 8th Edition incorporated bits of 4.1cBSD while 9th edition added things from 4.3BSD, but AFAIK plan9 never had symlinks in its heritage to rip out.

Rust 1.58.1 released

Posted Jan 23, 2022 15:58 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (6 responses)

I don't buy it. It seems to me that:

What Rust's standard library did here was clearly wrong and at least one person who looked at it had previously _told_ them it was wrong, but they hadn't appreciated that this means it's a grave security hole and must be fixed.

The code they had before 1.58.1 is a very obvious TOCTOU bug. I had no reason to ever look at this code, but if I had looked at it I'd think, huh, why isn't this a TOCTOU bug? And it looks like the answer is that the people writing didn't know any better. I'm pretty sure I'd have known this is a TOCTOU bug twenty years ago, and maybe even twenty-five years ago - there isn't something new and subtle here.

There are a lot of these subtle but scarcely "superhuman" problems to overcome in writing something like the standard library. Maybe Unix could deliver a better API that makes it harder to get wrong, but honestly I feel like a big part of this is just even knowing TOCTOU is a problem, and realistically if you don't know that you can't program systems that face the real world. Or at least you shouldn't.

It's notable that Rust calls out supported platforms where they can't fix this. Redox has the excuse of being a relatively small and obscure piece of software, (not an excuse to not fix it, but an excuse for not having already fixed it). What's Apple's excuse?

Rust 1.58.1 released

Posted Jan 24, 2022 11:55 UTC (Mon) by moltonel (subscriber, #45207) [Link]

It's worth noting that C++ stdlibs have the same vulnerability. Good luck getting a fix: the TOCTOU is known but the C++ spec just documents it as a source of UB, refusing to take responsibility. Apparently Go is vulnerable too, Python isn't, have fun checking other languages.

The same difference in attitude was seen with getenv/putenv last year : these APIs are inherently vulnerable, Rust created CVEs for them, while C and C++ just ignore the problem.

Rust 1.58.1 released

Posted Jan 24, 2022 18:38 UTC (Mon) by bwelling (subscriber, #13189) [Link] (4 responses)

The advisory says "macOS before version 10.10 (Yosemite)" is still vulnerable, so presumably Apple's excuse is the lack of a time machine to go back to before 2014.

Rust 1.58.1 released

Posted Jan 24, 2022 19:24 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

And said minimum requirement has been EOL for over 4 years, so anyone caring about *this* vulnerability probably has a few other things they should think about in their deployment anyways.

Rust 1.58.1 released

Posted Jan 25, 2022 0:17 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (2 responses)

That's an excuse for not fixing it now in 2022. It wasn't an excuse for not fixing it in 2014.

I haven't seen a preview of the apparently shocking revelations to come in jra's talk but TOCTOU symlinks bugs are super old. I'm pretty sure I first read about them on Bugtraq when I was an undergraduate, so that's the mid-1990s.

Rust 1.58.1 released

Posted Jan 25, 2022 2:30 UTC (Tue) by bwelling (subscriber, #13189) [Link]

The macOS change in 2014 was to add openat(), and the reason why the Rust fix doesn't work on older macOS is because openat() wasn't present. openat() wasn't present in Linux in the mid-1990s either, and was added to different systems at different times.

Also, the TOCTOU bugs are in applications, not the OS or the C library, so not something that can be fixed by anyone other the application developers.

Rust 1.58.1 released

Posted Jan 25, 2022 3:01 UTC (Tue) by jra (subscriber, #55261) [Link]

Not really shocking revelations, more of a realization that the problems are *not* fixable, and all the band-aids piled on band-aids in the POSIX API have created a horrible mess. We need a better way to deal with "feature" that serves mostly as a source of subtle security bugs for unwary application developers. I'm going to go into the details we had to consider to create a symlink-safe Samba on Linux, *BSD and others. Some of the problems we encountered are not obvious.

Rust 1.58.1 released

Posted Jan 22, 2022 4:00 UTC (Sat) by pabs (subscriber, #43278) [Link]

Perhaps they could be replaced with unprivileged bind mounts if those existed. Or porting the Hurd translator stuff to Linux.

Rust 1.58.1 released

Posted Jan 23, 2022 14:45 UTC (Sun) by ibukanov (subscriber, #3942) [Link]

No need to ban symlinks. Just make creation of one a privileged operation as it is on Windows.

Rust 1.58.1 released

Posted Jan 23, 2022 17:46 UTC (Sun) by ceplm (subscriber, #41334) [Link] (46 responses)

I am sorry, but that’s nonsense. Symlinks are currently so entrenched and useful, that you cannot remove them. That’s just destruction of users data as bad (or sometime worse) as rm -rf on some of them. I am pretty Linux-centric guy, but if Linux looses symlinks, that would be probably first real reason why to start thinking about switch to *BSD, because that would be suddenly less painful that continuing with Linux for me.

Rust 1.58.1 released

Posted Jan 23, 2022 20:49 UTC (Sun) by jra (subscriber, #55261) [Link] (45 responses)

Beware - where Linux leads, the *BSD's always follow. *BSD applications and libraries are not symlink-safe either.

Rust 1.58.1 released

Posted Jan 24, 2022 3:26 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (44 responses)

That's certainly not true; OpenBSD dropped /proc, for example. As much to the point, Linux won't lead there because it would lose Linux users, and many of them would be serious developers and sysadmins running major systems. Symlinks are simply too entrenched to toss them out.

From the other side, a TOCTOU using symlink attack? That strikes me as more of a theoretical attack than a real one on most systems, particularly the ones I use. Even for the high-security system I'm setting up professionally, if they get in far enough to set up a symlink, they've gotten too far, and at that point, I can't imagine a symlink attack being their next step. Most Unix boxes are single-user systems any more, and this is not the attack of concern on single-user systems.

(Oh, and may I repeat that C is fundamentally insecure, which has been known since the Morris worm took down a large part of the Internet in 1989? I see the rush, or lack thereof, to write Unix programs in a language that bound-checks arrays, the least that could be expected. And we're going to give up symlinks over a security concern? Not likely.)

Rust 1.58.1 released

Posted Jan 24, 2022 3:40 UTC (Mon) by jra (subscriber, #55261) [Link] (41 responses)

Apologies. OpenBSD does indeed go its own way. FreeBSD though, which is the most popular of the *BSD's certainly does follow Linux closely. I know as they work with us (Samba) to add the missing Linux features we depend on for performance and security.

You are also correct in that symlink attacks are largely theoretical, as there is so much lower hanging fruit to pick

But just *why* allow bad code using a horrible mangled set of APIs for a feature that has long outlived its usefulness.

Disks are really big now
Just move the data, don't symlink it.

Rust 1.58.1 released

Posted Jan 24, 2022 4:03 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (9 responses)

Whether or not it's outlived its usefulness, it's still used. Just looking at the root directory, Debian uses it to make it easier to switch the active (/vmlinuz) and backup (/vmlinuz.old) kernels without moving files around, and as a side effect I can tell what kernels are the active and backup by just using ls (e.g. "vmlinuz -> boot/vmlinuz-5.15.0-3-amd64").

Another example, much harder to replace, is the symlinking of /bin to /usr/bin. It allows for the collapse of the distinction between /bin and /usr/bin without anything with /bin or /usr/bin hardcoded in having trouble finding the files. I don't see any way to do this without ugly kernel patches.

Rust 1.58.1 released

Posted Jan 24, 2022 19:08 UTC (Mon) by jra (subscriber, #55261) [Link] (7 responses)

Admin-only symlinks fixes /bin -> /usr/bin. Symlinks are too entrenched to be removed completely, but restricting them to admin only removes applications having to deal with race conditions (if an admin did it, then it's meant to be :-).

Rust 1.58.1 released

Posted Feb 10, 2022 18:54 UTC (Thu) by nix (subscriber, #2304) [Link] (6 responses)

An alternative approach that breaks much less, probably nothing, while retaining the security benefits of root-only symlinks: non-root symlinks (or, more generally, symlinks with uids/gids not specified in a new file under /proc/sys somewhere: root only by default) are only followed by the user that created them.

Now hostile users can create all the symlinks they want, as can non-hostile users, and the use case the non-hostile users wanted (they can follow them) still works, while the hostile users find that only they are following the hostile symlinks they created, but nobody else can.

(Anyone still broken by this is probably using gid-shared directories for something, like it was the 1980s still. They're probably using ACLs too. A variant mode, off by default or on only in directories with the setgid bit active or perhaps which are gid-writable, which applies the same check based on the gid of the symlink instead of the uid, would suffice for that. I'm sceptical there are enough such systems around to worry about, though.)

Rust 1.58.1 released

Posted Feb 10, 2022 19:08 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (5 responses)

> An alternative approach that breaks much less, probably nothing, while retaining the security benefits of root-only symlinks: non-root symlinks … are only followed by the user that created them.

While that is *less* problematic than limiting symlinks to root, I am still certain that would break some cases where I have used symlinks in practice. Perhaps we could just leave symlinks *alone* and fix the buggy applications instead?

For example: Say I have "$HOME/bin" symlinked to "$HOME/.local/bin" (true story) and I want to run a script located, according to the $PATH variable, at "$HOME/bin/somescript" as root via sudo. Your scheme breaks this since the bin → .local/bin symlink was created by my non-root user and needs to be followed by root to execute the script. Or more generally—say I just want to run a command as root on some file which has a symlink (which I created) somewhere in its path. Or perhaps I have versioned directories with a "latest" symlink and I want to share the path to the latest version with another user. These are all perfectly legitimate, real-world use cases which absolutely depend on the ability to create symlinks as a non-root user which will be followed by other users.

Rust 1.58.1 released

Posted Feb 10, 2022 20:58 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (4 responses)

>Perhaps we could just leave symlinks *alone* and fix the buggy applications instead?

If fixing bugging applications were an easy way out, we wouldn't be having the conversation.

Rust 1.58.1 released

Posted Feb 10, 2022 22:56 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (3 responses)

> If fixing bugging applications were an easy way out, we wouldn't be having the conversation.

I never said that it would be *easy*. Finding and eliminating bugs rarely is. You could say the same about any change to symlink semantics, of course, and any additional bugs that might introduce above and beyond *planned* loss of functionality.

Rust 1.58.1 released

Posted Feb 10, 2022 23:07 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

There in lies the issue. The suggestion lacks the acknowledgment of this. If fixing bugs in the applications hasn’t worked well for so long it may very well be time to figure out what kind of mitigation strategies are possible. I will await a detailed proposal before dismissing it.

Rust 1.58.1 released

Posted Feb 11, 2022 6:52 UTC (Fri) by nybble41 (subscriber, #55106) [Link] (1 responses)

> If fixing bugs in the applications hasn’t worked well for so long it may very well be time to figure out what kind of mitigation strategies are possible.

To begin with, most applications don't need any mitigation. They just open certain paths and have no reason to care whether there were symlinks involved. If an application doesn't employ path-based access control or traverse a directory tree then symlinks probably don't affect it at all. (The issue with the Rust library function was of the latter type.) *Samba* has a particular problem with symlinks (among other things) because it's trying to implement non-native VFS semantics, validating requests in userspace before handing them off the the kernel, which is obviously going to be a rich source of TOCTTOU bugs—this, understandably, is the cause of jra's obsession. Personally I don't consider "symlinks don't mesh well with Windows-oriented network filesystem protocols" to be a good reason to cripple a very useful feature which has been part of Unix-based systems since the early days, but a Samba developer will naturally have a different perspective due to a narrow focus on the needs of their software.

Identifying the issue is easy: If your program has code like "if (check(path)) { operate_on(path); }", that's a bug. It may be a *tolerable* bug, depending on how the program is used, but it's vulnerable to changes in the filesystem between the condition and the operation. This is a classic pattern and not really filesystem-specific, much less specific to symlinks.

The mitigation strategy for filesystem TOCTTOU bugs is well-known but tedious to implement: consistently use handles rather than paths in every situation where you need to perform multiple operations on the same filesystem object. If given a path as input only use it once, to obtain the handle. Every operation after that should use the *at() family of functions, ensuring that it works on the original object even if something changes which affects path resolution—which is not just symlinks but also regular link/unlink/rename operations, changes in mount points, etc. This is exactly how the issue was mitigated in Samba, though it took several years to complete the conversion since path-based interfaces were so deeply ingrained.

Rust 1.58.1 released

Posted Feb 11, 2022 14:12 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> To begin with, most applications don't need any mitigation

We aren't talking about most applications, only the ones that are affected. If a few high profile applications get affected, that's enough to cause a large impact. It's never been about the majority.

> The mitigation strategy for filesystem TOCTTOU bugs is well-known but tedious to implement

Hence, the desire to have other mitigation strategies.

Rust 1.58.1 released

Posted Jan 24, 2022 20:26 UTC (Mon) by khim (subscriber, #9252) [Link]

> I don't see any way to do this without ugly kernel patches.

bind mount would solve that issue easily. But then, if you would give random user the ability to bind mount stuff easily…

Symlinks-as-admin-tool sounds like a nice compromise: most of the issues with symlinks are with TOCTOU and if normal users can't create them it's not an issue.

Rust 1.58.1 released

Posted Jan 24, 2022 4:09 UTC (Mon) by sfeam (subscriber, #2841) [Link] (4 responses)

Just move the data, don't symlink it.

I am quite puzzled by that suggestion. I have been using symlinks for a very long time now, but I am having trouble thinking of a case where the reason was to avoid moving data. In my experience the primary use for symlinks is to avoid having multiple copies of something, often a configuration file, because the various subsystems that want to read it can't get their act together and agree where it belongs in the system file structure. They are good practice and benefit security for the same reasons that having a single shared library is better than having fifteen copies of the same library. The same arguments apply to user data shared across multiple accounts, especially where the symlink points to a remote disk. Could you clarify what problem "move the data" is solving?

Rust 1.58.1 released

Posted Jan 24, 2022 19:21 UTC (Mon) by jra (subscriber, #55261) [Link]

Back in my dim and distant past doing UNIX system administration, most symlink use was to stitch together datasets that had grown too large for a disk without having to restructure paths for an application.

I think bind-mounts (or admin-only symlinks) would now serve the same purpose.

Rust 1.58.1 released

Posted Jan 24, 2022 20:27 UTC (Mon) by karkhaz (subscriber, #99844) [Link] (2 responses)

> In my experience the primary use for symlinks is to avoid having multiple copies of something, often a configuration file, because the various subsystems that want to read it can't get their act together and agree where it belongs in the system file structure

Another very important use for symlinks: there are far fewer things you can do atomically with a directory, compared to files. But if you instead have a symlink to a directory, you can do all the usual atomic stuff (delete/move a new directory on top of it/etc) that you can with files. Making symlinks admin-only would break a lot of legitimate usecases that are difficult to accomplish elsehow.

Rust 1.58.1 released

Posted Jan 25, 2022 0:50 UTC (Tue) by jra (subscriber, #55261) [Link] (1 responses)

Many of these things "move a new directory on top of it" are actually the same as symlink security exploits. So I'm not too concerned.

Rust 1.58.1 released

Posted Feb 20, 2022 15:18 UTC (Sun) by nix (subscriber, #2304) [Link]

Butttt... that means that unless you're planning to ban renaming of directories or mounting of directories on top of other directories, surely your crusade against symlinks is pointless, because all the same vulnerabilities remain?

Or perhaps I totally fail to understand. (Likely I do, becasue that position seems incoherent.)

Rust 1.58.1 released

Posted Jan 24, 2022 13:40 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (25 responses)

> Just move the data, don't symlink it.

Umm. How would you suggest that I replace my use of symlinks to make sure that build directories for software are:

- all in one place (to ease backup exclusions and cleanup if/when disk space is needed); and
- still easily accessible from the source tree?

Directory hardlinks could work, but those sound…even worse.

Rust 1.58.1 released

Posted Jan 24, 2022 16:18 UTC (Mon) by nybble41 (subscriber, #55106) [Link] (21 responses)

Not to mention that if you removed symlinks (to files!) you would completely break the configuration interface for systemd. Even just restricting symlinks to admins would break the configuration of user sessions. Symlinks also appear in essential roles in core interfaces like /sys.

Removing or restricting symlinks is a complete non-starter. They're too tightly integrated—and too useful—to eliminate. Just fix the apps and libraries which don't handle them properly.

Rust 1.58.1 released

Posted Jan 24, 2022 19:11 UTC (Mon) by jra (subscriber, #55261) [Link] (20 responses)

> Not to mention that if you removed symlinks (to files!) you would completely break the configuration interface for systemd.

Admin-only symlinks fixes this.

> Even just restricting symlinks to admins would break the configuration of user sessions.

Can you explain more ?

> Symlinks also appear in essential roles in core interfaces like /sys.

/sys then is a filesystem that allows symlinks.

> Just fix the apps and libraries which don't handle them properly.

Can't be done by mortal developers. How many years now have we been trying ? If it were possible we'd have done it already by now.

Rust 1.58.1 released

Posted Jan 24, 2022 19:28 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (2 responses)

While I understand your goals (I, too, would *love* to just remove some features of platforms I work on to make my life easier), can we add "remove NULL from programming languages", "NUL-terminated strings" to this wish list of things we're getting rid of that just can't be handled "properly" by "mortal developers"? :)

Rust 1.58.1 released

Posted Jan 24, 2022 19:39 UTC (Mon) by jem (subscriber, #24231) [Link]

>can we add "remove NULL from programming languages", "NUL-terminated strings" to this wish list of things we're getting rid of

Your comment is funny considering the parent article is about Rust.

Rust 1.58.1 released

Posted Jan 24, 2022 19:44 UTC (Mon) by jra (subscriber, #55261) [Link]

As another person already remarked, Rust was created exactly to fix these problems in C that "just can't be handled "properly" by "mortal developers" :-).

So we're already there on the language front. Now let's fix the filesystem API !

Rust 1.58.1 released

Posted Jan 24, 2022 20:10 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> Admin-only symlinks fixes this.

Hmm. systemd user services exist. GNOME is already using it by default and KDE, other projects are looking into this as well. So I am not sure how admin only symlinks are viable.

Rust 1.58.1 released

Posted Jan 24, 2022 20:30 UTC (Mon) by jra (subscriber, #55261) [Link]

That's why I think the solution is to be able to designate filesystems as only allowing "admin only symlinks" on mount (or bind-mount). That way you can isolate applications working in those areas and still leave the horror that is user created symlinks in the rest of the system (thus not disturbing things like systemd user services).

Rust 1.58.1 released

Posted Jan 24, 2022 22:05 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> Admin-only symlinks fixes this.

I'm using symlinks in all my projects in C++ to link build directories into the build cache, for example. And to link RPC schemas from a central module into consumers. There are no problems with symlinks when they are done within the same permission boundary.

Perhaps Samba should stop abusing the filesystem for its file server access control needs and grow its own storage?

Rust 1.58.1 released

Posted Jan 25, 2022 0:43 UTC (Tue) by jra (subscriber, #55261) [Link] (5 responses)

When we implement SMB2+UNIX extensions in Samba I am planning to store requests to create symlinks not as symlinks, but as an EA attached to a file and marked as a Windows reparse point (which is what Windows does internally for symlinks created via their NFS server). Hopefully this satisfies your request to stop abusing the filesystem :-).

Rust 1.58.1 released

Posted Jan 25, 2022 0:55 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Sure. It actually seems to make more sense for Samba. Does the Linux SMB2 client still sees that as a symlink?

Rust 1.58.1 released

Posted Jan 25, 2022 1:50 UTC (Tue) by jra (subscriber, #55261) [Link] (3 responses)

Yes, the Linux client will get a STOPPED_ON_SYMLINK message when trying to traverse through a symlink, then it will have to open the path and fetch the reparse data to get the symlink contents. On Windows, "This Is The Way" (with apologies to the Mandalorian :-).

Rust 1.58.1 released

Posted Feb 10, 2022 19:49 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

Of course, the problem with doing this is that the native filesystem doesn't work the same way, so samba becomes unsuitable for use as a Linux->Linux fileserver. (Which it already is, but only because the Unix extensions need an ancient version of SMB which is rightly not enabled by default.)

Of course this is a distinctly niche usecase and I am weird as all hell for even considering it. :)

Rust 1.58.1 released

Posted Feb 10, 2022 22:56 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

It's only weird because the choice for a "mainstream" system is Windows or Linux. I'm looking at this through the eyes of someone who started with a completely different sort of filesystem.

No mounts, no symlinks, sensible ACLs, ... I don't know whether or how it would have suffered from this sort of problem, but they do sound weird ...

Networking was nice and transparent - the only problem was that volume names needed to be unique across the network, and with a max length of six characters naming could be a problem :-) - but you just declared your disks at boot, and a declaration of "<LONDON> on westminster" meant that the volume LONDON would appear whenever system westminster was running.

Cheers,
Wol

Rust 1.58.1 released

Posted Feb 20, 2022 15:29 UTC (Sun) by nix (subscriber, #2304) [Link]

As someone who got completely hooked on the flexibility of the Unix system model from the moment he first encountered it, declaring your disks at boot sounds utterly dreadful. I mean, in a world before hotswap disks and USB storage it was probably harmess, but still I'm used to being able to say "oh dammit I'm out of space on $FS: bind mount in a new one and/or NFS-mount more storage from somewhere else, and the problem will be solved in seconds". This works even if $FS being out of space would break booting.

(This also saved my bacon in the past. When your main fileserver suffers multiple disk failures and its RAID is left degraded -- bloody awful 'enterprise' hard disks, shorter lifespan than any other disk I've ever used including cheap flash keys -- it's damn useful to be able to kick up a network block device on another system and remotely mount a spare block device of the right size to get your RAID un-degraded enough to be slightly safe for operation while you wait for another disk. Declaring your disks at boot is somewhat incompatible with that sort of emergency madness. Hardware RAID would probably hate it too. All praise to md etc.)

Rust 1.58.1 released

Posted Jan 25, 2022 0:12 UTC (Tue) by nybble41 (subscriber, #55106) [Link] (7 responses)

>> Even just restricting symlinks to admins would break the configuration of user sessions.
> Can you explain more ?

The same symlink-based interfaces systemd uses for system-level configuration are also used for user sessions and user services with configuration files and symlinks under ~/.config/systemd/user/, which are meant to be controlled by regular users, not admins. Remove the symlinks and you break KDE and Gnome login sessions, among other things. Besides systemd, Chrome and Firefox both keep symlinks in their configuration directories. Stack uses them in its versioned GHC installation. Steam needs them for its runtime (libraries, among other things). Wine uses symlinks in its user configuration directories (for drive letters, among other things). And of course there are many other less notable examples. All of these programs need symlinks which can be created and edited by ordinary users in their home directories.

>> Symlinks also appear in essential roles in core interfaces like /sys.
> /sys then is a filesystem that allows symlinks.

You can't usefully distinguish between filesystems which allow symlinks and filesystems which do not (for security reasons). /sys is well-bounded but your Samba shares may well be on the same filesystem as users' home directories, and you can't realistically limit symlinks there. Or in /etc (e.g. the Debian /etc/alternatives/ system is all based on symlinks, as is sysvinit for packages still using that). Or in the root (/bin -> /usr/bin) or in /usr (alternatives, aliased commands). And let's not forget the home directories, which are commonly exported as Samba shares. So Samba will just have to deal with it and support symlinks properly no matter where they may be found.

> Can't be done by mortal developers.

The fact that this issue in the Rust standard libraries has been resolved says otherwise. No one can guarantee perfectly bug-free code, of course, but it *is* possible to handle symlinks correctly. This CVE was never really a symlink issue in the first place. The minor symlink issue was easily dealt with in the original code by not following symlinks to directories. That part worked fine in a single-user, unshared environment. However, you can't just say "if (is_symlink(path)) { unlink(path); } else { recurse(path); }" in a shared filesystem without accounting for the fact that the entry at "path" can change. This is basic concurrency, made worse by the fact that you're sharing state with less privileged code. The same kind of bug could easily occur in other areas having nothing to do with symlinks, such as access checks. Developers have all the tools needed to avoid this issue (openat, fstat, unlinkat, fdopendir), at least on Linux—they just need to use them. As the new implementation does.

Rust 1.58.1 released

Posted Jan 25, 2022 0:40 UTC (Tue) by jra (subscriber, #55261) [Link] (3 responses)

> So Samba will just have to deal with it and support symlinks properly no matter where they may be found.

We do now. Most applications don't. I'm including some system-level security libraries in that description. It would be nice to have a sandbox where user created symlinks are disallowed in order to run such applications safely.

> The fact that this issue in the Rust standard libraries has been resolved says otherwise.

I respectfully disagree.

https://www.cvedetails.com/google-search-results.php?q=sy...

6,760 results. Yes, some of them are for Samba. Hopefully not anymore.

Rust 1.58.1 released

Posted Jan 25, 2022 2:01 UTC (Tue) by bmgordon (subscriber, #16077) [Link] (2 responses)

> We do now. Most applications don't. I'm including some system-level security libraries in that description. It would be nice to have a sandbox where user created symlinks are disallowed in order to run such applications safely.

Does MNT_NOSYMFOLLOW do what you're asking for? A mount with that flag can still contain symlinks, but they won't be followed. Chrome OS uses this when mounting partitions that can contain user-modifiable data, since it already doesn't have a history of allowing non-root users to create symlinks.

Rust 1.58.1 released

Posted Jan 25, 2022 2:41 UTC (Tue) by jra (subscriber, #55261) [Link] (1 responses)

I did not know about MNT_NOSYMFOLLOW ! When was that added ? This is why I love lwn :-).

Rust 1.58.1 released

Posted Jan 25, 2022 16:44 UTC (Tue) by bmgordon (subscriber, #16077) [Link]

> I did not know about MNT_NOSYMFOLLOW ! When was that added ?

I'm pretty sure it showed up in kernel 5.10 and util-linux 2.36.1.

Rust 1.58.1 released

Posted Jan 25, 2022 1:02 UTC (Tue) by jra (subscriber, #55261) [Link] (2 responses)

> No one can guarantee perfectly bug-free code, of course, but it *is* possible to handle symlinks correctly.

Tell that to the authors of:

Samba
AutoVNF tool for the Cisco Ultra Services Framework
chkstat
salt (config system)
TP-Link Archer
mariadb
PHP
FortiClient for Mac
Google Chrome
zsh
portupgrade (FreeBSD)
mount.cifs
XFree86
Gnome Glib
npm
audiolink
systemd
cups
pstopdf
Mercurial
tin
sudo
docker
pear
exmh
gcc
Emacs
AMD fglrx-driver
deepin
glint
xmcd
Tarantella Enterprise
CDE ToolTalk
pine
pico
GNU screen
postfix

and on, and on, and on (found by a casual perusal of the CVE site). But I expect you'll say all these authors just needed to be "more careful" :-).

When does a feature become too costly to support anymore ?

Rust 1.58.1 released

Posted Jan 25, 2022 16:54 UTC (Tue) by nybble41 (subscriber, #55106) [Link] (1 responses)

> Tell that to the authors of: Samba …

Are you going to list all the other non-symlink CVEs? Perhaps we should just give up on writing software altogether since no one writes bug-free code.

The fact that some projects have handled symlinks incorrectly in the past does not imply that it can't be done correctly.

> But I expect you'll say all these authors just needed to be "more careful" :-).

Well, yes. Most CVEs can be traced back to someone not being careful enough with their program's input, or not fully accounting for concurrency. If you're writing code which needs to accept input from, and perhaps share mutable data with, other untrusted processes then you need to be very careful indeed. There are lots of ways to mess this up. Symlinks are just one minor aspect to consider among many.

Rust 1.58.1 released

Posted Jan 25, 2022 21:57 UTC (Tue) by jra (subscriber, #55261) [Link]

> Are you going to list all the other non-symlink CVEs?

No. My point was all these CVE's happened because of one filesystem "feature". That amount of damage means this is a feature that should be removed IMHO.

They are a security problem that doesn't need to happen. That's my point. The existence of symlinks has mangled the POSIX filesystem API beyond repair trying to deal with their damage. I would like to fix that (OK, so not exactly "beyond repair" :-).

Rust 1.58.1 released

Posted Jan 24, 2022 19:19 UTC (Mon) by jra (subscriber, #55261) [Link] (2 responses)

Your convenience for building software does not outweigh the havoc that unrestricted user created symlinks create for application developers and the security of the applications they create (most of whom do not consider or think about symlink race conditions. At *all*).

Rust 1.58.1 released

Posted Jan 24, 2022 22:06 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

There are no problems whatsoever if application data is isolated and the shared data is accessed only through well-defined interfaces.

We should just stop pretending that unrestricted code can ever be secure and start moving to sandboxes.

Rust 1.58.1 released

Posted Jan 25, 2022 0:41 UTC (Tue) by jra (subscriber, #55261) [Link]

+1 from me. What an insightful comment !

Rust 1.58.1 released

Posted Jan 24, 2022 13:39 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

> Most Unix boxes are single-user systems any more, and this is not the attack of concern on single-user systems.

Actually, most Unix boxes are multi-user given that Android uses per-app users. Even if that weren't the case, is this any reason to ignore the case once the vulnerability is found? And even without Android, there are still shared systems out there (HPC, university course-provided clusters, etc.).

Rust 1.58.1 released

Posted Jan 24, 2022 19:27 UTC (Mon) by Wol (subscriber, #4433) [Link]

And how many systems are multi-user but not multi-head? That's possibly (maybe even probably) true of a lot of PCs now - if there's just the one family PC. Certainly our PC is (not as often as it was) regularly shared.

Cheers,
Wol