Vetting the cargo [LWN.net]

Vetting the cargo

Posted Jun 23, 2022 9:42 UTC (Thu) by Vipketsh (guest, #134480)
In reply to: Vetting the cargo by farnz
Parent article: Vetting the cargo

> The trouble with a standard library is that it's ossified - you can't change it easily. The benefit is that it's a human-curated set of "good default choices" for ways to do things. Keeping the standard library small is absolutely the right decision for Rust with its backwards compatibility goals, but means that you need to look elsewhere for the human-curated set of "good default choices" - and it needs the human curation to avoid the problem of getting stuck on a bad default that a standard library has.

Way to go passing the buck. Nothing magical happens when the adjective "standard" is attached to "library": if you have to depend on a library which then goes away or changes in an incompatible way you are the same screwed irrespective of whether it was "standard" or not.

Translated the Rust tactic seems to be*:
1, Divide your users amongst different libraries (implementations)
2, All those libraries will inevitably be replaced or change in an incompatible way (by design is the implication I'm getting from the parent comment)
3, Since it is not "Rust" that is changing Rust bears no responsibility (thus Rust always has good backwards compatibility)
4, It is actually the user's fault for "choosing the wrong library" (though no information is given on what should have been used)
5, Since the users are divided there are never enough voices for the problem to be heard

The end result is that everyone ends up just as screwed as if all this functionality was in a (changing) standard library.

*: I doubt that it is intentional by the Rust team, but that doesn't change the end result the way I see it.

to post comments

Vetting the cargo

Posted Jun 23, 2022 10:57 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

The lines up pretty well with Linux's "stable userspace ABI" and the mess of the userland ABI story above the absolute basics (glibc, X11, and a handful of others).

Such reimpls have happened before (cf. libjpeg-turbo). I think the difference is that the ecosystem has a lot more room to move than the guarantees of the stdlib because of the "if it works today, it'll work tomorrow" mindset. And yes, your crate will continue to work because the old crates are still accessible but you might not have updates.

The Rust internals Discourse definitely catches features which lay out semver hazards (or at least I try to point them out) so that library authors are not caught unaware when they make some otherwise innocuous change that their major number needs bumped. However, it isn't really possible to wrap everything in bubblewrap.

Vetting the cargo

Posted Jun 23, 2022 13:14 UTC (Thu) by Wol (subscriber, #4433) [Link]

> The lines up pretty well with Linux's "stable userspace ABI" and the mess of the userland ABI story above the absolute basics (glibc, X11, and a handful of others).

The obvious (and quite likely broken) way for linux to deal with this is to refactor the obsolete ABI into a module. Said module then gets (mostly) abandoned, and dropped when nobody notices.

Of course, the problem with this is when it's core functionality that can't (practicably) be modularised.

Cheers,
Wol

Vetting the cargo

Posted Jun 23, 2022 13:27 UTC (Thu) by farnz (subscriber, #17727) [Link] (6 responses)

Your conclusion doesn't follow from your argument.

The standard library cannot change in a backwards-incompatible fashion; this is simply not permitted under any circumstances. You can add it it, and you can fix bugs, but you can't break existing users. Further, as you update your language environment, you will get newer implementations of the standard library; one of the defining special cases about the standard library is that the compiler is allowed to assume that the standard library is present (which is why Rust currently has three standard libraries - core, which is things the compiler cannot function without, alloc, which is all of core, plus the things that the compiler needs when the heap is available, and std, which is the full-fat standard library when running in userspace on a normal OS).

This means we cannot remove anything from the standard library, ever. No matter how big a mistake it is, it has to remain present in the standard library forever, acting as a foot-gun for new users (see gets in C, for example), because removing it will break backwards compatibility. And because Rust documents all of its standard library, that means that it's visible to new users whether we want them to use it or not.

It's worth noting, too, that just because it's in an external crate doesn't mean it stops working; you can still use error-chain or failure, and the backwards compatibility commitment of the compiler and standard library means that they will still work into the future. They're just not being improved over time, and the ecosystem as a whole is no longer using them in new projects.

Vetting the cargo

Posted Jun 23, 2022 14:16 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

Note that Rust does have one handle to pull back an API: unsoundness. If an API cannot be used soundly, it is eligible for deprecation and eventual removal (this may involve an edition change) because an API just cannot be used properly. For example `std::mem::uninitialized()` is off limits because `let b: bool = std::mem::uninitialized();` is instant UB because a `bool` may only have a binary value of 1 or 0 (that is, it may appear to be neither `true` nor `false` with some other bit pattern). Now there is the `std::mem::MaybeUninit` type to handle the use case.

Vetting the cargo

Posted Jun 23, 2022 15:40 UTC (Thu) by Wol (subscriber, #4433) [Link] (4 responses)

> This means we cannot remove anything from the standard library, ever. No matter how big a mistake it is, it has to remain present in the standard library forever, acting as a foot-gun for new users (see gets in C, for example)

Then create a fourth standard library - TheFjords. All your footguns go in that library, and the linker complains loudly when it has to pull it in.

Users won't notice (unless they look), but developers will see it, wonder what the hell is going on (especially if cluesers complain it's there), and will hopefully eliminate it from their code.

So we have an effective method for removing dead/dangerous calls from actively maintained code. If the code isn't maintained then there's not a lot anyone can do, and that's true of all ecosystems ...

Cheers,
Wol

Vetting the cargo

Posted Jun 23, 2022 18:13 UTC (Thu) by farnz (subscriber, #17727) [Link] (3 responses)

The library name is part of the import path used in code. If something moves from (say) core to alloc, code that is written for just core (and does not have alloc or std) is broken and stops compiling. Because Rust does not want to cause working code to break just because you update to a new compiler, code can only move one way - from std to alloc to core - and even then, only because std re-exports everything from alloc and core in the places it "ought" to be in if there was only std, and alloc re-exports everything from core in the places it "ought" to be in if there was only core.

So a "TheFjords" standard library isn't helpful - it just means that it's an obvious fix for things that have been broken by the change, but there's not supposed to be breakage to begin with. Putting things in external crates that get deprecated is helpful - nothing breaks if you still use error-chain, but you're making more work for yourself than if you use thiserror and anyhow, and you can do that refactor at a time to suit you - rather than being forced to do it because you're trying to compile for a new architecture and need a new compiler that supports the new architecture.

Vetting the cargo

Posted Jun 23, 2022 23:18 UTC (Thu) by Wol (subscriber, #4433) [Link] (2 responses)

But that's a "break on build", no? So it'll break FOR A DEVELOPER?

What we don't want is for a system upgrade to break a working program. But if a system upgrade means you can no longer re-build a program without being forced to add TheFjords to the build process, then sorry. That's exactly what I want to achieve!

It's a (reasonably) simple fix, but it places developers on notice that there are problems. So my takeaway from what you say is that "a working binary will continue working, but if it has obsolete components it will no longer build". So if it's unmaintained the user isn't affected, and if is maintained the maintainer has to fix it ...

Cheers,
Wol

Vetting the cargo

Posted Jun 24, 2022 0:47 UTC (Fri) by mpr22 (subscriber, #60784) [Link]

> So it'll break FOR A DEVELOPER?

No, it will break for any person building the program.

You don't need to be a Rust developer to build a Rust program.

Vetting the cargo

Posted Jun 24, 2022 10:04 UTC (Fri) by farnz (subscriber, #17727) [Link]

I prefer the experience offered by RustSec - if you don't care about it, don't run the tool. If you do care, run the tool (cargo audit), and things like this deprecation warning will appear and tell you that you've got a risky dependency and why. If the crate comes back into maintenance, then the advisory gets updated to not trigger on new versions; similarly if recommended replacements get added, it'll be updated to tell you about them (when I first looked at RUSTSEC-2020-0036, it did not mention fehler, which has been added to the list).

This is something that's easier to do with external crates than with the standard library, because all the tooling already exists to do it, and you just don't run it if you don't care about it - a user who just wants to build the version of a program they had 5 years ago to re-generate some results can do that - while a developer who cares about security can run the tools and find out if they need to do anything (and if so, what).