RIIR [LWN.net]

RIIR

Posted Feb 12, 2025 20:41 UTC (Wed) by da4089 (subscriber, #1195)
Parent article: Rewriting essential Linux packages in Rust

While I appreciate Rust as a language, and the idea that in order to maintain things in the future when no one knows C they need to be evolved to newer languages, my most powerful reaction to all the rewrite-it-in-Rust efforts is dismay at the loss of innovation.

As stated, these 55 year old tools are remarkably bug free, have very few dependencies, and continue to work as designed. Their maintenance burden is not high.

But instead of putting effort into discovering new better approaches to tools (and operating systems) we instead spend the innovative potential of a generation to get back to where we started, even losing some features in the process.

We did this already through the 90’s and 00’s to extract our toolset from corporate stasis and sharding. Is doing it again to replace C with Rust really the best use of our effort’s potential?

I don’t mean this as a personal attack, nor as any a criticism of the quality and dedication involved. But it feels to me like an inward turning of the collective vision that quite dismays me.

to post comments

RIIR

Posted Feb 12, 2025 21:04 UTC (Wed) by kleptog (subscriber, #1183) [Link] (4 responses)

> But instead of putting effort into discovering new better approaches to tools (and operating systems) we instead spend the innovative potential of a generation to get back to where we started, even losing some features in the process.

They're replicating the results of those 55 years of development in few years, so what "innovative potential" is being lost here? Now you have a sound base to continue building, except every year of effort put in the Rust version is equivalent to multiple years of effort on the C version.

I don't understand this idea that these tools should have minimal dependencies. Why should "sort" implement its own sorting algorithm instead of using the battle-tested UltraSpeedParallelSuperSorting crate that is used by everyone else because it sorts blazingly fast?

We can build great programs because we are standing on the shoulder of powerful libraries. Re-use is almost always better than writing it yourself.

Let's face it: C programmers aim for minimal dependencies because dependencies in C programs are really difficult. In more modern language dependencies are one line in your package configuration and you're done.

RIIR

Posted Feb 12, 2025 23:56 UTC (Wed) by dvdeug (guest, #10998) [Link] (3 responses)

> In more modern language dependencies are one line in your package configuration and you're done.

But they're not, for many reasons.

Many programmers have used a cool new library and discovered that it was no longer supported after a few years and wouldn't build with newer versions of other dependencies, and had to replace it. One can recall the left-pad catastrophe, where one author withdrew a 17-line package from npm and thousands of programs stopped building. One can also remember the endless chase of libgtk versions.

You lose control over supported platforms. E.g. the Python-Cryptography package started using Rust, and now no longer works on many of the platforms it used to. Coreutils compiles on HPUX 11 on hppa and ia64. Even if Rust supported those, you're still depending on every dependencies you use to support those. Good luck with your x86 box when your dependency pivots to using Oxide, the programming language that's going to replace Rust.

The bug envelope changes. If sort implements its own sorting algorithm, then if it breaks, you look at the sort code, if necessary checking that it works on the same libc and gcc version. If it uses the UltraSpeedParallelSuperSorting crate, then you have to look at that crate, and any crates it uses. Coreutils supports a variety of C libraries and C compilers (even with the restriction to C99); it's easy to prove that any new bugs are due to your changes. You can say that this crate is used by "everyone else", but you'll end up using a crate that does what you need but not everyone else needs, or using a crate in a way that other people aren't, triggering bugs that nobody else is meeting.

That was benign bugs, too. The Jia Tan attack was made possible by the fact that libxz was linked into ssh. To make it worse, it wasn't even code that ssh used; it was unused code from using another dependency to support systemd. Every dependency adds security risk, and when they're built in this fashion themselves, they bring in more security risks with their dependencies.

If you're writing a KDE program or GNOME program, both come with an array of libraries you can solidly depend on, no problem. A major frontend program can pull in a key dependencies without issue; most people get Gimp or Darktable through their distro or Flatpak anyway. (But note that this idea that dependencies have no cost amplifies the problem. Installing a C library that depends on nothing or a few standard dependencies is low cost. Installing a dependency that depends on a host of other dependencies amplifies most of the problems above.)

But coreutils? Programs that are frequently run as root, are called from shell scripts all over the system, and thus are security critical? Programs that are critical to running on every system? Be it Rust or C, they should carefully consider every dependency.

RIIR

Posted Feb 13, 2025 9:41 UTC (Thu) by taladar (subscriber, #68407) [Link] (1 responses)

What I don't like about this argument is that it treats writing your own version of the dependency inside your own code base as if it was free and had no associated risks when that is far from true.

Your own internal version is very likely to be orders of magnitude less tested, optimized, maintained,... than even a moderately used third party dependency. It is also less likely to see the same scrutiny from security researchers or the same amount of tracking from security tracking systems like CVE or GHSA.

The number of programmers who know how it works will be lower. When it isn't touched for a while nobody (including the original author) will know how it works or how to change it. You will have more trouble to find people to add cross-platform support because they will need to use exactly your program, not just any one of the programs using it.

Also, what you say about the bug envelope doesn't work at all. If there is a bug in the sorting algorithm you wrote, you have to find it and invest time that will only benefit that one project. If the bug is in a dependency you have a good chance someone else already found it and even if not, your effort to find and fix it benefits everyone using that library.

You look at it from the perspective of comparing a Rust dependency that is maybe 5 years old with some internal version in some 50 year old C tool and there the cross platform support will likely come out on top because cross platform used to be so much more important in the bad old days when nothing was standardized and the hardware and OS landscapes were more fractured in terms of what baseline features you could expect from your hardware or OS but honestly, reusable and shared code is the better choice in the long term.

RIIR

Posted Feb 21, 2025 2:02 UTC (Fri) by dvdeug (guest, #10998) [Link]

> Your own internal version is very likely to be orders of magnitude less tested, optimized, maintained,...

If we're talking about coreutils, no, it's not. That's the context of this.

> When it isn't touched for a while nobody (including the original author) will know how it works or how to change it.

Good code is understandable. Not to mention, code that does what you need is sometimes easier to understand than interfacing with a powerful library with many features and options.

> Also, what you say about the bug envelope doesn't work at all. If there is a bug in the sorting algorithm you wrote, you have to find it and invest time that will only benefit that one project. If the bug is in a dependency you have a good chance someone else already found it and even if not, your effort to find and fix it benefits everyone using that library.

If there's anyone else using that library. If everyone else hasn't already moved on to the next big thing. If you didn't pick the library that was better, but never hit critical mass and the maintainer dropped it. If you aren't calling with a set of options that nobody else is using.

There are libraries like libz and libpng where there's little reason not to include. But so many libraries, crates, packages, etc. are created, uploaded, maybe even get a few releases, then get dropped. If you're writing coreutils in Rust and hope to replace the existing coreutils, you need to plan for a 50 year existence, most of which is going to spent doing minor changes on programs that work.

Cross platform is a choice. But Debian won't switch until it supports all release platforms, of which there are nine, and failing to support GNU/Hurd or several other semiactive ports is going to lose you some support. Sell to your audience.

> reusable and shared code is the better choice in the long term.

Making a tight program that runs where you need it to and runs even when other stuff is broken is incredibly useful. We have busybox with 0 dependencies, because coreutils today has too many dependencies in some cases. Reusable code is good, but again Jai Tan got in because ssh was being linked against a library which linked against a library that ssh didn't need. You've never had to deal with an upgrade mess when dealing with an incompatible ABI break at the base of a tree of dependencies. Or having to change a program to work with the new version of the library, because the old version of the library has an entirely different API but isn't getting bug fixes any more.

Again, you're writing a GNOME program, go wild with GNOME libraries. You're writing a program that needs a solid solver for 3-SAT, link in z3. You get a choice between writing 5 lines of code using a hash-map that's fast enough, or linking in a custom data structure that is optimized... if it's code for mass distribution and hopefully a long life, save yourself some headache down the road and write the 5 lines of code.

RIIR

Posted Feb 15, 2025 5:53 UTC (Sat) by marcH (subscriber, #57642) [Link]

> > In more modern language dependencies are one line in your package configuration and you're done.

> But they're not, for many reasons.

Indeed not.

https://cacm.acm.org/practice/surviving-software-dependen...
https://medium.com/@sdboyer/so-you-want-to-write-a-packag...
etc.

To be fair, it is much worse with C/C++ because then you have both the technical _and_ the less technical issues.

RIIR

Posted Feb 15, 2025 6:06 UTC (Sat) by marcH (subscriber, #57642) [Link] (9 responses)

Imagine someone invents some new plumbing material that never dissolves harmful pollutants, is cheaper to manufacture, easy and cheap to work with, does not burst when freezing and lasts forever. But in the daily life of the people using water, it makes absolutely no difference[*]. So, no innovation? I think plumbers and house builders would have a very different opinion.

[*] never having to call a plumber and not having cancer is huge but makes no "obvious" difference. And we can rather rarely tell where we got cancer from.

RIIR

Posted Feb 15, 2025 12:04 UTC (Sat) by pizza (subscriber, #46) [Link] (8 responses)

> Imagine someone invents some new plumbing material that never dissolves harmful pollutants, is cheaper to manufacture, easy and cheap to work with, does not burst when freezing and lasts forever.

And the production facilities and distribution are essentially free, because this magical material is produced by self-replicating spherical unicorns that feed on greenhouse gasses.

But even if all these properties were somehow true, it still makes zero economical sense to preemptively rip out and replace the existing plumbing in literally billions of structures.

RIIR

Posted Feb 16, 2025 19:27 UTC (Sun) by marcH (subscriber, #57642) [Link] (6 responses)

> it still makes zero economical sense to preemptively rip out and replace the existing plumbing in literally billions of structures.

This is where software analogies always breaks down: the Economy.

- With software, replacing the existing plumbing in ONE house is barely cheaper than replacing it in ALL houses.
- The duplication cost is so negligible that a single guy doing some DIY work in his garage for fun - NOT for economical reasons - can (and here: does!) end up replacing the plumbing in all houses across the world.

The previous point was not about the economy at all. It was about "innovation". While it's not visible to users at all, there is "behind the scenes" innovation in uutils because it's a production-grade, potentially mass-deployed Rust project. It's admittedly less rocket science and more engineering but the latter matters too.

RIIR

Posted Feb 16, 2025 22:41 UTC (Sun) by pizza (subscriber, #46) [Link] (5 responses)

> - With software, replacing the existing plumbing in ONE house is barely cheaper than replacing it in ALL houses.

That would be true if everyone, and every device, ran identical software in lock-step. Heck, even when the software components _are_ identical, they're often (usually!) integrated in different (if not completely bespoke) manners.

In the longer run, integration (and subsequent maintenance) is usually more work than writing the original software. (not unlike plumbing!)

> The previous point was not about the economy at all. It was about "innovation". While it's not visible to users at all, there is "behind the scenes" innovation in uutils because it's a production-grade, potentially mass-deployed Rust project.

"innovation" implies something new, not a 100% re-implementation (and necessarily quirk-for-quirk compatible) of existing (and well-maintained!) code.. Would you still be calling uutils "innovative" if it was written in oh, Ada, Java, Python, or PHP?

RIIR

Posted Feb 16, 2025 23:23 UTC (Sun) by marcH (subscriber, #57642) [Link] (4 responses)

> > - With software, replacing the existing plumbing in ONE house is barely cheaper than replacing it in ALL houses.

> That would be true if everyone, and every device, ran identical software in lock-step.

This is more and more irrelevant... I just gave a reminder that software marginal costs are negligible, which limits software analogies. How many devices are actually running what software does not matter: marginal software costs are still negligible and the economic aspect of software analogies still fails.

> "innovation" implies something new,...

Obviously.

> ... not a 100% re-implementation (and necessarily quirk-for-quirk compatible) of existing (and well-maintained!) code

That's part of your definition. My definition is "not in mass production yet" and it does not require cars to fly.

> Would you still be calling uutils "innovative" if it was written in oh, Ada, Java, Python, or PHP?

No because:
1) There would most likely be significant performance regressions (which obviously matter in "core" utils) instead of the progressions observed with Rust.
2) None of these languages are recent in 2025, they have all been in production for decades none of them needs any sort uutils-like project to help them get battle-hardened and "upgraded" to mass production.

So I suspect uutils in Java would be non-event.

Here's another analogy: designing and selling an electric car with a brand new, solid state battery design today would be "innovative" because it would help the solid state battery company scale up. That would be innovative even if the car company did nothing innovative besides trusting, partnering with and ultimately helping the innovative battery company doing all the innovative research and even when customers see "only" more range and less fire risk (= the battery equivalent of memory corruption?)

Once a few other Rust rewrites like uutils will be in "mass production" stage (e.g.: enabled by default in some distributions), then I agree the next ones won't be as "innovative" - and will get less press coverage. But there are several domains: first graphical application, first kernel driver, etc. Each domain has different engineering challenges requiring different "innovative" solutions.

RIIR

Posted Feb 17, 2025 14:32 UTC (Mon) by pizza (subscriber, #46) [Link] (3 responses)

> This is more and more irrelevant... I just gave a reminder that software marginal costs are negligible, which limits software analogies. How many devices are actually running what software does not matter: marginal software costs are still negligible and the economic aspect of software analogies still fails.

Just because *you* dismiss it as irrelevant doesn't make it so. The world is far larger than you.

"Marginal software costs" refer to the production of additional identical _copies_. If said copies are not identical, that margin rapidly grows. The computing world is anything but homogeneous.

Or are you seriously arguing that a binary driver written for a specific Windows version will enable that hardware to work on MacOS, all currently-supprorted Linux LTS distributions, and one of countless RTOSes running on the dozen-ish major microcontroller architectures? Heck, even if that driver was provided in source form, it's going to be a ton of work to enable the others -- and it's most likely to be simpler, faster, and cheaper to start over from scratch for each one.

> 2) None of these languages are recent in 2025, they have all been in production for decades none of them needs any sort uutils-like project to help them get battle-hardened and "upgraded" to mass production.

So... the "innovation" here is "Figuring out how to make Rust good enough to compete with stuff that's existed for decades?"

> Here's another analogy: designing and selling an electric car with a brand new, solid state battery design today would be "innovative" because it would help the solid state battery company scale up.

"helping the battery company scale up" is an example of longer-term planning and/or investment, not innovation.

(The battery technology itself, including specific manufacturing processes, may be innovative)

RIIR

Posted Feb 17, 2025 17:49 UTC (Mon) by marcH (subscriber, #57642) [Link] (2 responses)

> Just because *you* dismiss it as irrelevant doesn't make it so.

It's irrelevant to "Is uutils innovation?" which is the very specific disagreement in this subthread. Feel free to discuss software marginal costs or any other very interesting but unrelated question anywhere else.

> The world is far larger than you.

Thanks, that's very useful to know.

> So... the "innovation" here is "Figuring out how to make Rust good enough to compete with stuff that's existed for decades?"

Yes. It's a very important innovation milestone towards getting rid of memory corruption and 70% of vulnerabilities in operating systems, browsers (the other operating system) and more. You're entitled to your definition of "innovation" but mine does not stop outside the lab.

It's just a word definition; let's agree to disagree.

> "helping the battery company scale up" is an example of longer-term planning and/or investment, not innovation.

Not for the very first customer, no. It's most often a partnership and a huge bet.

RIIR

Posted Feb 17, 2025 17:57 UTC (Mon) by marcH (subscriber, #57642) [Link]

> ... your definition of "innovation" but mine does not stop outside the lab.

Simply because many "innovations" die once they leave the lab.

This is especially true with software which gets created and dies orders of magnitude more than in other fields (darn, and now I'm close to making your economic digression relevant...)

RIIR

Posted Feb 19, 2025 7:00 UTC (Wed) by marcH (subscriber, #57642) [Link]

> > So... the "innovation" here is "Figuring out how to make Rust good enough to compete with stuff that's existed for decades?"

> Yes. It's a very important innovation milestone towards getting rid of memory corruption and 70% of vulnerabilities in ...

From https://lwn.net/Articles/1008721/

> Ultimately, Poettering said that he is "happy to play ball" but does not want systemd to be the ones to solve the problems that need to be solved in order to use Rust.

That's the boring, last part of innovation: making it "mainstream". Rust is getting very close but not quite there yet. If ever?

RIIR

Posted Feb 20, 2025 7:07 UTC (Thu) by ssmith32 (subscriber, #72404) [Link]

Compared to the cost of the production and distribution of pipes (particularly large pipes and fittings for multi-unit buildings)?

Yeah, production/packaging and distribution is essentially free for the software in question ("small" command line tools).

You never have the "lone maintainer" problem for the production and distribution of large pipes and fittings, at least produced at the scales we're discussing, because you simply _can't_.

It may not feel that way for the lone maintainer, for sure, but an objective comparison of the costs demonstrates otherwise.