RIIR [LWN.net]

RIIR

Posted Feb 12, 2025 21:04 UTC (Wed) by kleptog (subscriber, #1183)
In reply to: RIIR by da4089
Parent article: Rewriting essential Linux packages in Rust

> But instead of putting effort into discovering new better approaches to tools (and operating systems) we instead spend the innovative potential of a generation to get back to where we started, even losing some features in the process.

They're replicating the results of those 55 years of development in few years, so what "innovative potential" is being lost here? Now you have a sound base to continue building, except every year of effort put in the Rust version is equivalent to multiple years of effort on the C version.

I don't understand this idea that these tools should have minimal dependencies. Why should "sort" implement its own sorting algorithm instead of using the battle-tested UltraSpeedParallelSuperSorting crate that is used by everyone else because it sorts blazingly fast?

We can build great programs because we are standing on the shoulder of powerful libraries. Re-use is almost always better than writing it yourself.

Let's face it: C programmers aim for minimal dependencies because dependencies in C programs are really difficult. In more modern language dependencies are one line in your package configuration and you're done.

to post comments

RIIR

Posted Feb 12, 2025 23:56 UTC (Wed) by dvdeug (guest, #10998) [Link] (3 responses)

> In more modern language dependencies are one line in your package configuration and you're done.

But they're not, for many reasons.

Many programmers have used a cool new library and discovered that it was no longer supported after a few years and wouldn't build with newer versions of other dependencies, and had to replace it. One can recall the left-pad catastrophe, where one author withdrew a 17-line package from npm and thousands of programs stopped building. One can also remember the endless chase of libgtk versions.

You lose control over supported platforms. E.g. the Python-Cryptography package started using Rust, and now no longer works on many of the platforms it used to. Coreutils compiles on HPUX 11 on hppa and ia64. Even if Rust supported those, you're still depending on every dependencies you use to support those. Good luck with your x86 box when your dependency pivots to using Oxide, the programming language that's going to replace Rust.

The bug envelope changes. If sort implements its own sorting algorithm, then if it breaks, you look at the sort code, if necessary checking that it works on the same libc and gcc version. If it uses the UltraSpeedParallelSuperSorting crate, then you have to look at that crate, and any crates it uses. Coreutils supports a variety of C libraries and C compilers (even with the restriction to C99); it's easy to prove that any new bugs are due to your changes. You can say that this crate is used by "everyone else", but you'll end up using a crate that does what you need but not everyone else needs, or using a crate in a way that other people aren't, triggering bugs that nobody else is meeting.

That was benign bugs, too. The Jia Tan attack was made possible by the fact that libxz was linked into ssh. To make it worse, it wasn't even code that ssh used; it was unused code from using another dependency to support systemd. Every dependency adds security risk, and when they're built in this fashion themselves, they bring in more security risks with their dependencies.

If you're writing a KDE program or GNOME program, both come with an array of libraries you can solidly depend on, no problem. A major frontend program can pull in a key dependencies without issue; most people get Gimp or Darktable through their distro or Flatpak anyway. (But note that this idea that dependencies have no cost amplifies the problem. Installing a C library that depends on nothing or a few standard dependencies is low cost. Installing a dependency that depends on a host of other dependencies amplifies most of the problems above.)

But coreutils? Programs that are frequently run as root, are called from shell scripts all over the system, and thus are security critical? Programs that are critical to running on every system? Be it Rust or C, they should carefully consider every dependency.

RIIR

Posted Feb 13, 2025 9:41 UTC (Thu) by taladar (subscriber, #68407) [Link] (1 responses)

What I don't like about this argument is that it treats writing your own version of the dependency inside your own code base as if it was free and had no associated risks when that is far from true.

Your own internal version is very likely to be orders of magnitude less tested, optimized, maintained,... than even a moderately used third party dependency. It is also less likely to see the same scrutiny from security researchers or the same amount of tracking from security tracking systems like CVE or GHSA.

The number of programmers who know how it works will be lower. When it isn't touched for a while nobody (including the original author) will know how it works or how to change it. You will have more trouble to find people to add cross-platform support because they will need to use exactly your program, not just any one of the programs using it.

Also, what you say about the bug envelope doesn't work at all. If there is a bug in the sorting algorithm you wrote, you have to find it and invest time that will only benefit that one project. If the bug is in a dependency you have a good chance someone else already found it and even if not, your effort to find and fix it benefits everyone using that library.

You look at it from the perspective of comparing a Rust dependency that is maybe 5 years old with some internal version in some 50 year old C tool and there the cross platform support will likely come out on top because cross platform used to be so much more important in the bad old days when nothing was standardized and the hardware and OS landscapes were more fractured in terms of what baseline features you could expect from your hardware or OS but honestly, reusable and shared code is the better choice in the long term.

RIIR

Posted Feb 21, 2025 2:02 UTC (Fri) by dvdeug (guest, #10998) [Link]

> Your own internal version is very likely to be orders of magnitude less tested, optimized, maintained,...

If we're talking about coreutils, no, it's not. That's the context of this.

> When it isn't touched for a while nobody (including the original author) will know how it works or how to change it.

Good code is understandable. Not to mention, code that does what you need is sometimes easier to understand than interfacing with a powerful library with many features and options.

> Also, what you say about the bug envelope doesn't work at all. If there is a bug in the sorting algorithm you wrote, you have to find it and invest time that will only benefit that one project. If the bug is in a dependency you have a good chance someone else already found it and even if not, your effort to find and fix it benefits everyone using that library.

If there's anyone else using that library. If everyone else hasn't already moved on to the next big thing. If you didn't pick the library that was better, but never hit critical mass and the maintainer dropped it. If you aren't calling with a set of options that nobody else is using.

There are libraries like libz and libpng where there's little reason not to include. But so many libraries, crates, packages, etc. are created, uploaded, maybe even get a few releases, then get dropped. If you're writing coreutils in Rust and hope to replace the existing coreutils, you need to plan for a 50 year existence, most of which is going to spent doing minor changes on programs that work.

Cross platform is a choice. But Debian won't switch until it supports all release platforms, of which there are nine, and failing to support GNU/Hurd or several other semiactive ports is going to lose you some support. Sell to your audience.

> reusable and shared code is the better choice in the long term.

Making a tight program that runs where you need it to and runs even when other stuff is broken is incredibly useful. We have busybox with 0 dependencies, because coreutils today has too many dependencies in some cases. Reusable code is good, but again Jai Tan got in because ssh was being linked against a library which linked against a library that ssh didn't need. You've never had to deal with an upgrade mess when dealing with an incompatible ABI break at the base of a tree of dependencies. Or having to change a program to work with the new version of the library, because the old version of the library has an entirely different API but isn't getting bug fixes any more.

Again, you're writing a GNOME program, go wild with GNOME libraries. You're writing a program that needs a solid solver for 3-SAT, link in z3. You get a choice between writing 5 lines of code using a hash-map that's fast enough, or linking in a custom data structure that is optimized... if it's code for mass distribution and hopefully a long life, save yourself some headache down the road and write the 5 lines of code.

RIIR

Posted Feb 15, 2025 5:53 UTC (Sat) by marcH (subscriber, #57642) [Link]

> > In more modern language dependencies are one line in your package configuration and you're done.

> But they're not, for many reasons.

Indeed not.

https://cacm.acm.org/practice/surviving-software-dependen...
https://medium.com/@sdboyer/so-you-want-to-write-a-packag...
etc.

To be fair, it is much worse with C/C++ because then you have both the technical _and_ the less technical issues.