LWN: Comments on "Filesystem-oriented flags: sad, messy and not going away"

Filesystem-oriented flags: sad, messy and not going away

gray_-_wolf — Mon, 23 Mar 2020 15:27:43 +0000

> least one language run-time (Go) that doesn't depend on a C library.

sometimes... would be nice if it never did but that is sadly not the case :/

Filesystem-oriented flags: sad, messy and not going away

draco — Fri, 20 Mar 2020 12:43:55 +0000

If they weren't done that way from the start, then that's an ABI change, which isn't allowed.

Filesystem-oriented flags: sad, messy and not going away

jezuch — Fri, 20 Mar 2020 10:15:02 +0000

The rich type system also means that there are a lot of generic functions taking traits as inputs, which are not "real" types. So either you have a lot of monomorphization ("instantiation" in terms of C++ templates) where you substitute real types for the parameters, or you have dynamic dispatch, which is, well, akin to uncoditional surrender. Forcing dynamic linking also forces the latter, and this has huge impact on performance, maybe as much as impeded inlining has.

(Disclaimer: I speak from theory, not practice, so I may be more than a little wrong :) )

Filesystem-oriented flags: sad, messy and not going away

BenHutchings — Thu, 19 Mar 2020 20:55:16 +0000

Indeed, there is no such thing as "Linux's libc". There's glibc, bionic, uclibc, musl, klibc, and at least one language run-time (Go) that doesn't depend on a C library.

Filesystem-oriented flags: sad, messy and not going away

justincormack — Wed, 18 Mar 2020 21:37:57 +0000

Yes it was just a normal library - you could make syscalls elsewhere but they were neither documented or stable.

OpenBSD has been taking this model to a more modern design, where libc is blessed, and only it can make syscalls, by having a special attribute set. This is designed as a security measure, to stop arbitrary code using syscalls.

Filesystem-oriented flags: sad, messy and not going away

k3ninho — Wed, 18 Mar 2020 11:07:30 +0000

We could manage versioning with a request broker*.
"Do you speak the ABI of versions in this range?"
"Not all of them, I can fall back to v.A.B.C as most recent. Is that OK?"
"Confirmed OK."

*: common object request broker isn't a model, it's an architecture ;-)

K3n.

Filesystem-oriented flags: sad, messy and not going away

roc — Wed, 18 Mar 2020 08:54:42 +0000

rr patches the VDSO to just do direct syscalls. It's as if the VDSO wasn't there.

Filesystem-oriented flags: sad, messy and not going away

roc — Wed, 18 Mar 2020 08:53:43 +0000

Lowest-common-denominator ABIs like COM are awful to work with.

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 21:47:05 +0000

Agree completely. You just beat me to it!

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 21:45:53 +0000

In these days of flatpack and image-based application distribution, where applications ship with private versions of all of the shared libraries that they use, it's easy to argue that the days of shared libraries being particularly useful, at least for supposed benefits of disk space or memory space savings, are long gone. From a language point of view, the model of separately compiled object files is too restrictive, and too much of a barrier to efficient abstraction. Most modern languages have a whole-program compilation model, and that includes C++, except for the cases where modules effectively isolate themselves behind a C API. (Go, rust, julia, haskell, all of the lisps...)
I view this trend as a good thing, btw. The modern languages have a lot going for them, and shared libraries really don't.

Filesystem-oriented flags: sad, messy and not going away

rvolgers — Tue, 17 Mar 2020 21:44:30 +0000

To clarify my point a bit more: the way native Rust APIs are written is just very different from how C APIs are written. Due to the rich type system you can have a lot more back-and-forth between a library and its consumers than you would have in a C API.

Consider for example the Iterator trait in Rust. People expect code written using iterators to compile down to something that you would find hard to distinguish from a C for loop in disassembly, which requires the compiler to inline a whole bunch of calls to tiny functions and remove some intermediate values. And not all those tiny functions have to come from the same library, they can come from many different ones, and many will have generic arguments or callbacks with generic arguments from still other libraries.

And it's not just Iterator, the same goes for asynchronous I/O using Futures, and probably more absolutely core functionality that I'm forgetting about right now. As soon as parts of that become dynamically linked, you start having to make some really tough calls about what the compiler can statically assume and optimize out.

Filesystem-oriented flags: sad, messy and not going away

rvolgers — Tue, 17 Mar 2020 21:33:55 +0000

That "huge downsides" article is not entirely fair perhaps.

Rust does link dynamically to libc, and many Rust programs link to e.g. OpenSSL because Rust has good support for using dynamically linked C libraries. In fact, there are dynamic libraries with a C ABI that are implemented in Rust (librsvg comes to mind).

Rust has really good support for dynamic linking! It just doesn't have good support for dynamic linking using its *native ABI*. You could look at this as discouraging dynamic linking, but you can also look at it as encouraging dynamic linking that integrates well with the rest of the open source ecosystem by using the C ABI as a universal interface.

Also, a ton of Rust code is just not desirable to dynamically link, ever. We could do a cute experiment and compile some popular Rust programs while absolutely forbidding the compiler to inline functions between different crates (i.e. "libraries"). Pretty sure that will cause a code size explosion and speed reduction that will make people scream a lot louder than using a couple more kb of disk space.

Filesystem-oriented flags: sad, messy and not going away

quotemstr — Tue, 17 Mar 2020 21:21:40 +0000

> In theory we could escape the dilemma by creating a stable ABI for shared libraries

COM solved that problem decades ago. We should seriously consider adopting something a lot like it. A stable object ABI that allows for both efficient intraprocess calling and extensible interprocess remoting is extremely powerful.

Filesystem-oriented flags: sad, messy and not going away

sorokin — Tue, 17 Mar 2020 21:21:35 +0000

I heard kernel can inject some code into userspace. I don't know the details, but the keyword is vdso -- I think kernel hackers here know much more about it than I do.

Wouldn't presence of vdso pose the same problem as the stability guarantee applied at shared library boundary?

Filesystem-oriented flags: sad, messy and not going away

ale2018 — Tue, 17 Mar 2020 18:42:55 +0000

I would have said that the situation would look less messy if similar flags were set at the same bit. Having 64-bit flags seems to allow it. Hmm...

/usr/include/linux/fcntl.h:#define AT_SYMLINK_NOFOLLOW	0x100   /* Do not follow symbolic links.  */
/usr/include/asm-generic/fcntl.h:#define O_NOFOLLOW	00400000	/* don't follow links */
/usr/include/x86_64-linux-gnu/bits/fcntl-linux.h:# define AT_SYMLINK_NOFOLLOW	0x100	/* Do not follow symbolic links.  */
/usr/include/x86_64-linux-gnu/sys/mount.h:  UMOUNT_NOFOLLOW = 8		/* Don't follow symlink on umount.  */

Filesystem-oriented flags: sad, messy and not going away

farnz — Tue, 17 Mar 2020 15:15:40 +0000

The equivalent in Rust is Editions; you can freely link code between different editions (currently only 2015 and 2018), but the compiler will translate each translation unit (crate in Rust) according to the edition you have specified for that crate.

Nothing, however, stops a Rust 2015 crate using a Rust 2018 crate as a dependency, or vice-versa, and you can freely share data types between the two editions. The only problem is that you might have to use r#identifier syntax if one crate uses a reserved word as an identifier.

Filesystem-oriented flags: sad, messy and not going away

Wol — Tue, 17 Mar 2020 11:53:23 +0000

The problem is too many developers only develop to solve the immediate problem. Spend a bit of time to define the *general* problem, define a state table and design the API to solve said state table, and then by all means just solve your bit of it.

That way, you can extend the function to fill the state table as and when, without having to redesign the interface.

Cheers,
Wol

Filesystem-oriented flags: sad, messy and not going away

roc — Tue, 17 Mar 2020 10:42:19 +0000

FWIW rr benefits enormously from the "stable kernel ABI" being the syscall interface. The same is true for strace and other tools that monitor and manipulate the user/kernel interface.

If stability guarantees applied at a shared library boundary like Solaris and Windows, then rr would have to choose between manipulating the unstable syscall interface or manipulating the shared library interface. The former would increase the rr maintenance burden considerably since we'd have to support every version of the syscall interface. The latter is difficult to do in a watertight way. Similar considerations would apply to strace etc.

I think it's also a great feature to have your stable ABI boundary enforced by hardware. On Windows people reverse engineer the syscalls and sometimes call them directly, bypassing the "stable ABI"; it's great that on Linux you simply *can't* bypass it.

Hardware being aware of the ABI boundary has other more esoteric benefits. For example rr needs to count performance events happening outside "the kernel"; on Linux the CPU supports us doing that, but on Windows/Solaris it doesn't if you consider that shared library to be "the kernel".

Filesystem-oriented flags: sad, messy and not going away

josh — Tue, 17 Mar 2020 10:25:52 +0000

I agree, and I prefer the Linux approach.

That said, it'd be interesting if we had a slightly more extensible syscall layer that could tell when an argument was passed or not passed, which would allow existing existing syscalls without having to create new ones.

It's looking increasingly like io_uring might be that extensible syscall layer.

Filesystem-oriented flags: sad, messy and not going away

roc — Tue, 17 Mar 2020 10:25:14 +0000

The stability of the C++ ABI also has massive downsides: https://cor3ntin.github.io/posts/abi/

In theory we could escape the dilemma by creating a stable ABI for shared libraries that you opt into at build time that would mostly be only for Linux distros. But even that would constrain language and library evolution as well as being a ton of work that no-one is really motivated to do.

Filesystem-oriented flags: sad, messy and not going away

smurf — Tue, 17 Mar 2020 09:51:25 +0000

This means that BSD's libc is tied to the kernel. Linux' libc is not.

A corollary is that statically-linked programs may or may not continue to work when you update your kernel, a notion which Linus emphatically rejects.

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 07:41:24 +0000

I'm afraid that I don't know. I had always assumed that it was a specially-blessed user-space library provided by the filesystem. I'm sure there are readers who know more about Solaris than I do (it wouldn't be hard).

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 07:02:46 +0000

I was wondering why commenters were talking about rust but then I noticed that I mis-typed. Rust isn't a (scheme)! I meant racket of course. Sorry for the confusion!

I know that rust is versioning its releases too, but I don't know whether that actually allows for the linking and use of code written against different language versions, the way racket does. Racket code can import and use r5rs or r6rs or experimental-dialect code, which is cool. I suppose that C++ can do similarly for separately compiled object files, but it can't include old headers into new code, and python3 can't import python2 modules, which IMO is a terrible shame.

Filesystem-oriented flags: sad, messy and not going away

josh — Tue, 17 Mar 2020 06:53:38 +0000

> The solaris model is that the syscall API is a shared object

How does Solaris provide that to userspace? Similar to the VDSO, or via a library provided on the filesystem that calls an unstable kernel interface?

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 06:47:57 +0000

The solaris model is that the syscall API is a shared object (along with the rest of libc). User-space code doesn't get to make syscalls at all. The libc shared library can be versioned and indeed multiple, so you can (theoretically) keep older cruftier ones only as long as you have any executables that need them, on an install-by-install basis. I believe that OpenBSD is considering a similar scheme in order to have some sort of protection about where syscalls can come from, to prevent trampoline and gadget-style malware, perhaps.

The BSD versioned syscalls are in the kernel (so you can still have static executables), but they can be supplied by loadable kernel modules (as the linux and SCO syscalls are/were), which can eventually be deprecated or not loaded as suits the use-case, without getting (too much) in the way of the "fresh" syscall API.

Filesystem-oriented flags: sad, messy and not going away

josh — Tue, 17 Mar 2020 06:37:01 +0000

On BSD, the syscall interface is subject to change, while the libc interface is stable. So it'd be libc's job to implement the old interface in terms of the new one.

Filesystem-oriented flags: sad, messy and not going away

Cyberax — Tue, 17 Mar 2020 06:33:11 +0000

> Yes, but the Solaris and BSD versioned syscall approach allows you to remove the old syscalls and their messy flags immediately
How? There will still be software that uses old flags, for the foreseeable future. You'll have to provide their emulation _somewhere_.

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 06:29:55 +0000

Yes, but the Solaris and BSD versioned syscall approach allows you to remove the old syscalls and their messy flags immediately, reducing complexity, size and technical debt in the kernel. For the extra win, the piece of libc code that has to provide the old API in terms of the new becomes a nice modular piece of API history that documents the change.

Filesystem-oriented flags: sad, messy and not going away

Cyberax — Tue, 17 Mar 2020 06:18:45 +0000

This is doable in Linux.

You can create a set of new syscalls with sane flags: openat_rational, link_rational, open_I_really_mean_it's_not_broken_this_time and so on. You then expose these new syscalls with their wonderful flags through libc, libc can also provide their emulation for the older kernels that lack the new syscalls.

Then after 20 years or so you can remove the old flags from libc, so that new code will be able to use the new flags. Then after another 10 years or so, the old syscalls can be removed from the kernel.

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 06:00:33 +0000

Yes, at the moment, but API versioning is the tool that you need in order to rationalize and tidy-up down the track, without breaking the code that will inevitably have been built against the current, messy set.

Filesystem-oriented flags: sad, messy and not going away

Cyberax — Tue, 17 Mar 2020 05:53:46 +0000

The problem is that these particular flags mirror the userspace. You will inevitably have the same mess _somewhere_, be it libc or the kernel.

Filesystem-oriented flags: sad, messy and not going away

Conan_Kudo — Tue, 17 Mar 2020 02:34:52 +0000

Afaik, Debian ships rust programs, and if I know anything about Debian packaging, static linking is not even close to being an option =]

Unfortunately, you'd be wrong. Debian ships packages with piles of source code, just as Fedora does. Applications statically link everything, because otherwise every rebuild of the compiler would necessitate rebuilding everything. It's just not practical. Maybe one day, the Rust community will care about us and work toward defining a native stable ABI. But I won't hold my breath. The Rust community thinks it's okay to have to constantly build everything for every change, despite the huge downsides.

Filesystem-oriented flags: sad, messy and not going away

tux3 — Tue, 17 Mar 2020 02:19:07 +0000

Rust does dynamic linking (and not just C), the caveat I think is that the ABI is not stable between compiler releases.
This is notably more unstable than the C++ ABI, which (visual studio excepted) only breaks for major events like the C++11 release.

Afaik, Debian ships rust programs, and if I know anything about Debian packaging, static linking is not even close to being an option =]

Filesystem-oriented flags: sad, messy and not going away

wahern — Tue, 17 Mar 2020 01:56:02 +0000

Rust also only supports static linking (simple C ABI interfaces notwithstanding), so as compared to C++ it's an easier problem space.

Filesystem-oriented flags: sad, messy and not going away

areilly — Tue, 17 Mar 2020 01:15:07 +0000

No doubt this sort of conflicting accretion of symbols and functionality is why Solaris eventually isolated its syscall interface with a shared library: possibility exists to support old code with compatibility versions while keeping kernel APIs flexible and cruft free. Not that I've ever looked at the Solaris kernel APIs to see if cruft was actually reduced.

Notionally also why the BSD executable versioning mechanism was introduced, although I don't know that that has ever been used as a mechanism to "tidy up" or unify older syscalls, rather than just providing OS emulation capabilities. The possibility exists though.

I like the way that the Rust (scheme) system handles language features and backwards compatibility: language version is an explicit part of the code preamble, which allows new code to link against old libraries without requiring the old libraries to be modified to match the current-version syntax or semantics. Something that neither C++ or python have managed during their history. All such a system requires is that whatever the new interfaces are, they must be capable of providing the original semantics somehow, so that the interface shim layer can re-implement the old API in terms of the new. Clearly you want a modicum of stability too, to minimize the number of old shims that you have to maintain.