LWN: Comments on "Identifying dependencies used via dlopen()"

Identifying dependencies used via dlopen()

bluca — Sat, 18 May 2024 10:24:46 +0000

Follow-up: I have taken over that PR, refined it a bit to do some C preprocessor crimes to have a JSON payload in the ELF note, written a spec ( https://github.com/systemd/systemd/blob/main/docs/ELF_DLO... will be published on systemd.io once we do the final release), written a debhelper plugin for Debian package builds (https://tracker.debian.org/pkg/package-notes available in unstable and soon testing) and used it for the latest systemd upload, which is now getting auto generated recommends/suggests for dlopens: https://packages.debian.org/unstable/libsystemd0

Also sent patches for readelf (both implementations) to be able to pretty-print the new note with --notes. Next would be enhancing glibc's ldd to parse this too (a bit further down the todo list), and RPM macros (that one's for the Fedora folks).

All in all, quite happy with how this has turned out in the end!

Identifying dependencies used via dlopen()

mathstuf — Thu, 25 Apr 2024 14:15:26 +0000

> Presumably, if the right function couldn't be found with dlsym(), then the library could be safely unloaded and ignored.

Once the library is loaded, you've already lost if it does anything that doesn't allow unloading. Touching thread-local storage on macOS will do this. As will any static initializers that *do* exist. One nice thing on Windows is that you can interrogate a library without loading it. One can use `libelf` for that with ELF platforms, but that is Another Dependency and you're left figuring out what the runtime loader will do with it on your own.

Identifying dependencies used via dlopen()

Cyberax — Wed, 24 Apr 2024 18:00:19 +0000

JFYI, Apple removed that anti-feature: https://developer.apple.com/forums/thread/131252

100% support that. Lazy loading of libraries is actively bad and leads to crazy designs.

Identifying dependencies used via dlopen()

Cyberax — Wed, 24 Apr 2024 11:00:45 +0000

I tried it for a while and couldn't get it to work. I went down the rabbit hole of reading the source code of reading the LDD code before finding this:

https://developer.apple.com/forums/thread/131252

> That feature has been removed from the linker. The only work around is to not link with library and instead dlopen() it at runtime and then use dlsym() to find the functions you want from it.

So yes, it's was a horrible feature that was barely used. So Apple did the right thing and removed it.

Identifying dependencies used via dlopen()

farnz — Wed, 24 Apr 2024 09:29:08 +0000

The code doing the loading can call a void * plugin_init(void) function, which does whatever your deferred dynamic initializers do today; call back to the opener to register a file format reader, a new renderer, a writer, whatever else they currently do at the moment they're loaded. At exit, it can call a void plugin_destroy(void * init_return) function, which does the destruction before the plugin is unloaded.

There's nothing that can be done via code running at load time that can't also be done via calling a function inside the library; the reason most psABIs run code at load time, rather than simply expecting that the start point runs constructors etc, is to make it easy to implement C++ deferred dynamic initialization.

If I were interested in playing with psABIs, with a view to producing an alternative to current ELF mechanisms, I'd be tempted to experiment with indirect symbol resolution as an alternative to running code on load; the idea is that when a symbol is resolved, instead of the loader looking up the symbol value directly (as it does normally), it would run code that provides the symbol value or a reference to another symbol to look up. You can then implement deferred dynamic initialization by having indirect lookup functions that check a flag, run static initializers if needed, and return the "real" symbol to the loader for it to look up - slightly slowing down symbol lookup if static initializers exist, but deferring it to the last moment that C++ permits.

Identifying dependencies used via dlopen()

farnz — Wed, 24 Apr 2024 09:11:54 +0000

I was referring not to dlopen, but to libraries that are lazily linked in to the binary via Mach-O dynamic linking; at the first use of a symbol from the library, the library's deferred dynamic initialization code is run.

Identifying dependencies used via dlopen()

ScottMinster — Tue, 23 Apr 2024 21:10:24 +0000

The code in question is fairly generic and just loads any .so "plugin" files found in the write location. It doesn't know what functionality any of them have, just that it has been configured to load them. In addition, there are a lot of different ways the plugin libraries themselves can extend functionality. Maybe they add support for reading a new file format. Or writing a file format. Or doing both. Or reading and writing several similar formats. The point is, it would be hard for the code doing the loading to know what registrations to call on behalf of the library. It is easier for the library to make those calls itself based on what it knows it can do.

As a further detail, the code is C++, so the initialization is actually being done through static initializers in the library (constructors on static objects). The registration method is a constructor on a base class that records the pointer to itself (this) in a list of pointers. Later, when asked to exploit the loaded functionality, it can call a pure virtual method (implemented in the derived class in the plugin library) to do the work.

If delaying static initializers until something in the loaded library was explicitly called, I would probably have to change the logic to make some sort of dummy call into each plugin library loaded. That is a bit annoying, though on reflection, if I set it up right, it would provide some kind of check that the library that was loaded really was meant to be loaded. That it really was a plug in to this software and not some mistake. Presumably, if the right function couldn't be found with dlsym(), then the library could be safely unloaded and ignored.

So delaying is an interesting idea, but would probably cause too many problems before software was fixed to expect it. But maybe it could be an opt-in flag to dlpen()?

Identifying dependencies used via dlopen()

Cyberax — Tue, 23 Apr 2024 18:04:30 +0000

This is basically replacing automatic initialization with an explicit init() function. Which is a great idea in general, because it makes initialization predictable.

Identifying dependencies used via dlopen()

Cyberax — Tue, 23 Apr 2024 17:56:50 +0000

I tried: https://gist.github.com/Cyberax/a60dce723d4e80a5d182d6067...

It's definitely not _that_ lazy by default. The initializers run during the dlopen() time.

Identifying dependencies used via dlopen()

ibukanov — Tue, 23 Apr 2024 16:36:41 +0000

Well, then give an option not to run the init code from .so until the app explicitly asks for it. This will be sort-of dlopen for linked libraries.

Identifying dependencies used via dlopen()

farnz — Tue, 23 Apr 2024 09:29:25 +0000

This is how iOS and macOS work; the initializer is run on library load, and library load is delayed until the first symbol from that library is referenced. It's also explicitly permitted in the C++ standard - deferred dynamic initialization takes place no later than just before the first use of a symbol from the translation unit that contains the static initializer.

Now, I'd agree with the notion that deferred dynamic initialization (runtime constructors for static objects) is a mistake. But it's a mistake baked into C++, and from there into psABIs.

Identifying dependencies used via dlopen()

Cyberax — Mon, 22 Apr 2024 19:50:00 +0000

This is a bad idea. It can inject unexpected failures into functions "can not" fail, and which the application doesn't expect to fail. This can even be caused by things like sandboxing, or unexpected environment conditions like file descriptor exhaustion.

It's really much better to just do initialization explicitly by calling something like "init()" after dlopen(), or just link libraries eagerly and incorporate initialization into the overall application initialization.

Identifying dependencies used via dlopen()

ibukanov — Mon, 22 Apr 2024 18:44:11 +0000

What I suggest is to have an option for a linked library to delay the initialization until the first call to the library function.

With such option all library functions point to a code that first ensures initialization of the library and then calls the function itself. From the library point of view this looks as if the library was loaded with dlopen followed by a call to the function.

Identifying dependencies used via dlopen()

rschroev — Mon, 22 Apr 2024 16:53:17 +0000

Can't the code that calls dlopen() to load the library also call a function like init() or register() or something like that in the library?

Identifying dependencies used via dlopen()

ScottMinster — Mon, 22 Apr 2024 15:57:28 +0000

Some of the software I work on makes use of that initialization phase of the library to have the library tell other parts of the software what it can do. The very act of calling dlopen() on these plugin libraries lets them register themselves. If the library initialization code didn't run until later, the library would end up never being called at all, even if it was needed.

Identifying dependencies used via dlopen()

Cyberax — Fri, 19 Apr 2024 13:02:26 +0000

That's fine. dlopen() to work around legal or legacy requirements is not a big deal. It's like using LD_PRELOAD to work around a bug in one of the libraries for an old proprietary app. Using LD_PRELOAD for all systems would not be anything close to being acceptable.

Identifying dependencies used via dlopen()

pizza — Fri, 19 Apr 2024 12:40:51 +0000

> First, "optional dependency" is kind of an oxymoron. It's either a dependency or not.

One of my printer drivers has an _optional_ dependency on an image processing library.

It is optional unless you have one of two specific models, then that library becomes mandatory if you want the printer to produce anything considered passable.

That library started out as proprietary, non-redistributable, and x86-only. So I reverse-engineered it and wrote an F/OSS replacement. Problem is that due to patent concerns, the F/OSS library isn't distributed in binary form -- or even as source in the upstream project.

Doing things this way lets distributions handle most of the support burden -- ie a single binary covering everyone, and that works for everyone without those two specific models. Folks with those models who get a license to use the proprietary library (that's guaranteed to work) can plop it in /usr/local/lib; or they can do a 'git pull && make && sudo make install` and get (so far) bit-for-bit identical results.

A dlopen() based approach is by far the simplest [1] way to accomplish the goals in a portable [2] manner.

[1] ie least burdensome to *me*
[2] Linux, MacOS, and even Windows. (ie probably every platform with libusb support)

Identifying dependencies used via dlopen()

Cyberax — Fri, 19 Apr 2024 11:25:49 +0000

> Optional features are obviously a thing, as the dependencies needed for an optional feature are obviously and clearly optional, this is just basic logic. Not only that, you are literally arguing against reality, given this is how it actually works, right now, there's nothing theoretical about this.

What "actually works"? I can have a completely static system with musl, or with iOS where apps are statically linked.

dlopen() hacks and "optional dependencies" are just sloppiness, nothing more. In your case it's to cover up the screwups that happened with journald.

> It's quite plainly obvious given this rant that you have a gigantic chip on your shoulder, so there's not much point in going on. It's very simple: we will use dlopen for most of our dependencies, libsystemd will stay exactly as it is other than that as it has the right design for a number of reasons, and we will work around outdated package managers with something like the elf note discussed in the article

Are you going to support mseal()/mimmutable() or are you going to give a middle finger to people who want security?

Identifying dependencies used via dlopen()

bluca — Fri, 19 Apr 2024 10:09:48 +0000

> First, "optional dependency" is kind of an oxymoron. It's either a dependency or not.

Optional features are obviously a thing, as the dependencies needed for an optional feature are obviously and clearly optional, this is just basic logic. Not only that, you are literally arguing against reality, given this is how it actually works, right now, there's nothing theoretical about this.

It's quite plainly obvious given this rant that you have a gigantic chip on your shoulder, so there's not much point in going on. It's very simple: we will use dlopen for most of our dependencies, libsystemd will stay exactly as it is other than that as it has the right design for a number of reasons, and we will work around outdated package managers with something like the elf note discussed in the article. If one day glibc+gcc give us the same feature mach-o implements with on-demand-resolved weak dt_needed, we'll switch to that.

Identifying dependencies used via dlopen()

Cyberax — Fri, 19 Apr 2024 08:58:44 +0000

> They provide the ability to have optional dependencies that can gracefully downgraded to the feature being disabled, without requiring separate builds, with a minimal overhead for developers

First, "optional dependency" is kind of an oxymoron. It's either a dependency or not.

A more correct way to put it: it papers over a bloated dependency set, by moving dependency resolution from compile-time to runtime. You can compare it with dynamic languages and static languages. Static languages check stuff at compile time, but dynamic languages allow users to just write whatever and delay checking until the code is run.

In this particular case, this also very much applies. dlopen() will interfere with sandboxing, or with mseal()/mimmutable(). It will also introduce failure modes if your code enters a namespace that doesn't have the dependencies.

And all of that, what, to avoid admitting the mistake and splitting libsystemd into libjournald and a lighter libsystemd?

Identifying dependencies used via dlopen()

bluca — Fri, 19 Apr 2024 08:45:53 +0000

They provide the ability to have optional dependencies that can gracefully downgraded to the feature being disabled, without requiring separate builds, with a minimal overhead for developers

Identifying dependencies used via dlopen()

Cyberax — Fri, 19 Apr 2024 01:02:06 +0000

I'll bite. Why are they good? What good do they provide to application writers and packagers?

Identifying dependencies used via dlopen()

dkg — Thu, 18 Apr 2024 21:03:19 +0000

IPC has very different failure modes compared to dynamically loaded libraries (timeouts, partial responses, duplicate responses, DOSes) and security problems (MITM).

I agree with you about the different kinds of failure modes, but the discussions in this thread suggesting lazy dynamically loaded libraries (whether done via dlopen() or by the dynamic linker itself) also introduce some novel failure modes.

figuring out how to wrap any of these scenarios in reasonable error-handling sections so that arbitrary tooling that relies on them can be updated smoothly is the real trick.

Identifying dependencies used via dlopen()

bluca — Thu, 18 Apr 2024 18:55:35 +0000

Nope, they are good and exactly what we need, and that's why we are implementing it everywhere

Identifying dependencies used via dlopen()

Cyberax — Thu, 18 Apr 2024 14:54:45 +0000

Not really. Optional dependencies inside foundational libraries are bad, however you wrap them.

When your feature requires reinventing something fundamental as ELF, then you're probably on a wrong track.

Identifying dependencies used via dlopen()

bluca — Thu, 18 Apr 2024 14:42:32 +0000

dlopen is good but not great - great is what OSX/Mach-o provides, that was already mentioned elsewhere, where the linker/loader can do it in a much nicer way

Identifying dependencies used via dlopen()

Cyberax — Thu, 18 Apr 2024 13:06:08 +0000

> and security problems (MITM)

Others replied about other issues, but I just want to note that MITM doesn't really apply for the local IPC. You can reliably verify the caller's identity using good old Unix sockets (SCM_CREDENTIALS and others).

Identifying dependencies used via dlopen()

farnz — Thu, 18 Apr 2024 11:41:54 +0000

Anyone running SSSD, NSCD, or NSLCD has these failure modes already, along with the failure modes of dynamically loaded libraries. Those three all supply nsswitch plugins that simply do IPC to a daemon to get authorization details. Additionally, SSSD supplies a PAM plugin that does authentication over a socket.

And users of NIS, NIS+, LDAP or Kerberos have this problem but worse, since the IPC is done over a network, not over a Unix Domain Socket.

There's also no reason to insist that everything goes via IPC; you can have a built-in handler for "files" and "pam_unix" file-based methods (optionally disabled, of course), with IPC for the cases where the built-in handlers don't work for your use case. Basically, have a hard-wired set of options, with IPC instead of dlopen for cases where the hard-wired set isn't enough.

Identifying dependencies used via dlopen()

dezgeg — Thu, 18 Apr 2024 11:41:42 +0000

Complex commonly used NSS modules like sssd already use IPC with a daemon.

Identifying dependencies used via dlopen()

gioele — Thu, 18 Apr 2024 11:29:49 +0000

> Definitely a positive for security; it might be a slight negative for performance

What I worry about are failure modes. IPC has very different failure modes compared to dynamically loaded libraries (timeouts, partial responses, duplicate responses, DOSes) and security problems (MITM).

Daemonized PAM/NSS will be hot targets for malicious actors. Will users of these PAM/NSS daemons correctly handle these novel failures?

Identifying dependencies used via dlopen()

Cyberax — Thu, 18 Apr 2024 11:05:44 +0000

Plugin managers are bad! Plugins are bad!

dlopen() to load optional dependencies, on the other hand, is great.

Right?

Identifying dependencies used via dlopen()

LtWorf — Thu, 18 Apr 2024 10:32:18 +0000

Do you see how many vulnerabilities browsers have all the time?

I don't think WASM is so secure as you think it is.

Identifying dependencies used via dlopen()

bluca — Thu, 18 Apr 2024 10:15:11 +0000

"plugin frameworks" make absolutely no sense whatsoever for any of this, and would be just dirty and silly workarounds for outdated and limited package managers

Identifying dependencies used via dlopen()

farnz — Thu, 18 Apr 2024 10:06:31 +0000

Definitely a positive for security; it might be a slight negative for performance in the case where your PAM or NSS stack is very fast in-process, but that should be an extremely rare case - the moment you use a network service for auth, it becomes slow enough that IPC is not the bottleneck.

With IPC to an authentication and authorization daemon (SSSD, for example), you can run the daemon in a known-good environment (so no surprises from LD_PRELOAD, ptrace or similar), with access to files that the process is blocked from accessing (can read /etc/shadow, even if the process isn't authorized to do so). In turn, this would let you lock down access to the configuration for your auth stacks, which you can't do today (/etc/nsswitch.conf has to be readable by any process that wants to ask about authorization, for example).

Identifying dependencies used via dlopen()

Cyberax — Thu, 18 Apr 2024 07:47:19 +0000

> Are we sure that swapping out dynamic linking for IPC is a net positive move?

Unquestionably, yes.

PAM and NSS modules have always been awkward, they depend too much on the environment that is not under their control. This can be kind of an issue for PAM modules, especially if you want them to access devices like hardware tokens or write/read files.

You also need to write them in a very careful manner to not interfere with the application that _uses_ them. You can't just add a dependency on libcurl if you're writing an NSS/PAM module, you can't easily launch threads or mess with signal handlers.

Even tasks that _should_ be straightforward like logging and auditing become non-trivial inside PAMs. And this is just not a good situation to be in, for such a critical part of the system.

We now have a very capable daemon management solution (systemd), with socket activation, proper support for logging and so on. It solves nearly all the issues with PAM/NSS.

The only major missing piece is a standard high-level IPC protocol, although even that role is somewhat adequately fulfilled by dbus these days.

Identifying dependencies used via dlopen()

donald.buczek — Thu, 18 Apr 2024 07:38:13 +0000

> Are we sure that swapping out dynamic linking for IPC is a net positive move?

Maybe not generally, but it would be w.r.t. the specific problems I have described. Also, if well done, it could be a more secure design. And to me, the pam code and API design doesn't look very nice and a replacement would have a chance to have better coding style.

On the other hand, we've utilized both, PAM and NSS with our own modules in the past and currently consider to create another PAM module for authentication. And while the authentication/authorization part of PAM could well be done with very minimal static code in the client talking to a server, which does the complicated or privileged things, there are several pam modules, which need to run complicated code in the client process, for example pam_limits or pam_env. So if you want to keep this functionality, you won't profit much from a split between processes. These difficulties don't exist for NSS, so I think NCSD might be a better design.

Identifying dependencies used via dlopen()

gioele — Thu, 18 Apr 2024 07:14:30 +0000

> Yes, I do. Both PAM and NSS need to die, there should be no dynamic dependencies in the basic stack.
>
> NSS can be replaced by a daemon talking the NSCD protocol, and PAM can be segregated into a separate authentication daemon (like SSSD) or replaced by something else (like ephemeral certificates for SSH).

Are we sure that swapping out dynamic linking for IPC is a net positive move?

(Sincere question from somebody that welcomes every reduction of complexity in the basic stack and TCB.)

Identifying dependencies used via dlopen()

guillemj — Thu, 18 Apr 2024 06:27:47 +0000

It is rather unfortunate that this is trying to promote and "standardize" the anti-pattern of dlopen()ing over an externally defined ABI boundary.

Using dlopen() like this might appear easier (for specific upstreams), than either properly defining a plugin framework (which should not amount to much code anyway), or moving the linking into independent programs that can then be optionally called. This is a hack and a workaround that goes behind the toolchain back, that pretty much just shifts the complexity elsewhere, with reduced safety nets and functionality (for example with dpkg we have dependency granularity up to versioned symbols). If the proposal was to improve the toolchain to support optional linking, then that would be a welcome change, but this is rather disappointing.

I've covered that in the past (with further references):

https://lists.debian.org/debian-mentors/2017/11/msg00196....
https://github.com/systemd/systemd/pull/17416#issuecommen...
https://github.com/systemd/systemd/pull/17416#issuecommen...

Identifying dependencies used via dlopen()

Cyberax — Thu, 18 Apr 2024 00:47:01 +0000

Yes, I do. Both PAM and NSS need to die, there should be no dynamic dependencies in the basic stack.

NSS can be replaced by a daemon talking the NSCD protocol, and PAM can be segregated into a separate authentication daemon (like SSSD) or replaced by something else (like ephemeral certificates for SSH).

Identifying dependencies used via dlopen()

lgerbarg — Wed, 17 Apr 2024 19:37:26 +0000

You can absolutely add platform specific extensions without the ceremony of going through the whole standardization process... Solaris actually has extended ELF in this way, they call it direct binding: https://en.wikipedia.org/wiki/Direct_binding