GNOME's new versioning scheme

Posted Sep 19, 2020 1:15 UTC (Sat) by mathstuf (subscriber, #69389)
In reply to: GNOME's new versioning scheme by nix
Parent article: GNOME's new versioning scheme

I think, but may be wrong, that other formats (Mach-O and PE mainly) actually have facilities to bind to symbols in given libraries, so collisions there are actually "okay" since you also look up symbols through a library name filter. I do wish ELF didn't do the "one big bag of symbols" approach, but we're stuck with that I guess. I suppose you could mitigate it with symbol versioning to differentiate between libgtk3 and libgtk4, but that's not always easier and the entire ecosystem would probably need to do it.

And yes, embedding libraries is a real pain. We do it because we have Windows and macOS deployments and who is going to have a viable HDF5 SDK just laying around there? No one, that's who. But I make sure that every library name, header file, and exported symbol is mangled to avoid conflicts. Sometimes static globals still bite us, but those errors usually have the symbol name in them so we can go and fix it. We don't mangle data types because no one should be including our mangled library and a real one in the same TU anyways. Alas, Python module paths/names are also nigh impossible to mangle in practice, so we do end up colliding there, but AFAIK, there's nothing we can do about that.

GNOME's new versioning scheme

Posted Sep 19, 2020 11:40 UTC (Sat) by nix (subscriber, #2304) [Link] (5 responses)

This is what the combination of -Bsymbolic (to bind to the locally-visible definitions at link time and avoid interposition: -z now might help too) and DT_GROUP (to prevent the locally-used Gtk from "leaking out" to supervening libraries) is meant to do. I'm not entirely sure it works 100%: I know glibc has historically had painful-to-even-describe bugs in this area. Still, in the absence of a working dlmopen (which is an even *bigger* pile of pain to use even when it does work), or renaming all symbols and publically-visible types (even worse!) this is the best we can do. It is not a very good best.

GNOME's new versioning scheme

Posted Sep 19, 2020 18:46 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (4 responses)

Hmm, my reading of the flags is as such:

- -Bsymbolic: don't look for symbols we know are in our own library anywhere else

So this doesn't seem like it would help with other code using my symbols from getting confused. Do you have documentation links for DT_GROUP? My manpages have nothing about it. From your descriptions, it sounds like it restricts symbol lookups to those found via DT_NEEDED for the library itself? So basically, a better `RTLD_LOCAL`. How does this interact with weak symbols (like, say, Anaconda where libpython.so symbols are meant to be provided by the loading executable)?

GNOME's new versioning scheme

Posted Sep 19, 2020 19:24 UTC (Sat) by nix (subscriber, #2304) [Link] (3 responses)

-Bsymbolic applied to (say) the link of Gtk 4 itself stops Gtk's own internal calls to its own symbols from being interposed by some other Gtk that happened to have been loaded earlier.

DF_GROUP (as usual I got this one wrong, it's a flag, not a tag) constrains symbol searches from within this library to happen only within the library and its transitive DT_NEEDED libraries: it stops global symbols or symbols in other branches of the search tree from interposing. The relevant ld flag to look for is '-Bgroup'. It is *not* the same as --start-group/--end-group or the linker script GROUP command: fairly confusing naming really. Weak symbols still work, but only if they're resolved by objects loaded by this shared library or things it loaded. (So the old pthreads trick wouldn't work, not that it's a good idea anyway in glibc 2.32+.)

GNOME's new versioning scheme

Posted Sep 22, 2020 16:33 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (2 responses)

Hrm. The fact that it is a library-wide flag works most of the time, but really fails for plugins. I'd like them to load their novel dependencies directly, but those from the loading application usually should just be expected to be there. Maybe `--warn-unresolved-symbols` can help to alleviate that though.

I have a PR (that's likely fallen on deaf ears) for macOS' linker to have a "any symbol you find in this library should be looked up at runtime" behavior which is, I think, exactly the kind of behavior I'd like. https://github.com/apple-opensource/ld64/pull/1

For reference, the search term to use for the ELF flag is `DF_1_GROUP`.

GNOME's new versioning scheme

Posted Sep 22, 2020 21:38 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

I don't know what "looked up at runtime" means: all symbols in dynamically-linked programs are looked up at runtime, that's what dynamic linking *is*. Perhaps you mean "look for these symbols in the main program"? Because if so, that's the ELF default: the search scope for all symbols starts with the main program and goes on from there. (Well, OK, actually it starts with ld.so itself, and the main program comes right after that, usually followed by libc.)

GNOME's new versioning scheme

Posted Sep 22, 2020 21:54 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

As opposed to getting "this symbol was not found" errors at link time. So if I use `-Bgroup`, symbols will be searched for in the loading chain and then *only* the DT_NEEDED libraries? Or does the library/executable providing a plugin API also need to be in the DT_NEEDED section for DF_1_GROUP to work appropriately?