LWN: Comments on "The ABI status of ELF hash tables"

The ABI status of ELF hash tables

sammythesnake — Wed, 26 Oct 2022 18:36:02 +0000

I don't have any skin in this game, but if I were to try to justify classifying this as "not an ABI change", I think I'd say that the format of the symbol hash is an implementation detail not intended for use outside of the linker.

If the format isn't documented[0] in the documentation for developers using the library, that would be consistent with that position.

It's a weaker guarantee than the one provided by the kernel, though (using the Hyrum's Law[1] model of what constitutes an ABI) but it's a valid position to take if the hash format *is* an implementation detail.

[0] I have very little interest in reading the libc documentation, so maybe it is. The *newer* format is apparently not well documented, though, and the old format predates glibc's use of it, so it being documented for that reason doesn't necessarily torpedo this line of reasoning - it being listed inn the glibc documentation in language other than as an implementation detail would, though...

[1] https://www.hyrumslaw.com/

The ABI status of ELF hash tables

sammythesnake — Wed, 26 Oct 2022 17:54:34 +0000

So, "intuitive minimax" then?

https://en.m.wikipedia.org/wiki/Minimax

To be fair, I use this kind of thing (along with other reasonable understandings of less deterministic but still somewhat predictable behaviour - something more akin to a Bayesian model, but again more intuitive than rigorous) all the time while doing such boringly quotidian things as driving - it's not at all rare that I twig somebody's planning to execute a turn a fair time before they remember that indicators are a thing, for example (assuming they ever actually do - BMW/Audi/etc. stereotypes notwithstanding) It's literally saved my life on a few occasions...

The ABI status of ELF hash tables

daenzer — Fri, 16 Sep 2022 08:25:38 +0000

> The game should have shipped with all required .so libraries in the first place, though. Including libc.

I don't think I've ever seen a game ship its own copy of libc, but I have seen lots of them ship their own copies of libstdc++ (and various other libraries).

The result was invariably that the game stopped working sooner or later, because its copies of those libraries were too old for other things (usually GPU driver components) mapped into the process. Removing the games' copies of the libraries made them work again.

libc would be even more likely to hit this, since basically everything links against it, it never changes SONAME, and it often bumps the symbol version of existing ABI symbols.

The ABI status of ELF hash tables

nix — Tue, 06 Sep 2022 08:52:08 +0000

> I also find it appalling that they say that they are somehow "upstream ELF development"

Looking at it practically... they are, and have been for well over a decade at this point. Perhaps 15--17 years: i.e., since about as long ago as the introduction of DT_GNU_HASH!

The ABI status of ELF hash tables

Darkstar — Fri, 02 Sep 2022 13:45:10 +0000

> I’m suspicious of anything that requires me to run a driver in kernel mode just to play a game.

You probably wouldn't play any competitive First-Person Shooter games anyway, I assume?

Because if you did, and you knew that everyone else could be cheating on you without you knowing, I guess you would be more open to having a kernel-based anti-cheat driver.

The ABI status of ELF hash tables

Vipketsh — Thu, 01 Sep 2022 19:43:47 +0000

Surely you jest. You are also complaining to all those new fangled static-link only languages for wasting your disk space, right ?

On my raspberry, which still has the DT_HASH entry, the section is less than 16kb. So, the glibc developers decided that compatibility is just not worth 16kb, in an otherwise 2mb big binary. It's not like the developers need to put in work just to keep backwards compatibility -- it's just an option to an external program for crying out loud!

It's very elitist how the glibc developers are handling this case.

The ABI status of ELF hash tables

Wol — Mon, 29 Aug 2022 08:56:26 +0000

Can you do that if it's a binary you don't have source for?

Eg I've got a libc5 binary I would love to run, but it was a commercial product ...

Cheers,
Wol

The ABI status of ELF hash tables

HenrikH — Sun, 28 Aug 2022 21:04:36 +0000

>If you want to run binary on older system you have to use older libc. It's as simple as that.
Or you compile the binary for the old symbol with the new glibc, since glibc contains versioned symbols including all the old ones you can tell the linker exactly which version of the function that your binary wants to use. So while it is a real pain it can be done.

The ABI status of ELF hash tables

WolfWings — Sat, 27 Aug 2022 21:36:00 +0000

To clarify, this is the first time I'm aware of that such a change landed that can't simply be aliased around by tacking an older version of a given function into your codebase, or even in most cases simply aliased to the old version.

They've never done a breaking change at the startup libc functions before.

The ABI status of ELF hash tables

milesrout — Thu, 25 Aug 2022 04:47:15 +0000

The bigger that my disk is, the bigger 1% is. Of course 1% disk space is still appreciable in the modern age.

The ABI status of ELF hash tables

JanC_ — Tue, 23 Aug 2022 15:24:39 +0000

I think you mean DT_HASH (not DT_GNU_HASH) is de facto deprecated.

Keeping it available might be useful for some general purpose distros that want compatibility with very old binaries, but seems useless for e.g. most embedded systems or cloud images.

The ABI status of ELF hash tables

JanC_ — Tue, 23 Aug 2022 15:12:29 +0000

DT_HASH wasn’t removed though, it’s just no longer included by default if you use the upstream build options.

The GNU/glibc people should have documented this better though (both the new table format as well as the changes to the defaults). Based on that, operating systems (distros) could then choose to maintain compatibility with SysV or not.

The ABI status of ELF hash tables

mathstuf — Tue, 23 Aug 2022 11:47:30 +0000

> Not to mention if you load C++ libraries with RTLD_LOCAL they usually can't be unloaded anyway

Pretty much anything with global statics is nigh unloadable already. macOS just no-ops unloading anything that mentions `thread_local` as well. I don't see much value in being able to unload libraries. Windows is a bit different because there are APIs to query libraries without actually loading them (but is not that useful to anything caring about cross-platform support).

The ABI status of ELF hash tables

bartoc — Tue, 23 Aug 2022 02:55:07 +0000

What windows does is a lot closer to dlmopen than it is to RTLD_LOCAL. In windows if you load (or link against) two dlls that both link against the same dll themselves, then they can very well use different versions of that dll.

You can actually have stuff like multiple different C runtimes in the same process (although it's super likely to result in sadness and chaos as soon as one touches the other). Still, if you are very careful it is possible. They do need to be named differently though.

Not to mention if you load C++ libraries with RTLD_LOCAL they usually can't be unloaded anyway, _AND_ some (but not all) of their symbols will actually be used to resolve references in future libraries. RTLD_LOCAL can also be promoted to RTLD_GLOBAL if the library is a dependency of an RTLD_GLOBAL library.

dlmopen provides something much more similar to windows DLLs, but you can only have 16 namespaces and according to the manual (typos from there):
> As at glibc 2.24, specifying the RTLD_GLOBAL flag when calling
dlmopen() generates an error. Furthermore, specifying
RTLD_GLOBAL when calling dlopen() results in a program crash
(SIGSEGV) if the call is made from any object loaded in a
namespace other than the initial namespace.

So it seems like it's a little useless. Someone should fix this stuff, and perhaps allow some kind of namespacing for stuff used via the linker, without dlsym. I guess if you really want that you can just use wine's loader.

The ABI status of ELF hash tables

bartoc — Tue, 23 Aug 2022 01:49:29 +0000

The "norm" should probably be to use -z defs -z lazy --no-allow-shlib-undefined. That is "report a link-time error if any symbols are undefined, but don't tell the loader to resolve all symbols at startup, and also report undefined symbols that are in shared libraries I linked to". For a lot of applications using "-z now" is allright, but for a lot of libraries (and some apps) -z lazy is better because the set of used symbols depends highly on the path execution takes. Think of a library like OpenImageIO or VTK that has a ton of format parsing libraries linked to it, most of which won't be used by a given application.

The ABI status of ELF hash tables

bartoc — Tue, 23 Aug 2022 01:38:26 +0000

Go kinda has to avoid glibc because of how it's coroutines work, fundamentally it has to be able to turn what would be syscalls in glibc into "submit async work then start running a new coroutine"), This is basically why I really don't love go's concurrency approach, I would rather just eat the size of the kernel stack and use a real thread.

You probably need something like io_uring to actually save syscalls with this approach, and I don't think go uses it.

The ABI status of ELF hash tables

bartoc — Mon, 22 Aug 2022 21:54:27 +0000

This kinda stuff works if ALL libraries that depend on libc (which basically means all libraries) are bundled with the game. Unfortunately, that is not a good option, since you really want to use (at least) the graphics stack from the host system. Additionally, you probably do want libraries like openssl to update out from under you, either from something like the steam runtime, a flatpak runtime, or the distribution itself. Guess what: libraries like openssl and gnutls load plugins, and those plugins assume that they are being loaded into a process with a glibc at least as new as the one the openssl/gnutls was built for.

Basically, it's really bad to bundle shared libraries if they could possibly be dependencies of anything you want to load from the base system. If you really want to bundle such libraries, you should probably consider reangling their symbols, but that's not a good option for something like glibc as it would make things like threads work in really bizarre ways.

The ABI status of ELF hash tables

flussence — Mon, 22 Aug 2022 19:58:21 +0000

> Consider an esports context where an entire team's computers (and thus, half the computers in the game) could be colluding.

If it's an entirely remote event then the problem is interesting, but at a live event I'd imagine a client-server setup where one team's clients are raising red flags with the server and the others are silent would quickly become embarrassing to the cheaters.

FWIW, even proprietary anti-cheat software hasn't stopped people cheating all the way to finals on stage and then getting caught there.

The ABI status of ELF hash tables

mathstuf — Mon, 22 Aug 2022 17:00:47 +0000

It is not the same. RTLD_LOCAL is not usable in open-ended "I load plugins via dlopen" mechanisms. The problem is when you have:

pluginA.so -> liblang.so
pluginB.so -> liblang.so

Loading pluginA with RTLD_LOCAL makes liblang.so enter the same state. When pluginB is loaded, it fails because it is not "allowed" to access liblang.so because it is "local" to pluginA. It's probably more useful when you know plugins are standalone or you're doing some restricted delay-load mechanism.

The ABI status of ELF hash tables

larkey — Mon, 22 Aug 2022 14:40:45 +0000

Yep, I often find myself on the GNU side of things (despite still being heart broken over Solaris) but this?! DT_HASH is

* standard
* universally available
* never officially deprecated (blog posts don't count)
* documented
* cross platform

The GNU alternative is

* GNU/Linux specific
* undocumented
* non-standard

Simply removing it is just absurd and the reason stated is also just sad. If I care about a *very* small foot print I wouldn't be using the GNU tooling anyway, likely :-)

I also find it appalling that they say that they are somehow "upstream ELF development" and further that everyone dealing with this need to follow all their blogs? This is just hybris.

The ABI status of ELF hash tables

cortana — Mon, 22 Aug 2022 08:40:59 +0000

Hmm

I've always found glibc's documentation patchy at best. Lots of it is documented nicely in the manual, but there are chunks (pthreads, getaddrinfo, the _l variants of the ctype/string functions) that are missing entirely!

The ABI status of ELF hash tables

NYKevin — Mon, 22 Aug 2022 07:12:21 +0000

The rules prohibit such collusion, by assumption (or else it would not be brought up in a discussion of cheating in the first place). So your suggestion boils down to "don't play if you don't like other people cheating." Which is terrible for obvious reasons.

The ABI status of ELF hash tables

maskray — Mon, 22 Aug 2022 00:19:57 +0000

> Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair.

Probably. But DT_HASH has mostly disappeared from Fedora for 16+ years and several other distributions for similarly long years.
libc.so.6 does contain DT_HASH for a long time, but it is just a rare exception.

> The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on. Chances are those developers are as surprised as anybody and haven't just been ignoring the "switch to DT_GNU_HASH" entry languishing in their issue tracker for the last decade or so. Regardless of blame, though, something needs to be done to solve this problem and save gamers from the prospect of having to get some actual work done.

I think it is fair to say that DT_GNU_HASH is de facto deprecated.
DT_GNU_HASH not in generic ABI is not an issue since In general a processor supplement ABI or an operating system ABI can replace a generic ABI feature, and we should not read too much from the generic ABI wording.

I have more to say on https://maskray.me/blog/2022-08-21-glibc-and-dt-gnu-hash

The ABI status of ELF hash tables

k8to — Sun, 21 Aug 2022 23:52:18 +0000

Either way, the easy anti-cheat developers just admitted that this component of their tool is easily misled, which kind of invalidates the utility of having that functionality especially now that it's public knowledge.

The ABI status of ELF hash tables

bartoc — Sun, 21 Aug 2022 21:53:40 +0000

sometimes (probably more often) called "game sense" it's the ability to basically use very limited information to predict what your opponent is doing. Basically knowing the game the metagame so well you can basically tell what your opponent is doing by just thinking "what would I do" and evaluating what actions they could do would be worst for you.

In any event really good game sense can result in you taking actions to respond to stuff before you can see it. You see this in Starcraft a lot, it can look like someone is using maphacks but in reality they are just really experienced.

The ABI status of ELF hash tables

jkingweb — Sun, 21 Aug 2022 20:33:22 +0000

> You could try and detect that the player was seeing more info than they should behaviorally, but then you could start detection "star sense" as cheaters, which is not great.

What is "star sense"?

The ABI status of ELF hash tables

fw — Sun, 21 Aug 2022 16:41:36 +0000

glibc already prefers DT_GNU_HASH over DT_HASH when an object has both. I assume it's been this way since the introduction of DT_GNU_HASH.

I haven't look at the tool. It's not just consistency checks those DT_HASH lookups could be used for. Maybe they use DT_HASH lookups to find the original definition of dlsym after interposing it.

The ABI status of ELF hash tables

k8to — Sun, 21 Aug 2022 13:23:56 +0000

I love it.

The ABI status of ELF hash tables

marcH — Sun, 21 Aug 2022 13:21:08 +0000

> Is it surprising that languages other than C avoids the C runtime library?

Yes and no https://gankra.github.io/blah/c-isnt-a-language/

> pretty much all other Linux distributions are "GNU/Linux" and use glibc.

If you look beyond "interactive" users and at the number of instances of Linux kernels running, I suspect Linux "distributions" are still niche. Still waiting for the "Year of the Linux Desktop" and breakages like this one don't exactly help.

> Rust, of course, uses glibc.

... on Linux "distributions".

The ABI status of ELF hash tables

jhoblitt — Sun, 21 Aug 2022 13:15:47 +0000

There is a package to add glibc to alpine as there is a surprising amount of software which is broken by musl. It don't use alpine as an OCI base layer as often as I used to because of the occasional random problem caused by software that depends on glibc.

For better or worse, glibc is defining a defacto ABI for "Linux".

The ABI status of ELF hash tables

jengelh — Sun, 21 Aug 2022 11:14:50 +0000

>on Windows the symbols are namespaced to the name of the dll they bind to, [on Linux and other ELF platforms symbol names are global]

That sounds just like RTLD_LOCAL on ELF systems. Moreover, I don't see ld.so using RTLD_GLOBAL during the initial program start, that seems to be a dlopen-exclusive. So shouldn't symbols be appropirately namespaced? (But indeed I remember symbols sharing the global scope, dunno what's going on.)

The ABI status of ELF hash tables

willy — Sun, 21 Aug 2022 10:38:04 +0000

Nobody seems to have pointed out the obvious solution here -- make it in the EAC developers interests to fix the bug.

If the linker preferentially uses DT_GNU_HASH and ignores DT_HASH, and EAC only checks DT_HASH, then create cheats for popular games that include a faked DT_HASH section which pretends everything is OK.

The ABI status of ELF hash tables

jengelh — Sun, 21 Aug 2022 09:30:53 +0000

>Consider an esports context where an entire team's computers (and thus, half the computers in the game) could be colluding.

Any 2-team n-vs-n player setup reduces to a 1-vs-1 problem, because the other team could *also* be colluding.
Playground piece of wisdom: If you don't like the rules or how they are applied by the other player(s) and/or the umpire (if there is one), stop participating.

The ABI status of ELF hash tables

comex — Sun, 21 Aug 2022 06:52:02 +0000

At least on macOS, there is no shipping of old versions of libc. Instead, there's a compiler flag to explicitly declare the minimum OS version you want the binary you're compiling to run on. Each function declared in OS header files includes a minimum OS version, and attempting to use an API that's too new is a compilation error. In situations like the mentioned one, where the ABI for existing APIs changes (such that newer OSes can run old binaries but not the reverse), the header files, or sometimes the compiler or linker itself, will pick the old or new ABI depending on the same declared minimum OS version.

That said, Xcode does eventually drop support for targeting old OS versions; the current lowest minimum-OS option is 10.9 from 2013.

The ABI status of ELF hash tables

drago01 — Sun, 21 Aug 2022 05:44:07 +0000

No, but in that case it is even documented.

The ABI status of ELF hash tables

k8to — Sun, 21 Aug 2022 04:54:58 +0000

Maybe I'm really slow, but I'm not sure how this is particularly relevant.

I don't see the internal details of the structures used by the dynamic linker as much of an ABI myself. Yeah, the old format is mandated by SysV, but I don't care if my linux distribution is SysV compliant. I don't see anything useful provided by the SysV specification at this point. Needless change is wasteful but that exact point in history doesn't seem interesting to me. I also think it's right to chide those responsible for moving us to an undocumented format. It's not something many people will need to know, but a transparent system is a fixable system.

But all that aside, I don't see how symbol namespacing really means something one way or another in this case.

The ABI status of ELF hash tables

bartoc — Sun, 21 Aug 2022 03:02:06 +0000

The problem isn't really the syscall ABI, a bigger problem is nss plugins. Have fun not being able to resolve hostnames properly because your nss plugins won't load with an old version of glibc. You can static link them, but who knows if you'll get everything you need, or in the right configuration.

Plugins in general are what really screw up applocal deployment on linux, you just have to be aware of them, and glibc loads plugins on almost all linux systems.

The ABI status of ELF hash tables

bartoc — Sun, 21 Aug 2022 02:57:02 +0000

You could try and detect that the player was seeing more info than they should behaviorally, but then you could start detection "star sense" as cheaters, which is not great.

Ultimately an open source anti-cheat probably could catch a lot of people, anti-cheat by it's nature tends not to catch the kind of people that can decipher anti-cheat source code. It's more to catch people using known cheats I think.

The ABI status of ELF hash tables

josh — Sat, 20 Aug 2022 23:41:35 +0000

I'm normally all for "any interface people rely on is ABI", as the Linux kernel does. But there's a limit to that. Suppose something hashed the assembly code of `memcpy` and broke if it changed? Suppose something hashed ld-linux.so (the dynamic linker) and broke if it changed? Would you consider those "ABI"?

The ABI status of ELF hash tables

LtWorf — Sat, 20 Aug 2022 23:30:38 +0000

You never ran ldd on a go program uh?

They do link libc