The ABI status of ELF hash tables

When the dynamic linker starts a program, it must resolve all of the symbol references into shared libraries (including glibc). That can involve looking up thousands of symbols in long lists. Since this process must complete before an application can actually start running, it needs to happen quickly. Nobody likes a long delay between starting nethack and facing off against that first kobold, after all. So it is not surprising that some effort has gone into optimizing symbol lookup.

When the ELF file for a shared object is created by the linker, one of the sections stored therein contains a hash table for the symbols in that file. This hash table can be used to speed the lookup process and get the application underway. For many years, the System V standard for the format of this table has been DT_HASH; that format is supported by the toolchains on Linux. In 2006, though, the DT_GNU_HASH format was added as well; it includes a number of improvements intended to get nethack players into their dungeons even more quickly, including a better hash algorithm and a Bloom filter to short-circuit the search for missing symbols. This format is not well documented, but this 2017 blog post gives an introduction.

Since the hash table lives in its own ELF section, there is nothing preventing an ELF file from having more than one of them. Linkers on Linux systems can be told to create one format or the other — or to create both, each in its own section. Until recently, glibc has been built (by default) with a linker option explicitly requesting that both formats be created. That changed, though, with the glibc 2.36 release at the beginning of August; it contained a simple patch from Florian Weimer causing only the DT_GNU_HASH format to be generated.

At this point, glibc 2.36 has been installed onto a number of systems running the faster-moving Linux distributions, and few people have noticed the change; the DT_HASH format has not been used for anything on those systems in many years, and the only consequence of its removal is regaining a small amount of disk space. Game players, though, were not so lucky; naturally, the problem relates to a piece of proprietary software that cannot be easily changed.

The Easy Anti-Cheat (EAC) system is a proprietary tool from EPIC intended to prevent game players from cheating by way of modifications to the game executable itself. As one might expect, the actual heuristics used are not documented anywhere and the code is secret, but one of the techniques involved appears to be looking at the symbols in the game executable and ensuring that they match the expected values. To do this, EAC goes rooting through the DT_HASH tables; if an expected table isn't present, EAC proudly proclaims that it has caught a cheater and does not allow the game to run.

Gamers, it seems, find this behavior disappointing. Arkadiusz Hiler, for example, blogged:

I think this whole situation shows why creating native games for Linux is challenging. It’s hard to blame developers for targeting Windows and relying on Wine + friends. It’s just much more stable and much less likely to break and stay broken.

Hiler reported the problem in the glibc bug tracker; the discussion later moved to the project's mailing list as well. The report stated that the change "breaks SysV ABI compatibility", suggesting that it should be reverted. The responses from the project were not entirely sympathetic to that cause, though. Weimer answered that: "Any tool that performs symbol lookups needs to support DT_GNU_HASH these days". Adhemerval Zanella said: "I am not sure this characterizes as an ABI break since the symbol lookup information would be indeed provided (albeit in a different format)"

Carlos O'Donell agreed that this change was not an ABI break:

Software that is an ELF consumer on Linux has had 16 years to be updated to handle the switch from DT_HASH to DT_GNU_HASH (OS-specific).

While I'm sympathetic to application developers and their backwards compatibility requirements, this specific case is about an ELF consumer and such a consumer needs to track upstream Linux ELF developments.

He went on to say that characteristics of the generated ELF file are not part of the glibc ABI, even if changes there break applications, and suggested that the bug should be closed as "won't fix". He also requested that the EAC developers provide reasons for why DT_HASH should be retained by default. As of this writing, the bug remains open.

Blaming the EAC developers for not keeping up with Linux ELF hash-table formats might not be entirely fair. The DT_HASH format is mandated by the System V ABI specification, the DT_GNU_HASH format is undocumented, and there has been no deprecation campaign to get users to move on. Chances are those developers are as surprised as anybody and haven't just been ignoring the "switch to DT_GNU_HASH" entry languishing in their issue tracker for the last decade or so. Regardless of blame, though, something needs to be done to solve this problem and save gamers from the prospect of having to get some actual work done.

If EAC were free software, of course, chances are there would already be a patch circulating to deal with the problem. As it is, only its owner can deal with this problem directly. Meanwhile, though, there is another workaround available: distributors can easily patch the glibc build to restore the DT_HASH section and make the problem go away for now. Doing that and giving EAC (along with a few other programs) some time to move to DT_GNU_HASH seems like the best solution from just about any point of view.

Note: you can avoid this step in the future by logging into your LWN account.