LWN: Comments on "GNU C library 2.33 released"

GNU C library 2.33 released

nix — Mon, 08 Feb 2021 15:55:00 +0000

Other fallout: openssh fails on 32-bit platforms due to needing some 64-bit time syscalls added to the seccomp filter list. (Patch submitted.)

Also some obscure --enable-stack-protector=all failures, but I suspect that use of that particular configure flag is quite rare (patches in preparation but they're all in the testsuite anyway and do not affect glibc itself).

GNU C library 2.33 released

nix — Mon, 08 Feb 2021 13:49:04 +0000

Nope, this is real. march=... does not (reliably? at all) affect the ISA stamping done by sysdeps/x86/isa-level.c, leading to e.g. 32-bit glibcs being built on fast machines for execution on old slow machines being stamped as requiring whatever fast x86-64 box you did your builds on, even if it was built with -march=i586 or something. (ld.so validates this using the CPUID and has no way to override the validation, so everything seems to work when you test it on your build host, even if you test in a VM, and then fails disastrously when you install it on the target machine, with every binary failing to run, even though they would all run just fine if only ld.so let them).

See this thread: https://sourceware.org/pipermail/libc-alpha/2021-February... (HJ's initial patch is definitely not sufficient).

For now, it's safest to turn off ISA stamping if you're building on one machine for execution on any machine which might be older (or otherwise of a lower ISA level). Of course there is no configure flag for this: you have to override the cache variable by adding libc_cv_include_x86_isa_level=no to your configure line.

GNU C library 2.33 released

nix — Sun, 07 Feb 2021 17:54:11 +0000

On second thoughts, I think this particular instance was caused by the confluence of multiple local patches (one for the system gcc that passes in -march=native by default, then a specs file that overrides it with march=geode when building for the firewall, combined with a glibc makefile that does not always pass local CFLAGS down, so that one object file ended up being built without -march=geode. It didn't use any of the things the higher ISA level called for, but the arch notes system doesn't know anything about that, and the lack of an explicit arch on that one file led to everything being contaminated with an arch of x86-64-v2. Sigh.)

GNU C library 2.33 released

nix — Sun, 07 Feb 2021 17:27:43 +0000

Well upgrading my firewall was fun! We have more x86-32 rot! (This is *after* recovery, with all /lib links pointed back to 2.32.)

nix@fold 4 /home/nix% /lib/ld-2.33.so --preload /lib/libc-2.33.so /bin/ls
/lib/libc-2.33.so: CPU ISA level is lower than required

Looks like -march=geode does not cause binutils to guess right, at least as of binutils 2.35 (yes, I know this is old, but it still shouldn't cause glibc to fail like this):

Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x0000000c NT_GNU_PROPERTY_TYPE_0
Properties: x86 ISA needed: x86-64-v2

(will report to the glibc list.)

Looks like the only way to turn off this stuff right now is to pass libc_cv_include_x86_isa_level=no to configure, which gentoo and others have just started doing.

GNU C library 2.33 released

anton — Sat, 06 Feb 2021 17:49:26 +0000

At least Skylake (and its variants up to Comet Lake) can do AVX/AVX2 work with the upper 128 bits of the unit powered down, at lower performance (and it works that way until the upper 128 bits have been powered up). So a better way to deal with both market segmentation and partially failing SIMD units would be to support AVX but on the Pentiums run it without the upper 128 bits of the functional unit. If you wonder about non-working lower 128 bits: then SSE won't work and they cannot sell the die even with disabled AVX.

GNU C library 2.33 released

anton — Sat, 06 Feb 2021 17:06:06 +0000

Well, it seems that story got a happy ending, in that the -hwcaps is the ostensibly vendor-neutral solution to the same problem the old vendor-specific "haswell" subdirectory tried to solve?

Sorry for mentioning hwcaps, which played no role in this. As the bug report explains, "haswell" is documented as vendor-neutral (despite its name), with the same meaning as "x86-64-v3", but the implementation worked only on Intel CPUs as documented, by explicitly checking for Intel CPUs (like Intel does in the MKL) and using the instruction set extensions only for them. The bug was not fixed for two years, and it's not clear that it is fixed now; and the reason given does not sound convincing at all. So, can we expect that "x86-64-v3" will fare any better than "haswell"?

Concerning chip area, AMD has demonstrated that AVX can be implemented with 128-bit wide functional units, including the low-power low-cost Jaguar (competition for Intel's Atom family).

Concerning packaging, at least with Debian's multiarch support, treating them as separate architectures and compiling all libraries for every level might be less work. But distributions that only have lib32/lib64 to deal with multiple architectures cannot go there.

GNU C library 2.33 released

Wol — Sat, 06 Feb 2021 16:54:54 +0000

> Stupid indeed. One can understand not wanting to spend chip area on large vector units, but fusing out functional units for product segmentation reason? Argh.

Or is this a bit of complex real-estate that has a habit of not working? Fusing out stuff that fails QA and segmenting the product has some logic behind it.

Cheers,
Wol

GNU C library 2.33 released

joib — Sat, 06 Feb 2021 12:23:24 +0000

> However, looking at Bug 23249, one can get the impression that Intel succeeds in hampering AMD CPUs not just in code compiled with the Intel compiler and using Intel-supplied libraries, but also in free software; and if they succeeded in the hwcaps case, why should they fail in the instruction-set-level case?

Well, it seems that story got a happy ending, in that the -hwcaps is the ostensibly vendor-neutral solution to the same problem the old vendor-specific "haswell" subdirectory tried to solve?

> Lots of Intel CPUs disable this instruction set feature or that, even on silicon where it is implemented (e.g., Skylake Pentiums have AVX disabled); apparently Intel's marketing thinks that they can perform market segmentation in that way.

Stupid indeed. One can understand not wanting to spend chip area on large vector units, but fusing out functional units for product segmentation reason? Argh.

> Will distributions compile every packet for four different levels and install the one for the particular machine?

I would guess that the binary package for libfoo would contain the .so's for all 4 microarchitecture levels (or if the developers or distributors do benchmarking, only those levels where it makes a difference). Then again, I recall Debian/Ubuntu used to have libc6-i686 which was glibc compiled for the i686 target rather than the default i386 or i486, so maybe they will do separate packages. Sounds tedious, but oh well.

GNU C library 2.33 released

joib — Sat, 06 Feb 2021 12:04:13 +0000

> The architecture levels are something that you set the compiler to compile for and for the dynamic linking mechanism to use in searching for libraries (not sure how it will work out for finding executables; do we have to set the PATH depending on which machine we are working on?). So the idea is that people will compile a library for, say, the v3 level, and if the auto-vectorizer finds something to vectorize, it will use AVX and AVX2 rather than, at most, SSE2.

Yes, that is my understanding as well. I don't think there is an equivalent mechanism for executables (like you say it could be done with $PATH, but, ugh).

> So this will be more automatic and require less development effort to make use of instruction set extensions than what glibc does for its memmove variants.

Yes. Though they are separate, there is a (minor) bloat reduction potential in not generating function versions "below" the microarchitecture level. But I guess the implementation could be somewhat hairy, and it's just a little bit of extra space on disk as it won't be loaded into memory, so maybe not worth worrying about.

> I wonder whether the benefits are enough to make this a success. It reminds me of the x32 ABI; it had some demonstrated performance advantages over the i386 and x86-64 ABIs, but in the end it was not enough to make it fly.

I think the big difference here is that it's not a new ABI, requiring the recompilation of everything. Rather this can be opt-in on a per-library basis, with the levels used chosen by the developers or distro packagers depending on what performance impact it might have. Say, glibc could provide all of x86-64{,-v2,-v3,-v4}, libfoo could provide x86-64{,-v3} and libbar could provide only the base x86-64, and depending on what hardware you run on the dynamic loader would use the best available variant of each library, mixing them freely.

GNU C library 2.33 released

anton — Sat, 06 Feb 2021 10:40:07 +0000

That could be the case, unless somebody adjust the source code to not compile it unless the level is v2.

Anyway, reading this message makes the whole idea much clearer to me: The architecture levels are something that you set the compiler to compile for and for the dynamic linking mechanism to use in searching for libraries (not sure how it will work out for finding executables; do we have to set the PATH depending on which machine we are working on?). So the idea is that people will compile a library for, say, the v3 level, and if the auto-vectorizer finds something to vectorize, it will use AVX and AVX2 rather than, at most, SSE2.

So this will be more automatic and require less development effort to make use of instruction set extensions than what glibc does for its memmove variants.

I wonder whether the benefits are enough to make this a success. It reminds me of the x32 ABI; it had some demonstrated performance advantages over the i386 and x86-64 ABIs, but in the end it was not enough to make it fly.

GNU C library 2.33 released

anton — Sat, 06 Feb 2021 10:20:41 +0000

It is unlikely that AMD will add their own extensions, at least as long as they don't dominate the market for AMD64 architectures. They tried that with 3DNow!, it did not become popular, and since about 2011 AMD CPUs have dropped 3DNow! (but they had to support it for quite a while, in K8 and K10). FMA4 is a similar story.

But if AMD should choose to not implement AVX-512 (I expect they will support it at some time), AMD users run the code compiled for v3; I don't see a problem here. However, looking at Bug 23249, one can get the impression that Intel succeeds in hampering AMD CPUs not just in code compiled with the Intel compiler and using Intel-supplied libraries, but also in free software; and if they succeeded in the hwcaps case, why should they fail in the instruction-set-level case?

As for Intel CPUs not supporting instruction set extensions, it's not just the Atom product line. Lots of Intel CPUs disable this instruction set feature or that, even on silicon where it is implemented (e.g., Skylake Pentiums have AVX disabled); apparently Intel's marketing thinks that they can perform market segmentation in that way.

The only thing it achieves is that almost nobody uses the more advanced features. This instruction-set level idea has a certain potential at improving this situation, but I wonder: Even if it's as easy as invoking gcc with some architecture-level option, I guess that most upstream developers will set up their build infrastructure to just compile for the base level by default (and the builder configure to have it compiled with specific options). Will distributions compile every packet for four different levels and install the one for the particular machine?

GNU C library 2.33 released

dvdeug — Sat, 06 Feb 2021 06:08:57 +0000

Looking at the PDFs joib so helpfully linked to, the latest level is AVX-512, which AMD has apparently issued no public plans to support. It seems not impossible that AMD will never support AVX-512, and will add its own extensions that GCC wants to support. H. J. Lu, the lead author on the PDF, works for Intel, and none of the authors list amd.com emails; it seems likely they're more tied into Intel's plans than AMD's plans.

From another angle, https://www.extremetech.com/computing/312673-linus-torval... mentions that Atom CPUs don't even support AVX; what happens when Atom CPUs come out with new instructions specialized for low-power ChromeBooks? Or even some specific instructions, like AES enhancements, make their way all the way down the line to Atom CPUs?

Either way, AMD adding its own instructions without supporting AVX-512 or newer instructions making their way down to cheap CPUs with limited AVX support, it seems entirely out of the control of the GNU toolchain people that there are no new instructions they need to support on CPUs that don't support AVX-512.

GNU C library 2.33 released

joib — Fri, 05 Feb 2021 20:17:43 +0000

LLVM has also added support for these microarchitecture levels: https://github.com/llvm/llvm-project/commit/012dd42e027e . Though I don't know if they have any input into the design, or whether they're just copying what the GNU toolchain is doing.

GNU C library 2.33 released

nix — Fri, 05 Feb 2021 19:14:19 +0000

Yeah, most of these things use runtime checks anyway. The hwcaps scheme that this replaces was meant for things like cmov which had pervasive impact on compiler output, so that rebuilding whole shared libraries for different subarchitectures was worthwhile -- so really the question isn't "what are x86 CPUs capable of" which is indeed a bloody awful mess, but rather "what things that x86 CPUs are capable of does GCC generate noticeably better code for while not being backwardly-compatible to the older models". Since GCC's code generation for different x86 processors is completely under the control of the GNU toolchain people (and, indeed, mostly the exact same GNU toolchain people who specified this feature), I think we can say they probably won't get torpedoed in the future, since they'd literally be torpedoing themselves.

GNU C library 2.33 released

atai — Fri, 05 Feb 2021 07:42:06 +0000

In the lwn weekly summary this item was referred to as "glib release" instead of glibc release

GNU C library 2.33 released

joib — Thu, 04 Feb 2021 21:03:03 +0000

But there is no interaction between the glibc-hwcaps and function multiversioning, no?

So the x86-64-v4 libc.so will still contain __memmove_ssse3 etc. though it will never be used?

GNU C library 2.33 released

joib — Thu, 04 Feb 2021 20:53:24 +0000

> Basically AMD64 is a mismash of supported features, across all chips and even within just Intel chips.

AFAIU the glibc-hwcaps and the microarchitecture levels are meant to provide somewhat decent baselines that apply to many CPU's out there and are expected to be "standard" features in x86 CPU's going forwards. It's up to the psABI developers to not be too aggressive in adding stuff there that might disappear from the market or are of interest only to a small subset of the market (both of these applying to the Xeon Phi's you mentioned).

> I've got "System V Application Binary Interface: AMD64 Architecture Processor Supplement (With LP64 and ILP32 Programming Models) Version 1.0", but I don't see any details about the microarchitecture levels; if that's the right document, the newer versions only come as TeX files.

For some reason the PDF's are excruciatingly hard to find. If you click on the "Wiki" section you get a placeholder page saying that the project has no wiki pages. However, the PDF's are there, the URL is

https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-p...

(The microarchitecture levels are in Table 3.1)

GNU C library 2.33 released

arekm — Thu, 04 Feb 2021 12:54:49 +0000

That way, ah! Thanks.

GNU C library 2.33 released

fweimer — Thu, 04 Feb 2021 12:24:15 +0000

Distributions can build glibc multiple times with different build flags, and install the shared objects into those subdirectories. Some distributions already did this with a POWER9-optimized glibc in the legacy power9 AT_PLATFORM subdirectory.

GNU C library 2.33 released

arekm — Thu, 04 Feb 2021 10:03:33 +0000

Actually I'm asking about optimized variants loaded from "under the glibc-hwcaps directory. I don't see any like this in 2.33.

GNU C library 2.33 released

anton — Thu, 04 Feb 2021 10:00:46 +0000

glibc has alternative implementations for some functions already before (e.g., glibc-2.28 contains names such as __memmove_ssse3, __memmove_avx512_unaligned, __memmove_avx512_no_vzeroupper and more). So this seems to be a mechanism to help select between such versions.

GNU C library 2.33 released

anton — Thu, 04 Feb 2021 09:45:33 +0000

This mess is probably the reason for the new scheme. So yes, glibc may not use all the features of the CPU at hand, but then, does it make a difference? And if it makes a difference, I guess the shared object for, say, x86-64-v4, can still do a CPUID check for the additional feature and then put the appropriate function pointer in the dispatch table, but at least the code can rely on all the v4 features and does not have to worry about whether these are available.

GNU C library 2.33 released

dvdeug — Thu, 04 Feb 2021 06:39:46 +0000

Except that looking at https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#... :
For various AVX-512 features--see Wikipedia for the descriptions--
Knights Landing supports ER.
Knights Mill supports ER, 4FMAPS and VPOPCNTDQ.
Skylake-SP supports VL (and none of the above).
Cannon Lake supports VL and IFMA.
Cascade Lake-SP supports VL and VNNI.
Ice Lake supports VL, IFMA, VBMI2, VPOPCNTDQ and VNNI.

That is, Knights Mill is a superset of Knights Landing; Skylake is a subset of Cannon Lake and Cascade Lake (neither of which are a subset of each other), and Ice Lake is a superset of Cannon Lake and Cascade Lake (but not Knights Landing).

Likewise, FMA4 can never be supported in this scheme, since more recent processors don't support it.

Basically AMD64 is a mismash of supported features, across all chips and even within just Intel chips. Even if you're not concerned about getting optimal performance out of older chips, nobody could have known when FMA4 came out if and how it was going to be supported in the future. Looking at the full AVX-512 table, I don't know what features in Ice Lake are going to be standard in the future; it seems like either they're going to be running far behind the leading edge, or end up defining the top "logical architecture level" as something that it turns out no new processors are supporting, and thus being forced to break the architecture levels.

I've got "System V Application Binary Interface: AMD64 Architecture Processor Supplement (With LP64 and ILP32 Programming Models) Version 1.0", but I don't see any details about the microarchitecture levels; if that's the right document, the newer versions only come as TeX files.

GNU C library 2.33 released

arekm — Tue, 02 Feb 2021 22:32:54 +0000

Does 2.33 already provide some optimized variants or it just adds feature for loading but without any optimized versions?

GNU C library 2.33 released

josh — Tue, 02 Feb 2021 16:52:12 +0000

This is the headline feature:

> The dynamic linker loads optimized implementations of shared objects from subdirectories under the glibc-hwcaps directory on the library search path if the system's capabilities meet the requirements for that subdirectory. Initially supported subdirectories include "power9" and "power10" for the powerpc64le-linux-gnu architecture, "z13", "z14", "z15" for s390x-linux-gnu, and "x86-64-v2", "x86-64-v3", "x86-64-v4" for x86_64-linux-gnu. In the x86_64-linux-gnu case, the subdirectory names correspond to the vendor-independent x86-64 microarchitecture levels defined in the x86-64 psABI supplement.

This is a replacement for the old hardware capabilities mechanism, which caused a combinatorial explosion of checks for each capability. This new mechanism groups capabilities together into logical architecture levels that are increasing supersets, so it can just look for one of them at a time.