An update on BPF generation from GCC

The generation of binary code for the kernel's BPF virtual machine has been limited to the Clang compiler since the beginning; even developers who use GCC to build kernels must use Clang to compile to BPF. Work has been underway for some years on adding a BPF backend to GCC as well; the developers involved ran a session at the 2024 GNU Tools Cauldron to provide an update on that project. It would seem that the BPF backend is close to being ready for production use.

The session was run by David Faust, Cupertino Miranda, and José Marchesi, with Faust doing most of the talking. He started by saying that the core of the BPF backend is now nearly in a production-ready state. It is able to compile all of the kernel test cases — but they do not yet all run correctly. In each case, the generated code is correct, but it is unable to get past the BPF verifier. Most of the self-tests pass at this point, though.

The BPF backend for GCC has been shipped with Oracle Linux for some time, and has more recently found its way into Debian, Fedora, and Gentoo, which is now using GCC to build systemd's BPF programs.

Keeping up with the latest BPF developments is an ongoing task for the GCC developers; there are a number of new instructions that have been added in recent years that must be supported and used to their full potential. These include unconditional byte-swap instructions, jumps with a 32-bit displacement, signed memory-load and register-move operations, and signed division and modulus instructions.

Another significant project in the last year has been addressing one of the biggest divergences with Clang, having to do with the handling of compile-time integer overflow, such as can happen when, for example, assigning a negative number to an unsigned value. Both compiler teams agreed to accept any value that will fit into the intended data type and let the BPF virtual machine deal with it from there.

The GCC backend can now place additional useful information into the generated ELF header, including the version of the BPF virtual machine that was targeted. That information can be used by disassemblers to interpret the binary code correctly.

An important milestone is the completion of support for the compile once, run everywhere ("CO-RE") mechanism, which allows BPF code to run correctly on multiple versions of the kernel. The compiler now produces the necessary (for CO-RE) BPF type format (BTF) data by default when the -g option is provided. The compiler will also generate the somewhat strange, C-like assembly syntax produced by Clang, despite a strong preference on the part of the GCC developers for something that looks more like traditional assembly languages. There is, in any case, a lot of assembly using this format in various header files, so the format must be supported.

There is a new feature macro, __BPF_CPU_VERSION, indicating the version of the virtual machine that a program is being compiled for; the -mcpu= flag can be used to target a specific machine version. There are also options to enable specific instruction types that would otherwise not be available in a given machine version; evidently there are distributors out there that backport implementations of some instructions to older kernels and would like those instructions to be supported.

Segher Boessenkool pointed out that the PowerPC backend used to have a similar ability to enable specific instructions, but that the developers had been forced to remove that capability. As the number of possible instructions grows, the number of combinations that must be tested explodes. Faust agreed that this could be a problem, but said that the testing is manageable for now.

Miranda spoke briefly about the CO-RE implementation, which uses type-based relocation to adjust offsets at program-load time. Imagine a BPF program that accesses specific fields within a kernel structure. If a newer kernel version adds a field to the beginning of the structure, the offset of all the other fields will change. CO-RE makes a note of all these references, calculates the proper offsets when a program is loaded, and adjusts the program accordingly.

This functionality, Miranda said, is implemented using various GCC builtin operations, but the developers have been working to reduce them over time. Instead, relocation information is placed into the GIMPLE intermediate representation of the program. The result is a simpler and more correct implementation, and the kernel's CO-RE self-tests are now passing when built with GCC.

Faust returned to talk more about the generation of BTF data, which is necessary for the loading of programs into the kernel. BTF, he said, shares ancestry with the CTF type format; it is "an ugly domain-specific cousin". GCC can generate BTF, but only for programs written in C. The BTF code has been refactored over the last year, partly with the idea of making it work properly with link-time optimization — a goal that remains unrealized.

Originally, GCC was emitting BTF for every type that was declared in a given program, regardless of whether the type was actually used. Since BPF programs typically have to include a number of kernel headers, the declaration of unused types is a regular occurrence. The result was that even the most trivial of programs would include BTF data for something like 9,000 types, leading to the creation of output files that were vastly larger than the ones Clang produced.

With the new -gprune-btf option, though, GCC can now remove BTF for unused types, eliminating that waste. This option is not the default because in some cases, such as when the kernel itself is being built, the creation of BTF for all types is necessary. Clang does not produce BTF in this way for full-kernel builds; instead, the pahole tool is used to translate the needed information from DWARF debugging info.

A useful future enhancement would be to add a way to encode the flags used by the kernel in type declarations in BTF. These flags include __user, which indicates that a given pointer contains a user-space address. The BPF verifier could make use of this information to further check program correctness. The proposed solution is to encode this information as strings in the DWARF and BTF output. It would be nicer to treat a flag as a type qualifier (like const, for example), but doing so can break applications that consume DWARF data. For the time being, this information can only be attached to the child of a pointer type, so it only works with pointer values. There are some schemes being considered to improve this solution, but they are in an early stage.

Another future task is adding proper BTF support to the binutils package. That would allow, for example, the merging and deduplication of BTF data in the linker, which is needed to support link-time optimization.

An ongoing problem is verifier-aware compilation. Getting code past the BPF verifier is an ongoing frustration for BPF developers. Often, code will be transformed by a compiler's optimization passes into something that the verifier is unable to understand; one version of a compiler might work, while others do not. This topic was discussed at the 2023 GNU Tools Cauldron; Faust said, but little progress has been made since then.

Marchesi took the microphone as time was running out to say that there is now a defined memory model for BPF programs, based on the kernel's memory model. He would like to provide built-in functions that provide proper atomic operations implementing that model in BPF.

He also touched briefly on maintenance issues for the BPF backend. One ongoing task is developing a better assembly syntax for BPF — then getting the rest of the world to adopt it. Meanwhile, there are a lot of BPF users who have found themselves stuck using one specific version of the Clang compiler, since it is the only one that will generate code for their programs that can pass the verifier. GCC users, he said, can be expected to find themselves in a similar situation, but it would be good to find a way to do better. He closed the session with an unanswered question about what sort of maintenance model might help to improve this situation.

[ Thanks to the Linux Foundation, LWN's travel sponsor, for supporting our travel to this event. ]

Index entries for this article
Conference	GNU Tools Cauldron/2024

---

to post comments