An update on continuous testing of BPF kernel patches
Ihor Solodrai has been working on the BPF subsystem's continuous-integration (CI) testing for the last six months. At the 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit, he remotely shared an update on his work, and solicited feedback on how the tests could be further improved. Much of the work he's done has been specific to the BPF subsystem, but some is more generic and could potentially be of use to other subsystems. He also shared some general lessons learned from working on the BPF CI tests.
Before getting to his main update, he first showed a
visualizer he has
been working on for understanding verifier failures. If one uploads a log file
produced by the verifier, the tool shows "something like a debugger view
"
that lets one step through the verifier's process. Solodrai was optimistic that
it could help with debugging, especially for people working on extending the
verifier. He said that the tool is currently a prototype, and he would
appreciate feedback on it.
The main task Solodrai has been working on since the 2024 Linux Plumbers Conference has been expanding the coverage of the BPF CI tests. In the course of that work, he has fixed a number of other things. Still, his main focus has been on adding GCC to the test suite alongside Clang, and adding testing for sched_ext. At the time of the summit, GCC's BPF implementation was not passing the BPF self-tests, and therefore not included in the CI testing. Since then, that appears to have changed, because recent test runs have included testing for GCC 14.
The sched_ext tests are in a similar position; at the time of the summit, Solodrai was running into problems with divergence between the kernel Git repository used by the sched_ext developers and the one used by the BPF developers. Trying to run the sched_ext tests against bpf-next was turning up a lot of problems due to missing commits. Now, it also appears that these have been enabled in recent test runs.
Expanding the number of tests being run has also meant expanding the number of test runners. The BPF CI uses GitHub Actions to run tests on AWS-hosted virtual machines. When Solodrai first started looking at the project, it was using a fixed number of test runners. That meant that a spike in the number of patches to test could cause a backlog of jobs that would take some time to clear. Worse, many of the kernels being tested were cross-compiled to other architectures from x86_64, which left the limited number of x86_64 runners frequently overwhelmed. In order to fix that, Solodrai explained that he had changed the tests to do the build step using a separate service that automatically scales up and down as required. This lets the dedicated test runners focus just on running the tests.
Ideally, the test runners would scale automatically as well. The problem is that testing a kernel build essentially requires an actual virtual machine. User-space software can frequently get away with running tests in a container, but that doesn't suffice for testing the kernel. It should still be possible to make automatic scaling work in theory, Solodrai said, but the existing software that other projects use to manage testing in GitHub actions won't suffice.
Without scaling for the test runners, however, Solodrai hasn't quite accomplished what he wanted when he first started working to improve the usability of the tests. Before, building the kernels was the bottleneck. Now, the actual testing is — but it's still a bottleneck.
As an example, he went into some detail about the test runners for the s390x
architecture. "I have a difficult relationship with them,
" Solodrai said.
They are hard to maintain for a variety of reasons, but primarily because s390x
is not in common use. For example, the control software that GitHub Actions
connects to in order to request that a test start doesn't have an official s390x
build. IBM provides a modified test runner for s390x, but it doesn't quite work
for this use case, Solodrai explained.
So he ended up needing to reverse-engineer the build process of IBM's modified
test runner and apply that to the official GitHub Actions code in order to
produce a build that can be run on his s390x virtual machines. Every time GitHub
Actions releases an update, there's a 30-day window for clients to update
before things start breaking, "and on s390x, it's a roll of the dice.
"
One of Solodrai's questions for the attendees was whether s390x support was really worth it. On the one hand, s390x is one of only a handful of big-endian architectures, which makes testing on it a great way to find endianness bugs in the BPF code. On the other hand, it's not in common use, available CPUs don't have great performance compared to CPUs for other architectures, and it's taking a lot of his time that could otherwise be directed to improving the test suite in other ways.
The participants didn't have a solid answer for him — and the s390x tests identified an endianness problem less than a week after the summit. So for the moment, the tests are still running and proving useful. In case anyone else wanted to step up and help maintain or expand the BPF tests, Solodrai went through a number of hard-earned lessons on how to do that.
His main advice was to use as little YAML as possible. "You really don't want
to use YAML. You have to use YAML, but you'd much rather write a bash
script.
" He also recommended setting things up to use sparse checkouts,
environment variables rather than GitHub Actions inputs, and being cautious
about using the built-in caching support since it has confusing rules. Reusable
workflows are also only somewhat useful; they can help deduplicate code, but the
security rules are tricky here too.
Solodrai warned people to remember that there are many ways to run code on
different architectures, but most of them are actually
QEMU under the hood.
At one point, he had "an adventure
" with three seemingly unrelated
problems all cropping up at once; it turned out that he needed to manually
install a newer QEMU that had fixed the bug.
Finally, Solodrai made sure that all of the attendees knew how to make sure the CI tests ran for their patch set: make a fork of the kernel-patches/bpf repository on GitHub, put the patch set on a branch, and create a PR. The automation will see the PR and kick off the tests.
At that point, the session opened up into a group discussion of the current coverage of the tests and what, if anything, they should be extended to cover. Solodrai suggested testing multiple compiler releases, long-term-support kernels, and 32-bit architectures. There was a lot of general enthusiasm about testing long-term support-kernels, and an acknowledgment that the other things would be useful as well if possible. Alexei Starovoitov specifically wanted to test LLVM 19, which has support for BPF arenas, to obtain better coverage in that area.
Solodrai was also interested to know which features of the BPF CI tests people were using. For example, he asked whether putting more effort into speeding up the tests would make a real difference for anyone's work. That question prompted extensive discussion, but the ultimate conclusion was that it would not — the tests are always going to be long enough to interrupt someone's work, at which point the difference between 45 minutes and two hours is not massively important.
Since the summit, Solodrai seems to have made good progress on putting the things discussed there into practice. The kernel's testing infrastructure is somewhat patchwork, with many subsystems and independent testing efforts each working on their own areas of interest. The BPF CI tests, however, seem to be increasingly useful and reliable.
| Index entries for this article | |
|---|---|
| Conference | Storage, Filesystem, Memory-Management and BPF Summit/2025 |
