Shielding running kernels against exploits with BPF

Cisco has some unusual challenges when it comes to deploying security patches across the company's many devices running custom kernels. John Fastabend spoke about his work preventing exploits with BPF at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. The technique could substantially reduce the time necessary to respond to kernel vulnerabilities, but it will not be fully effective unless more hooks are added to the kernel.

Network switches encompass a big range of hardware, Fastabend began. From small single-rack systems all of the way through huge high-speed devices. Each of Cisco's supported platforms has its own kernel team that builds custom kernels using Yocto. At any given time, those teams are supporting a large number of different kernels — mostly stable kernels, fortunately, he added. All of these widely deployed, internet-connected devices with custom kernels make tempting targets for attackers.

Cisco publishes security updates, obviously, but it takes time to identify a problem, write a fix, create a new build, test it, provide it to customers, and then let switches update, especially because rebooting switches can cause network disruptions. Those disruptions require clients to plan and manage downtime, and the whole process can take months from discovery to patching the last vulnerable systems. The goal of Fastabend's work has been to use BPF to observe attacks in real time, and then allow them to be addressed on demand, without rebooting. Ideally, the whole process would take only minutes, he said. "We won't be there for a while, but that would be the dream."

Tetragon, the open-source BPF-based monitoring and enforcement tool, is used to collect "lots of data" about running systems. At any time, the monitoring infrastructure on a switch can show which programs ran at which times, and which network connections they made. That data is stored in a time-series database. Tetragon does currently depend on a user-space agent to remain operational, but Fastabend and his colleagues have been working toward making the BPF components survive even if an attacker manages to kill the user-space agent. When a new CVE is discovered, he explained, he wants to be able to check against that database to find out whether it was ever exploited. The data can also be used to see the symptoms of an attack, such as data exfiltration or connections to command-and-control servers.

Once an exploit is identified, it can be blocked directly from BPF. If a particular system call is necessary to trigger the exploit, BPF can override the return value of the system call to refuse the operation. It is also possible to use tracepoints to verify that the arguments to internal kernel functions are correct. Fastabend's team uses both uprobes and kprobes for that. Those probes don't reliably allow changing return values, however, so Linux security-module (LSM) hooks are used for that.

Andrii Nakryiko asked how many events per second were being checked and potentially intercepted by this design. The routing of network packets is mostly done by dedicated hardware, Fastabend explained, so the kernel only needs to manage control-plane traffic and user-space applications. Overall, there are only hundreds or thousands of events per second, not billions, even if the switch is moving billions of packets.

One complication is that Fastabend's team wants to use probes to operate on inlined functions as well. That is possible by using debugging symbols and setting a probe at a raw offset. Cisco has a build farm that is used to produce all of its kernel packages, he explained. The build machines save the build IDs and debugging information from all of the builds, including both BTF and normal debugging symbols. That information is used to debug customer problems, but also to make it possible to write live kernel patches or BPF programs that are specific to the structure of deployed kernels.

Jakub Sitnicki noted that he has experienced a problem attaching probes to functions that are partially inlined, since information on where those functions have been inlined is not presently included in the kernel's BTF, but the problem is being worked on. Alan Maguire said that the topic would be covered in one of the sessions he had proposed for the next day of the conference.

Blocking exploits

Fastabend then showed an example of a BPF program that could be used to block the effects of the recent copy fail vulnerability. The program just made the splice() system call return an error when called in a way that would trigger the bug. His team calls such BPF programs "shields". Something like this would technically be possible to do with a normal kernel live patch, he admitted, but Cisco has so many concurrent product lines, with many different stable kernels running on them, that it would require a huge investment of developer time to patch them all. With BPF, the same program can typically run across all of the supported kernels — it just needs to be written once, and then automatically tested on each kernel to make sure it doesn't break anything.

BPF shields are great when they work, but there are some occasional hiccups. Often, Fastabend's team will find a CVE that doesn't have any relevant hooks in the affected kernel subsystem to build a shield around, he said. Most recently, there was an exploitable use-after-free bug that simply didn't have anywhere convenient to hook close to the source of the problem. The team eventually settled on hooking the system call that could lead to triggering the bug, but it made for more complex code.

Therefore, the main change that Fastabend would like to see in the kernel to support efforts like this is a more inclusive policy for ALLOW_ERROR_INJECTION(), the macro that is used to mark functions that can be subjected to the kernel's error-injection framework. While normally used for testing, the framework allows kernel programmers to override the return values of internal functions with custom BPF programs, which is a neat match for Fastabend's use case. Unfortunately, only a subset of kernel functions have been marked for use with error-injection. Ideally, he would like to be able to use BPF to modify the return value of any function that returns an integer which is compared to zero and has that error propagated up the stack. There are plenty of functions that match this criterion, he said. There should be LSM hooks for all of these, he said. Fastabend asked whether those functions could easily be made hookable from BPF.

Nakryiko thought that the complexity of error-handling code would make any change like that a manual process. Alexei Starovoitov suggested that it should be possible to do automatically for Rust code, given that the compiler-generated cleanup logic in Rust code has a predictable structure. For C code, he suggested asking the people submitting vulnerability reports to introduce a relevant LSM hook as well. I suggested using Coccinelle to make the change.

If it did end up being a manual process, the most important place to target would be the kernel's netlink code, Fastabend thought. For whatever reason, his team sees a lot of attacks targeting that area of the kernel.

The technique of hooking internal kernel functions as a protection against vulnerabilities is certainly useful to other users of the Linux kernel as well. The shield he showed was only a handful of lines of code, and it is easy to see how, with sufficient coverage of the kernel, that kind of simple fix would provide a way for companies or distributions to deploy vulnerability mitigations across multiple kernel versions with much less hassle. That said, the work of adding LSM hooks to the relevant areas will be a large change, and may take some time — if the LSM maintainers approve of the work at all.