|
|
Log in / Subscribe / Register

BPF in the agentic era

By Daroc Alden
June 3, 2026

LSFMM+BPF

Alexei Starovoitov gave "less of a presentation, more of a scream of realization" at the BPF track of the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. He shared a set of ideas for how BPF could change to avoid being swept away by the sea-change in programming represented by modern large language models (LLMs) and the coding agents based on them. In a follow-up session, the discussion covered more problems with how coding agents use tools like bpftrace, and the current deluge of patches in need of review in the BPF subsystem.

He wanted to be clear that these ideas were not something he was unilaterally deciding on, they were "just looking at how the world has to be". Coding agents do their best work when given tight feedback loops, he said. They write code, see errors, and make fixes. This makes them good at handling tedious jobs such as large refactoring operations. That feedback loop doesn't work well in the BPF ecosystem, especially when one has to boot a virtual machine in order to properly test a change.

[Alexei Starovoitov]

Even when a BPF programmer has attempted to load their program into the kernel, errors from the verifier "are insane" — huge dumps of data that obscure the actual source of the problem. In Starovoitov's experience, writing BPF code is the only case he has observed where LLMs will give up rather than producing something.

BPF can never crash the kernel. "That's what BPF stands for." That is the one thing that makes the ecosystem unique. So, the BPF verifier isn't going away, and it needs to run in the kernel because user space is untrustworthy. This adds a certain amount of unavoidable latency and complexity between writing a BPF program and getting feedback on it. Given that constraint, how can BPF developers shorten the feedback loops between writing a BPF program and getting a useful error message?

Right now, the verifier essentially has two roles: finding places where the programmer made a mistake, and acting as a security boundary. The first part doesn't need to be done in the kernel; languages such as Rust do a great job of confronting programmers with their mistakes in user space. The BPF developers should be working to make BPF easy to use with Rust, so that simple mistakes can be caught quickly, Starovoitov said. Any program that does pass the Rust compiler — and doesn't do anything obnoxious with unsafe code or inline assembly — should pass the verifier as well. Obviously programmers will still be able to cause verifier errors if they try, but well-intentioned, normal BPF programs written in Rust should not cause verifier errors, he said.

Making this happen may require further work on the verifier, or the careful addition of run-time checks to BPF. But it's a goal that Starovoitov believes is possible and worthwhile. There are other ways to shorten the feedback loop for BPF programmers, of course. If BPF could be made to work in user-mode Linux, for example, there would be no more need to test BPF code in virtual machines.

This confused one member of the audience, who objected that Starovoitov had previously soundly rejected the idea of moving the BPF verifier to user space. The real kernel can't trust user space, Starovoitov said, but it should be possible to run the verifier in user-mode Linux in order to shorten the time it takes to receive a verifier error if one is forthcoming.

All of these changes are coming quickly. In a few months, "there will be no fighting the verifier anymore," Starovoitov promised, as long as one writes BPF programs in Rust. The verifier will use run-time checks where necessary to make everything work. José Marchesi asked whether that change was currently in development. Starovoitov confirmed that it was.

Daniel Borkmann asked how the output from the verifier would change. Current verifier errors are far too opaque, Starovoitov said. They say things like "register is not init" — what is a programmer supposed to do with that? The verifier needs error messages that say what happened and how to fix it. This is another area where Starovoitov wants to take inspiration from Rust, and, in particular, the formatting of its error messages. "You guys are all dinosaurs; Rust won as a language," he said, at least in part because of the quality of its error messages.

In the end, he wants there to be almost no verifier errors. The ones that remain should include details about what went wrong and how to fix it. This change isn't going to happen overnight, he said, but "it is where we have to go".

There was some discussion about even bolder proposals, such as integrating WebAssembly into the kernel. Starovoitov was supportive of the idea if it could be made to work, but didn't think that was likely. When I spoke to him after the session, he said that he thought previous attempts to bring WebAssembly to the kernel were too slow for serious use, but that if someone was willing to do the work to bring it up to par with the existing BPF JIT implementation, they should do so.

At that point, Starovoitov dove into the details of how his suggested transformation could be achieved. There are a number of limitations of the verifier that must be removed, he said. These include the inability to handle functions with more than six arguments, indirect function calls, returning structures by value, support for 128-bit integers, larger BPF program stacks, deeper call chains, removing the one-million-instruction limit, and long jumps in BPF programs. Also, the BPF developers will need to remove the discrepancies between the verification of static and global functions. "Of course, this helps humans too."

Some of these things, such as larger program stacks, have already had the groundwork laid by previous improvements to BPF. For the rest, Starovoitov hopes to move more data into BPF arenas, which have one flat address space. Once that is done, a "global" memory allocator (really one per arena) makes many of the other problems significantly simpler. That plan does pose problems for Rust code that uses virtual dispatch tables; currently, the Rust compiler emits these tables intermixed with the code, which BPF is not set up to handle. Starovoitov suggested that changing the compiler to emit the tables differently might be the simplest solution.

In fact, changing existing tools may be the only way to remain "in distribution" (within the area that LLMs have been trained on), he said. LLMs are not as good at handling tooling that isn't in their training sets; ensuring that BPF objects can be built and linked with existing tools is important to letting coding agents take advantage of the improvements.

Amery Hung asked whether that extended to introducing new concepts, such as new kinds of BPF maps. Starovoitov thought that it did. When it was added, he had thought that BPF arenas would be the last map type needed. Since then the BPF subsystem has also added rhashtable-based maps, but he doesn't envision the need for any more.

The biggest change that will be required in the verifier is improved handling of loops, Starovoitov continued. There have been lots of struggles with BPF loops over the years; their handling has improved, but some loops are still too much for the verifier. Currently, the verifier will go through a loop multiple times until it proves one of three things: that the loop will exit after a number of iterations, that the loop returns to (a subset of) a previously analyzed state, or that the program will hit the maximum number of verified instructions and trigger an error. Starovoitov's proposed solution, called widening, would make it so that when verifying a loop takes more than a small number of iterations, the verifier state is generalized to consider larger numbers of loop-variable values at once ("widening" the state). The downside is that widened states are less precise, and potentially require more explicit run-time checks. This change could let the verifier handle all loops with a fixed maximum number of iterations, at the cost of requiring run-time checks for the loops that require widening.

The early results of this approach are "really great", Starovoitov said, but there is a lot more work to do to make it usable. Still, he hopes to have something together in relatively short order. At that point, the scheduled session was coming to a close, but he had enough still to cover that the talk was extended to a second session the next day.

Tracing and code review

The bpftrace and DTrace tools use domain-specific languages for configuring tracepoints; this makes them good for humans, but bad for LLMs, Starovoitov said. LLMs struggle with the syntax of niche languages. He thinks that the future of tracing looks like having documentation for coding agents that explains how to use standard tools, with explicit examples. That could help bpftrace and DTrace remain usable, but would be even better for tools like drgn that use a standard language's syntax, just with extra capabilities. Ultimately, he wants to be able to tell a coding agent to "find a performance issue on this kernel" and have it use BPF and drgn to identify and debug the problem.

One audience member thought that more examples in the documentation would be a good thing, but didn't think that using the kernel's tracing infrastructure with coding agents was viable unless one does not care about crashing the machine on which they are running. Starovoitov thought that this was ultimately solvable by encoding information about the kernel's functions and types in a format that can be checked automatically by existing tools, such as providing a "vmlinux.rs" set of Rust bindings.

Starovoitov also wanted to discuss code reviews. The BPF subsystem is swamped with low-quality patches that look, on first impression, really good. At this point, he is ignoring the commit log on the assumption that it is written by LLMs and just looking directly at the patches themselves. But, in general, "maintainers don't scale". Starovoitov called on everyone who submitted patches to the BPF subsystem to also perform code reviews.

There is no official number or scoring system, but contributing to BPF does require a fair and honest effort to review code. Cupertino Miranda asked whether he could invite people to come review code in GCC as well, since he is contributing to the kernel as part of his GCC BPF work. Starovoitov said that if he had a patch for GCC, he would certainly do some code review there as well, but "I last touched GCC 20 years ago". He did suggest that perhaps people could consider using LLMs to review their patches, in GCC and elsewhere.

Steven Rostedt complained that not everyone has access to an infinite number of tokens, and that a dependency on cloud-hosted LLMs isn't sustainable. Starovoitov agreed that it was a dependency that needed to be managed, but didn't think that could be avoided: "We are done with coding manually. It will all be agents in the future." He then called it more of a way of life than a dependency.

Rostedt objected that LLM companies don't have a proven business model, and so they're giving things away for free that they will not be able to afford to continue providing to kernel developers in the long term. Starovoitov did not think that LLMs were going to go away, even if prices rose in the future. Another audience member proposed that these companies should put together a fund for open-source developers, and let programmers pick which model to use, so that they would not be locked into one model that could disappear. Someone else pointed out that the Cloud Native Computing Foundation already does give out licenses for things like this to the maintainers of their projects. Starovoitov pointed out that open-source models also exist, and predicted that "coding will be possible on a laptop, once the technology advances". Rostedt agreed that things will become cheaper, but that didn't entirely ameliorate his concern.

Borkmann said that the assembled BPF developers need to think about new people coming to the community. They should be able to review code, eventually, but there is a learning curve. How can they learn to do that when simple code review is done by LLMs? Starovoitov claimed that "all of the newcomers, they're all skilled in agentic coding". All of the patches that they are sending are written by coding agents, he said. It lowers the bar to kernel contribution.

Rostedt didn't think this was a good thing — if ten times as many people are contributing, but the same quantity are sending good code, "that's not a plus". Starovoitov said that this is why more code review is needed. Sashiko has been helpful for that; he doesn't review patches until they've had an LLM comment.

At that point, discussion broke up into a series of increasingly outré proposals for how to improve the review situation, such as asking LLMs to regenerate commit logs based on what is in the patches. Starovoitov gave one final announcement that BPF office hours (regular video meetings where one can discuss problems with the BPF maintainers) are now ad hoc, not regularly scheduled. People interested in attending one should send an email to the mailing list to find a time that works.


Index entries for this article
KernelBPF
ConferenceStorage, Filesystem, Memory-Management and BPF Summit/2026


to post comments

Reads like ragebait

Posted Jun 3, 2026 17:03 UTC (Wed) by hDF (subscriber, #121224) [Link] (3 responses)

>"You guys are all dinosaurs; Rust won as a language," he said, at least in part because of the quality of its error messages.
>"We are done with coding manually. It will all be agents in the future."

pretty obnoxious, but maybe something got lost in translation

Reads like ragebait

Posted Jun 3, 2026 17:12 UTC (Wed) by ballombe (subscriber, #9523) [Link] (1 responses)

It is not a problem to be a dinosaur as long as you are a t-rex!

Reads like ragebait

Posted Jun 4, 2026 12:51 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Or rather as long as you're a bird

Reads like ragebait

Posted Jun 3, 2026 17:19 UTC (Wed) by daroc (editor, #160859) [Link]

I have done my best to accurately represent what Alexei said, including actual quotes where I was quick enough to get them down verbatim in my notes. But there is always something lost in going from speech, which has tone-of-voice and body language to clarify meaning, to text. He did make the dinosaurs comment with a smile, and people seemed to take it well.

note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default

Posted Jun 3, 2026 18:20 UTC (Wed) by adobriyan (guest, #30858) [Link] (2 responses)

> Rust won as a language," he said, at least in part because of the quality of its error messages.

Rust doesn't have error messages, Rust has error wallsoftextssages.

I guess some Rust devs don't see the problem because use editor overlays which show only the first line.

And then there are obvious silly things like

#[derive(Debug)]
pub struct S {
field1: ...,
}

warns about "unused" field even if the field is used for a fact by "{:?}" somewhere outside of the module defining.

For some reason it always uses more than 1 terminal screen.

note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default

Posted Jun 3, 2026 18:51 UTC (Wed) by bertschingert (subscriber, #160729) [Link]

> Rust doesn't have error messages, Rust has error wallsoftextssages.

I agree that the overly long error messages can be pretty annoying - especially when in the middle of doing a refactor where I know the cause of the errors and just want to see all the places I need to fix in one screen. Luckily `cargo build --message-format short` helps for that sort of situation.

note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default

Posted Jun 4, 2026 10:51 UTC (Thu) by ojeda (subscriber, #143370) [Link]

The compiler clarifies:

note: `S` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis

...which seems fine to me.

And if you really have a reason for such code, then you can always tell the compiler you really mean it by locally (in the field itself!) tagging it as such.

Temporary problem

Posted Jun 4, 2026 7:30 UTC (Thu) by kleptog (subscriber, #1183) [Link] (11 responses)

> Rostedt objected that LLM companies don't have a proven business model, and so they're giving things away for free that they will not be able to afford to continue providing to kernel developers in the long term.

While true, I think this is a temporary problem. We know that human level intelligence is possible on <100W of power, we just haven't figured out how to do that in silicon yet. I also think a lot of companies have spent a lot of money on GPUs that are ultimately never going to be used.

Temporary problem

Posted Jun 4, 2026 9:28 UTC (Thu) by paulj (subscriber, #341) [Link] (10 responses)

Biological neural nets grow actual connections as part of "training", and remove unused connections.

Silicon uses logic to apply mathematical operations to long rows of numbers, effectively with 'unused connections' being emulated by multiplying elements by 0 to null out their effect on the next set of operations. Those numbers having to be copied in and out of various kinds of RAM. The physical connections are always there, and they are being used and consuming power transferring bits, in order to compute the "not used" result on them. Where the biological system truly consumes nothing for a non-existent connection.

Etched silicon systems have no known way to /physically/ 'grow' and 'prune' connections, outside of manufacture.

Further, with (to a limited degree) the exception of things like Cerberus wafer-scale 'chips' (which I think is on the inference side - not training), we need to use networking infrastructure to connect together the logic and the needed RAM. That consumes even more power again. Networking is approaching half of the power budget of these rack-scale AI systems. Even Cerberus is at the limit on on-wafer SRAM, and will end up with networking for multi-wafer systems or off-wafer-RAM for larger LLMs, just on inference, from what I understand from what analysts have written.

I don't know of anything anywhere near the horizon that addresses the massive power inefficiency of current silicon "AI" hardware architectures. Be glad to pointed at something I've missed.

Temporary problem

Posted Jun 4, 2026 10:39 UTC (Thu) by malmedal (subscriber, #56172) [Link] (9 responses)

In principle we know how to reduce the energy used per computation. I believe we can relatively easily increase efficiency by a factor of at least ten probably even a hundred.

Problem is that most ways of increasing efficiency also decreases max clock speed. The brain runs at around 20 to 200Hz, we should be able to eventually reach similar efficiencies if we ran our silicon at that speed, obviously this means we'd need at lot of silicon.

In the near term we are very constrained on chip supply so I expect the focus is on better cooling so we can run the chips hotter and even less efficient.

Temporary problem

Posted Jun 4, 2026 11:55 UTC (Thu) by paulj (subscriber, #341) [Link] (1 responses)

Which ways are you thinking of?

If we look at the underlying architecture of biological neural nets, and work on the assumption there is something to learn there on energy efficient NNs, then that suggests we need to move to asynchronous architectures of very large numbers of simple units that combine both communication with 'nearby' units and integrating those signals (memory and compute). (Where current architecture is using large numbers of simple computational units, with hierarchies of /separate/ memory, and then emulating the communication side with compute over that separate memory - which can take multiple steps).

So SNNs possibly, running on some kind of much, much, much more massive SpiNNaker-type tiled system. But, SpiNNaker hasn't yet provided results that have been... revolutionary (let's say). ?

Or you're thinking of something else?

Temporary problem

Posted Jun 4, 2026 16:28 UTC (Thu) by malmedal (subscriber, #56172) [Link]

That yes, but also things like process changes.

Since the availability of chips currently is the bottleneck, people run them as hard as possible, if the production increases to the point where the limit is the available power there are many things that can be done to get more compute per joule of energy. Currently it appears to be mostly academic research in older processes, e.g:

https://www.researchgate.net/publication/358974979_An_Ult...

but I believe if it becomes commercially interesting then industry can quickly get efficiency gains of 10 to 100 times. Same goes for network interconnect.

There are also pie in the sky things like reversible computing, https://www.sciencenews.org/article/computer-chip-reuses-...
Theoretically feasible, wouldn't hold my breath while waiting though.

Temporary problem

Posted Jun 4, 2026 13:36 UTC (Thu) by dskoll (subscriber, #1630) [Link] (6 responses)

A big difference between the brain and silicon chips is that the brain can make connections in three dimensions, while silicon chips are effectively 2D. Even with stacked chips, the connectivity in the third dimension is much less than in the other two. So the brain can have a much more dense and complex set of interconnections than a silicon chip and I think it's one reason it can operate so well with less energy and a lower "clock speed".

Temporary problem

Posted Jun 4, 2026 13:47 UTC (Thu) by daroc (editor, #160859) [Link] (5 responses)

This is a reason that chip fabs are working on ways to make chips with 3D stacked layers of transistors. They haven't cracked it yet, but it's one of those research areas that someone is always poking at because a discovery would be revolutionary.

Temporary problem

Posted Jun 4, 2026 17:03 UTC (Thu) by malmedal (subscriber, #56172) [Link]

I believe the issue is mostly heat, and a little bit of interconnect. For instance 3D Flash is currently around 300 layers, but to try that in normal logic would make the inner layers melt.

Hmm, or it might posibly work if you ran such a chip on 200Hz like the human brain does.

Temporary problem

Posted Jun 5, 2026 9:32 UTC (Fri) by paulj (subscriber, #341) [Link] (3 responses)

Stacked dies are already common practice for high-speed DRAM on GPUs and high-speed cache on some CPUs. A.k.a. HBM.

This still doesn't solve the inefficiency of having compute logic in one place, and memory in another place, and having to constantly transfer information back and forth between the two to a) carry out the /emulation/ of the network b) so as to enable the computation of this emulated network. In the brain, compute and storage are part of the same unit and the network is intrinsic to the physical structure of the 'computer'. In silicon NNs, we are spending power to /emulate/ a network, on top of a physical structure that doesn't resemble the desired network in any meaningful way.

Temporary problem

Posted Jun 5, 2026 11:32 UTC (Fri) by malmedal (subscriber, #56172) [Link] (2 responses)

Several companies for instance Cerebras.ai does what you suggest.

It is clear that this is faster, it is not clear if it is more power-efficient or not.

Simplifying a lot but basically to produce a new token you have a big matrix and multiply it with a 1d vector. If you make the vector a two wide matrix you can produce a token each for two unrelated sessions for just a tiny bit extra work. You can keep widening the matrix quite a bit before it becomes too slow.

Several providers give you a fast mode e.g. 3 times faster for three times the price which I assume means they give you fewer neighbors in the matrix multiply.

Eventually I expect custom chips will replace the current GPUs, but right now the basic architecture is still in flux, previously RNNs were the hot stuff for AI, currently it's mostly transformers but there is promising research into replacing those with diffusers. So for the time being we want something flexible.

Temporary problem

Posted Jun 5, 2026 12:52 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Yes, I mentioned Cerberas in one of my comments antecedent to these. Cerebrus puts SRAM on the tile, but this is still separate from the compute logic. Obviously so from the die shots. Have a look at DOI:10.1109/MM.2023.3256384. It's going a bit in that direction, but it's still emulating the NN network architecture with a traditional Von Neumann machine, with separate memory and compute and a load-compute-store architecture (however parallel, and emulating the network by modelling it as a matrix).

Temporary problem

Posted Jun 5, 2026 14:30 UTC (Fri) by malmedal (subscriber, #56172) [Link]

Yeah. I'm sure full custom will the end-point, but currently we don't know what the optimal architecture is. For instance the brain does have a number of long-range connections, which appears to be important, but while these can be grown dynamically in the human brain we can't yet do that in silicon, so for the time being I believe it's necessary to have something where the connections can be reprogrammed at will.

skilled in agentic coding

Posted Jun 7, 2026 22:20 UTC (Sun) by aimannajjar (subscriber, #184277) [Link] (2 responses)

> "all of the newcomers, they're all skilled in agentic coding"

Agentic coding is not a skill, it's nothing but writing a request in English, a de-skilling activity is more accurate way to put it. With that mindset, there will be no one to maintain open source code in the next decade. Keep the slop out of open-source please.

skilled in agentic coding

Posted Jun 8, 2026 7:48 UTC (Mon) by kleptog (subscriber, #1183) [Link] (1 responses)

> Agentic coding is not a skill, it's nothing but writing a request in English, a de-skilling activity is more accurate way to put it.

Writing clear unambiguous instructions in English (or any natural language) *is* a skill, as anyone who has tried to direct a junior developer can attest. Or really giving instructions in any area where people are involved. Natural languages are notoriously ambiguous and context is everything.

We created programming languages *because* natural languages are imprecise. LLMs understand natural languages better, but they won't replace programming languages.

skilled in agentic coding

Posted Jun 8, 2026 13:04 UTC (Mon) by aimannajjar (subscriber, #184277) [Link]

> Writing clear unambiguous instructions in English (or any natural language) *is* a skill

That reminds of something. Did you know that there is a 70k-star "Claude Skill" project that allows you to communicate with Claude like "cavemen" with the aim of minimizing token usage: https://github.com/juliusbrussee/caveman ? Yes, it seems that trending skill today is how to minimize tokens by downgrading your English to caveman-level.

All that to say, there is no proof that prompt style has real effect on the quality of the output. This is why "prompt engineering" has quickly become an obsolete term, along with RAG and many other hyped terminology (I'm sure "agentic coding skills" will follow suit). The fact is, no matter how precise you are, you will have to repeatedly engage the LLM to get to the desired output, and in the process you will accumulate so much technical debt, as well as cognitive debt.

> We created programming languages *because* natural languages are imprecise

That is the crux of the problem, LLMs are inherently imprecise, due to them being stochastic and non-deterministic.

We are really doing a disservice to young, aspiring and smart engineers by giving them the illusion that they're investing their time in a valuable skill such as writing good prompts, when they should be their hands dirty on real coding exercises.


Copyright © 2026, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds