Leading items

Welcome to the LWN.net Weekly Edition for July 16, 2026

This edition contains the following feature content:

• An update on the scraper situation: the battle between bots and publishers continues.
• Lockless MPSC FIFO queues for io_uring: a look at performance gains coming to io_uring in 7.2.
• Continued coverage from the 2026 Linux Storage, Filesystem, Memory Management, and BPF Summit:
• Topics in filesystem testing: a discussion on testing filesystems, especially on patches aimed at the stable kernels.
• Shielding running kernels against exploits with BPF: John Fastabend discusses a strategy for quickly immunizing vulnerable kernels against new exploits.
• Sending packets directly from BPF: a facility to allow BPF to make outgoing network requests.
• Kitty chases the mouse: a look at the Kitty GPU-based terminal emulator.
• QBE 1.3: metaprogramming, performance, and cross-platform support: a lightweight alternative to compiler backends such as LLVM and GCC.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

An update on the scraper situation

Our article "Fighting the AI scraper bot scourge", published in early 2025, discussed the problem of widespread scraping of web sites in search of training data for large language models and related projects. This activity overwhelms sites with traffic. Over a year after that article is published, the problem is still growing. The hammering of sites by shadowy actors has reached new heights, and the open web is becoming increasingly difficult to maintain. Where is this traffic coming from, and what can be done about it?

Residential proxies

As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term "residential proxies" is used to describe systems that are used in this way.

There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.

More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.

The second sort of operator works more overtly, pretending to a degree of legitimacy and offering "ethically sourced" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a "free" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.

There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.

While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.

Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt. They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.

What isn't clear is who is using the residential proxies; somebody is paying them to run these attacks on web sites. There is no evidence (that I am aware of) that the frontier-model companies are using those networks. If it were to turn out that they are doing so, though, the increase in global astonishment would barely register. Those companies are feeding their models somehow, they are not forthcoming about how they get their training data, and they have not distinguished themselves with their level of respect toward content creators — or toward anybody who might have concerns about their operations.

For every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organizations (to the extent that they are distinct from governments) probably also want to have their own models. These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.

Defending the open Internet

In response to all of this, web-site operators have been scrambling to defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by requiring a proof of work, is now widespread. Other sites use commercial services, which sometimes make themselves known with a "prove you are human" button. Or sites force users to pick out squares containing streetlights (but only those with LED bulbs), place puzzle pieces, or hum a song while holding down the space bar. Many site features have been placed behind login gates or paywalls. Some sites attempt to actively poison the data sent to scrapers with tools like iocaine.

Both the need to set up and maintain these mechanisms, and the requirement that users cope with them to access a web site, constitute a heavy tax placed on the world as a whole by scrapers and those who pay them.

Recently, LWN was subjected to what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.

What we can say is that we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.

There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.

We have aggressively optimized parts of the site, and found ways to minimize expensive operations during times when the site is under attack. Anonymous readers may occasionally encounter one of those measures; logged-in users will not. Amusingly, the response time when the site is under attack is often better than during the calm times, when the defensive measures are dormant. We have learned better than to think that the problem is solved, though; consideration must be given to our next steps once the current measures are no longer effective.

On July 2, Google announced that it had, in coordination with the US Federal Bureau of Investigation and others, taken down a residential-proxy network called "NetNut". For the time being, that action would, indeed, seem to have succeeded in reducing the level of scraper attacks somewhat. Experience shows, though, that this welcome peace will only last so long. Google takes pains to point out that its Play Store will now check for NetNut-infected apps, but all of the major vendors are silent on the topic of why it is so easy to put apps with residential-proxy functionality into their app stores.

It would be good to find a more lasting solution before the entire Internet is driven behind defensive walls, and the open network that inspired so much creativity is lost. The industry that is driving these attacks seems entirely at ease with turning independent web sites into smoking craters after having pillaged their contents — an attitude that extends to the planet and its economies as well. Some of us, though, object to that idea and will fight against it. Someday, with luck, the world as a whole will decide to hold the companies behind large language models and related technologies to a minimal ethical standard. Until then, though, this behavior will continue, and we will have no choice but to defend ourselves against it.

Comments (155 posted)

Lockless MPSC FIFO queues for io_uring

Processes that use io_uring tend to keep a lot of balls in the air; being able to have many operations underway at any given time is part of the point of that API in the first place. The io_uring subsystem must, as a result, keep track of a lot of tasks that have to be performed at the right time. In current kernels, io_uring uses a standard kernel linked-list primitive to track those work items. As of the 7.2 kernel release, though, io_uring will, instead, use a new lockless, multi-producer, single-consumer (MPSC) queue, resulting in some notable performance gains. Lockless algorithms tend to be tricky, but the one used here is relatively approachable and shows how these algorithms can work.

The old way's shortcomings

Task queues in pre-7.2 io_uring are based on the kernel's lockless singly linked list (llist) API. At the core of this type is a simple structure:

    struct llist_node {
	struct llist_node *next;
    };

This structure, when embedded within another structure containing the actual data of interest, contains the links that bind those outer structures into a list.

There are a few reasons why this list type, despite being designed for performance, is not ideal for io_uring. Since an llist is a singly linked list, it can only realistically be accessed from the head. For a list where producers add items and consumers remove them, an llist is essentially a stack. Work items in io_uring need to be processed in the order they were received, for basic fairness purposes if nothing else, so a pass must be made over the task queue to reverse its order before each processing run. To make things worse, io_uring might choose not to process the whole list if it is too long, but the remaining items, having been reversed in order, cannot simply be put back onto the task list. So a separate list for reversed-but-not-processed items must be maintained. Finally, adding items to an llist involves accessing a single head pointer; that can be done without taking locks, but it does require a retry loop. For heavily contended lists, those retries (and associated cache-line bouncing) can hurt.

Solving these problems requires a data structure that is better suited to the needs of the io_uring subsystem. It must handle multiple producers putting work items on the lists, without locking and with a minimum amount of contention. There is a single consumer for each list; that consumer should avoid cache contention with the producers to the greatest extent possible. And, clearly, the need to reorder lists before processing should not exist.

Lockless MPSC queues

The solution is the lockless MPSC queue, posted by Jens Axboe and using an algorithm credited to Dmitry Vyukov. This queue still uses struct llist_node to tie the entries of the list together, but the head of the list looks like this:

    struct mpscq {
	struct llist_node	*tail;
	struct llist_node	stub;
    };

The term "head" is actually a bit misleading, since there is no pointer to the head of the list here; we'll get to that later. This view of the list is intended for the use of the producers, who need to add items to the tail of the list. The stub entry is a sentinel that is only present on the list if it is the only entry there — if the list is empty, in other words. When the list is initialized, the tail pointer is set to point at the stub entry.

The stub entry's next pointer is set to NULL. The addition of a node to the tail of the list is done with a call to this short function:

    static inline bool mpscq_push(struct mpscq *q, struct llist_node *node)
    {
	struct llist_node *prev;

	node->next = NULL;
	prev = xchg(&q->tail, node);
	WRITE_ONCE(prev->next, node);
	return prev == &q->stub;
    }

The next pointer in the new entry is set to NULL, indicating that it is the end of the list. Then, the xchg() call atomically stores a pointer to the new entry in the list's tail pointer, returning the previous value of that pointer; in the case of an empty list, that will be a pointer to the stub entry. The next pointer of the previous end-of-list entry (which, again, might have been the stub) is then set to the new entry, completing the task of adding that entry to the list.

There is a subtlety to lockless algorithms that is worth noting here. Once the tail pointer has been aimed at the new list entry, that entry is visible to the rest of the world. Among other things, that visibility implies that the next pointer in the new entry must be properly set before changing tail. Normally, either the compiler or the CPU could feel entitled to reorder the assignments to next and tail, which could result in the new tail being visible before the new node is fully initialized. The xchg() operation, though, is defined as a full barrier, meaning that operations that happen prior (such as the assignment to next) must be visible to the rest of the system before the exchange can take place. In the absence of a barrier operation, it would have been necessary to manually insert a barrier between those two assignments.

If multiple CPUs attempt to add entries to the same list at the same time, the xchg() call will serialize them, ensuring that the tail pointer is updated in an orderly manner. If two CPUs perform their xchg() calls simultaneously, one will "win" and go first, followed by the other. That could result in a list that, briefly, looks like this:

Since each xchg() call returns the previous state of the tail pointer, each addition knows where the preceding entry in the list is. That allows it to set the next pointer accordingly. Once the two additions complete, the list will look like:

There is no locking required to keep the list in a coherent state, and no retry loops are needed, so addition operations are fast. Additions can be made while running within any kernel context as well.

The consumer's view

The consumer side is just a little bit more involved, starting with the fact that the consumer maintains a head-of-list pointer separately from the mpscq structure; it is a simple struct llist_node pointer holding the address of the first entry in the list. The purpose of this separation is to ensure that the head and tail pointers are placed in separate cache lines, avoiding cache contention between the producers and the consumer. To remove the first entry in the list, the consumer will pass that head pointer to:

    static inline struct llist_node *mpscq_pop(struct mpscq *q,
					   struct llist_node **headp);

There are a few cases that this function has to be prepared for. When the list is first created, in an empty state (as shown above), the head-of-list pointer will contain the address of the stub entry. Since additions to the list do not change the head pointer, that situation will remain until the first item is removed from the list. Imagine that no items have yet been removed from the list shown above; with the separate head pointer, the picture looks like this:

Item removal for that case is handled this way:

	struct llist_node *head = *headp, *next;

	if (head == &q->stub) {
	    head = READ_ONCE(head->next);
	    if (!head)
		return NULL;
	    q->stub.next = NULL;
	    *headp = head;
	}

Remember that the addition of the first entry to the list set the next pointer in stub to that first entry; here the code checks that pointer. If it is NULL then the list is empty, so NULL is returned. Otherwise the head is advanced to the value of the stub's next field, which is subsequently set to NULL.

The stub has no further role in the management of the list until it is emptied again.

Now that it has been established that there is an entry on the list, the next check is to see if it is the last entry. In the negative case, when further entries exist, the head pointer can be advanced to the next of those entries, and a pointer to the head entry returned:

	next = READ_ONCE(head->next);
	if (next) {
	    *headp = next;
	    return head;
	}

After an item is returned in this way, the situation is:

If the next pointer is NULL, though, then there are no more entries in the list, and the tail pointer must, once again, be set to the stub. There is a twist, though: there may be a producer adding a new entry to the list at the same time. So a compare-and-exchange operation is needed to attempt to reset the list to the empty state:

	if (try_cmpxchg(&q->tail, &head, &q->stub)) {
	    *headp = &q->stub;
	    return head;
	}
	return NULL;

The try_cmpxchg() call compares the tail pointer to the head pointer (which, remember, points to the one entry in the list). If the two are equal, it atomically sets the tail to point to the stub, resetting the list to the empty state shown at the beginning; it then returns the last entry in the list.

If, however, the try_cmpxchg() call fails, then the consumer has raced with another producer, and that producer has changed the tail pointer behind the consumer's back. In this case, NULL is returned and the last entry is left on the list until the next time the consumer retries. The consumer can distinguish this case from the list-is-empty case by seeing whether the tail pointer is aimed at the stub entry:

    static inline bool mpscq_empty(struct mpscq *q)
    {
	return READ_ONCE(q->tail) == &q->stub;
    }

That describes the entire API. The code can be found in io_uring/mpscq.h; it is not, at this point, placed under lib/ and made available to the rest of the kernel. That could, of course, be changed if an interested user outside of io_uring were to emerge.

As of 7.2, this new queue type is used for a couple of different task lists within io_uring. The results, as described in this patch, are a significant increase in performance with reduced overhead — more work is done more quickly, while simultaneously reducing the amount of time spent executing in the kernel. The io_uring code is also simplified somewhat, since it no longer needs to reverse the lists or maintain a separate list of work items that were removed from the task list but not yet acted upon. All told, it would appear to be an optimization whose time has come.

(Postscript: this topic was at risk of being passed over, but it received enough votes on the LWN public topics page, which is available to subscribers at the Project Leader level and above, that I decided to give it another look. My thanks go to the LWN readers who thought this development was worthy of an article.)

Comments (7 posted)

Topics in filesystem testing

It should come as no surprise that a gathering of filesystem developers would discuss filesystem testing; it has been a mainstay of the Linux Storage, Filesystem, Memory Management, and BPF Summit over the years and the 2026 summit was no exception. Ted Ts'o led the discussion this time; he had a few different topics to raise, including his perception of increasing regressions for ext4 in the stable kernels and what can be done to help reduce them. As with other similar sessions at the summit over the years, there is a lot of interest in collaborating on test inputs and outputs, but finding a way to centralize that information has so far eluded the filesystem community.

Ts'o began by noting that he has been noticing more ext4 regressions in the stable kernels of late. Part of the reason is that the ext4 developers have been working on features like support for folios; some of those patches "have subtle dependency requirements that aren't necessarily getting picked up by the automation".

Another factor is that patches are being backported into older kernels more frequently, possibly with the assistance of LLMs, he said. So he has seen features that were backported into the 6.1 and 6.6 stable kernels, which led to bugs in those kernels. Some of the bugs caused the kernel to crash on certain tests in the fstests suite. Since there were more than a dozen patches backported, it was "quite painful to actually find those issues". He wondered if other filesystems that had not opted out of the automated patch selection for stable kernels were also encountering that problems.

He has set up a test runner that monitors the patches bound for stable kernels; it will run fstests on kernels with those patches. He has not had time to review the results and compare them to a baseline to find regressions, however. It is something he can automate, but has not gotten there yet.

Ts'o said that any filesystems that can be tested with fstests in his test runner, "which is most of them", could be added into the mix for testing with the stable-kernel patch candidates. He has the capacity to do so and can provide reports via email so that more filesystems can be tested with the stable backports. He also put out the word that he was looking for a Python programmer to develop a program to compare the test output from two runs to find regressions between them.

Ts'o has also been spending some time on automation for his xfstests-bld test appliance. He has added support for doing Git bisection, including for situations where the kernel crashes while running the tests. He would be happy to help get that set up for anyone interested. Kdevops is another option for filesystem developers to use. He suspects that there will be a lot more activity due to bug reports and patches from LLMs; "testing is the only way we can stay on top of it".

He then opened the floor for others to share their ideas about filesystem testing. One attendee suggested a shared database of the test matrix and the test results, noting that the idea has come up before. Others agreed, but noted that the environments used for testing—real hardware, virtual machines, different kinds of storage, and so on—make it hard to compare results from testing efforts.

Chuck Lever said that the kdevops project has an archive of results from running fstests, which might be a good starting point. He had also just found out that the kernel networking subsystem (netdev) stores its continuous-integration (CI) test results in patchwork, which has the ability to store data with the patches being tracked. Netdev is using that for storing its CI results. (More information can be found on the Netdev Infrastructure for Patch Automation wiki.)

Ts'o said that he had asked Konstantin Ryabitsev about setting up a mailing list for test results that would be archived at lore.kernel.org. When the request was made a year or so ago, Ryabitsev was not enthusiastic about having automated test results stored that way, probably because of the volume of data that might be produced. If others thought there might be value in a list like that, Ts'o said that he could raise the idea again.

Two attendees described dashboards that are used for testing reports in their companies. Ts'o suggested that any open-source efforts of that nature should be posted to the fstests mailing list, since there may be other developers who would use them. Developing a central database for test results with a dashboard that can be used to monitor them is an idea that comes up at every summit, Lever said. He thought it might be a "moonshot", but perhaps the Linux Foundation could be enlisted to help make that happen. Ts'o thinks the foundation believes it is already solving the problem with the KernelCI project, but that effort is not well-suited to filesystem testing.

Ts'o said it might be easier to get some one-time funding to simply develop a tool, rather than creating a project like KernelCI, but for filesystem testing, which requires ongoing fundraising to maintain. He suggested that getting filesystem developers to agree on what is needed, maybe around a prototype that someone has vibe-coded, might lead to funding to create a production-ready version of the tool. "We should put our heads together offline."

Another item that Ts'o wanted to raise was the files of test failures that he is maintaining. They are like fstests expunge files, which list tests that should not be run, but are based on the kernel version where the test does not pass. They cover per-filesystem-type failures as well as failures based on a combination of filesystem-type and test scenarios; they document tests that do not work in various long-term-stable (LTS) kernel versions and that likely never will work in those versions.

He said that distributions tend to simply pick up an older version of fstests that corresponds with the kernel they are using. But he does not want to maintain multiple versions of fstests and thinks there is value in running the latest tests; the newer versions of fstests will sometimes point out which kernel version fixed a bug, which may indicate useful backports. Running a newer fstests version does lead to more noise in the results because there are more tests that will never pass on, say, 6.1 or 6.6. That is why he maintains the test-failure files.

Those files currently live with his test appliance, but he wondered if they should move elsewhere and be maintained collaboratively. His focus is on ext4, so that's well-covered by the test-failure files. Lever suggested that the test-failure information be added to the fstests repository, but noted that there may be pushback to fix the tests instead. Ts'o said that the fstests maintainer has made it clear that they are not interested in tracking "what got fixed in what versions". There is a certain amount of sense to that because different people are using the tests in different ways; Ts'o only tracks LTS kernels, while distributions will want to track their kernels, which may diverge from the upstream kernel versions.

Christian Brauner raised the problem of "flaky" tests, those that only pass sometimes. Ts'o said he has an internal version of his harness that allows tests to be marked flaky; if they fail, they are run three more times and are only reported if all of those fail. He has meant to add that feature into the public version, since it is useful, but has not found the time.

Various people have their own versions of the expunge files for different kernel versions, so it would be nice to put them all together, Ts'o said. Since fstests is not the right place, maybe the kernel would be, he suggested. With that, the discussion wound down and the session ended.

[I would like to apologize for any errors here. The acoustics in the room were problematic for both hearing and recording. Misunderstanding and misidentification may have resulted.]

Comments (6 posted)

Shielding running kernels against exploits with BPF

Cisco has some unusual challenges when it comes to deploying security patches across the company's many devices running custom kernels. John Fastabend spoke about his work preventing exploits with BPF at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit. The technique could substantially reduce the time necessary to respond to kernel vulnerabilities, but it will not be fully effective unless more hooks are added to the kernel.

Network switches encompass a big range of hardware, Fastabend began. From small single-rack systems all of the way through huge high-speed devices. Each of Cisco's supported platforms has its own kernel team that builds custom kernels using Yocto. At any given time, those teams are supporting a large number of different kernels — mostly stable kernels, fortunately, he added. All of these widely deployed, internet-connected devices with custom kernels make tempting targets for attackers.

Cisco publishes security updates, obviously, but it takes time to identify a problem, write a fix, create a new build, test it, provide it to customers, and then let switches update, especially because rebooting switches can cause network disruptions. Those disruptions require clients to plan and manage downtime, and the whole process can take months from discovery to patching the last vulnerable systems. The goal of Fastabend's work has been to use BPF to observe attacks in real time, and then allow them to be addressed on demand, without rebooting. Ideally, the whole process would take only minutes, he said. "We won't be there for a while, but that would be the dream."

Tetragon, the open-source BPF-based monitoring and enforcement tool, is used to collect "lots of data" about running systems. At any time, the monitoring infrastructure on a switch can show which programs ran at which times, and which network connections they made. That data is stored in a time-series database. Tetragon does currently depend on a user-space agent to remain operational, but Fastabend and his colleagues have been working toward making the BPF components survive even if an attacker manages to kill the user-space agent. When a new CVE is discovered, he explained, he wants to be able to check against that database to find out whether it was ever exploited. The data can also be used to see the symptoms of an attack, such as data exfiltration or connections to command-and-control servers.

Once an exploit is identified, it can be blocked directly from BPF. If a particular system call is necessary to trigger the exploit, BPF can override the return value of the system call to refuse the operation. It is also possible to use tracepoints to verify that the arguments to internal kernel functions are correct. Fastabend's team uses both uprobes and kprobes for that. Those probes don't reliably allow changing return values, however, so Linux security-module (LSM) hooks are used for that.

Andrii Nakryiko asked how many events per second were being checked and potentially intercepted by this design. The routing of network packets is mostly done by dedicated hardware, Fastabend explained, so the kernel only needs to manage control-plane traffic and user-space applications. Overall, there are only hundreds or thousands of events per second, not billions, even if the switch is moving billions of packets.

One complication is that Fastabend's team wants to use probes to operate on inlined functions as well. That is possible by using debugging symbols and setting a probe at a raw offset. Cisco has a build farm that is used to produce all of its kernel packages, he explained. The build machines save the build IDs and debugging information from all of the builds, including both BTF and normal debugging symbols. That information is used to debug customer problems, but also to make it possible to write live kernel patches or BPF programs that are specific to the structure of deployed kernels.

Jakub Sitnicki noted that he has experienced a problem attaching probes to functions that are partially inlined, since information on where those functions have been inlined is not presently included in the kernel's BTF, but the problem is being worked on. Alan Maguire said that the topic would be covered in one of the sessions he had proposed for the next day of the conference.

Blocking exploits

Fastabend then showed an example of a BPF program that could be used to block the effects of the recent copy fail vulnerability. The program just made the splice() system call return an error when called in a way that would trigger the bug. His team calls such BPF programs "shields". Something like this would technically be possible to do with a normal kernel live patch, he admitted, but Cisco has so many concurrent product lines, with many different stable kernels running on them, that it would require a huge investment of developer time to patch them all. With BPF, the same program can typically run across all of the supported kernels — it just needs to be written once, and then automatically tested on each kernel to make sure it doesn't break anything.

BPF shields are great when they work, but there are some occasional hiccups. Often, Fastabend's team will find a CVE that doesn't have any relevant hooks in the affected kernel subsystem to build a shield around, he said. Most recently, there was an exploitable use-after-free bug that simply didn't have anywhere convenient to hook close to the source of the problem. The team eventually settled on hooking the system call that could lead to triggering the bug, but it made for more complex code.

Therefore, the main change that Fastabend would like to see in the kernel to support efforts like this is a more inclusive policy for ALLOW_ERROR_INJECTION(), the macro that is used to mark functions that can be subjected to the kernel's error-injection framework. While normally used for testing, the framework allows kernel programmers to override the return values of internal functions with custom BPF programs, which is a neat match for Fastabend's use case. Unfortunately, only a subset of kernel functions have been marked for use with error-injection. Ideally, he would like to be able to use BPF to modify the return value of any function that returns an integer which is compared to zero and has that error propagated up the stack. There are plenty of functions that match this criterion, he said. There should be LSM hooks for all of these, he said. Fastabend asked whether those functions could easily be made hookable from BPF.

Nakryiko thought that the complexity of error-handling code would make any change like that a manual process. Alexei Starovoitov suggested that it should be possible to do automatically for Rust code, given that the compiler-generated cleanup logic in Rust code has a predictable structure. For C code, he suggested asking the people submitting vulnerability reports to introduce a relevant LSM hook as well. I suggested using Coccinelle to make the change.

If it did end up being a manual process, the most important place to target would be the kernel's netlink code, Fastabend thought. For whatever reason, his team sees a lot of attacks targeting that area of the kernel.

The technique of hooking internal kernel functions as a protection against vulnerabilities is certainly useful to other users of the Linux kernel as well. The shield he showed was only a handful of lines of code, and it is easy to see how, with sufficient coverage of the kernel, that kind of simple fix would provide a way for companies or distributions to deploy vulnerability mitigations across multiple kernel versions with much less hassle. That said, the work of adding LSM hooks to the relevant areas will be a large change, and may take some time — if the LSM maintainers approve of the work at all.

Comments (7 posted)

Sending packets directly from BPF

Tetragon, the BPF-based security monitoring tool, uses BPF to monitor different aspects of a running kernel and enforce user-specified policies. It sends its data to a user-space process, which forwards the data to a central monitoring service elsewhere in the network, however. This presents a point of vulnerability: if an attacker can kill Tetragon's user-space agent, it won't be able to properly report on the situation. Song Liu, Mahé Tardy, and Liam Wiseheart spoke about their work removing the need for the user-space agent at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit.

Wiseheart clarified that he works at Meta, not on Tetragon, but he has an interest in solving the same problem of allowing BPF programs to run entirely disconnected from any user-space components. Currently, Meta takes the approach of pinning programs at system boot time, which at least prevents the programs from being removed if the user-space components are killed, but doesn't completely avoid the problem.

The way Tetragon (presumably like Wiseheart's similar program) communicates with user space is via a ring buffer, Tardy explained. The user-space component is mostly responsible for reading messages from that ring buffer, sending them to a remote server, receiving replies, and then putting the replies back into the ring buffer. It would be a lot more efficient if the BPF program could simply talk to the remote server directly. BPF programs can already intercept incoming network packets; the missing part is the ability to send data directly from BPF.

In 2025, Tardy and company had presented a solution using splice(), but that solution was unpopular. At the time, Andrii Nakryiko thought that a synchronous option was probably a bad fit for BPF. The kernel developers at that session suggested using the netconsole code, which allows the kernel to send log messages that would ordinarily go to a serial port to a remote location instead.

"We tried that," Tardy said, and it seems to work. Netpoll, the kernel infrastructure behind netconsole, lets kernel code send packets from any context and bypasses the normal networking stack, which is handy. So, they wrote a kfunc and sent a patch set for it, with an updated version later in the year. To use it, a user-space loading program would pass network address information to the BPF program, which would call bpf_netpoll_create() to create a netpoll context. Then bpf_netpoll_send_udp() is used to send UDP packets containing arbitrary data.

Tardy then did a little demo. He booted a virtual machine, started a demo agent that installed a BPF program that sent occasional pings to the host machine, and showed those messages coming through on a separate terminal. Then he killed the user-space agent and showed that the pings kept arriving. The new packet-sending functions can be combined with the kernel's existing cryptography API to send encrypted packets, he added. Currently, his demo uses a simple symmetric key, but it's possible to use more complex schemes.

Even though the netpoll solution works, "we've had some feedback that UDP is evil," Liu said. For one thing, the packets sent by netpoll bypass the normal networking stack, which means that if a BPF program sends too much traffic, it could steal bandwidth from other processes with no real way to limit contention. One audience member suggested that it wasn't necessary to use netpoll's ability to send from any kernel context — the BPF interface could spawn a kernel thread and use that to send packets in the normal way, which would let the networking code apply all of the normal networking settings to the traffic.

As of ten hours before the talk, Liu said, they had experimented with a version that sends TCP traffic instead of UDP. The networking folks were more comfortable with that, but it prevents the kfuncs from being used in atomic contexts, much as the kernel-thread-based solution would. That prompted an extensive discussion about the merits of UDP versus TCP, although most of the assembled seemed to fall on the pro-UDP side. Alexei Starovoitov didn't see a reason to prefer TCP, especially given that netpoll already exists and is used in the kernel. John Fastabend thought that UDP was sufficient for Tetragon's use case. Wiseheart pointed out that it was harder for a broken security module hook or similar problem to interfere with netpoll-based logging, since netpoll bypasses the normal networking stack.

Starovoitov shared an example of that robustness in action: he had at one point come across a situation where a network-interface card (NIC) had partially broken, with no ability to receive packets, which messed up the whole networking stack. But the netpoll code still worked to send on that device. Another person objected that the only real problem with the netpoll-based solution was bandwidth management on the host, but that it was a serious problem. Netpoll only uses a single queue on the NIC, Starovoitov said, which he doubted was enough to cause problems. Liu asked Starovoitov for his help convincing the networking maintainers of this point of view, which he promised to provide.

Daniel Borkmann asked how many NIC drivers even supported netpoll, or whether it was a generic utility that worked on every driver. Starovoitov thought that around 90% of drivers did not support netpoll — either intentionally, or just because things were broken. Only the drivers regularly used at Meta definitely handled it correctly, he said.

Nobody seemed to disagree with this, which made it unclear whether netpoll really was the best approach to use. Unfortunately, at that point the session was out of time. Since the conference, Tardy and company have continued working on the problem, posting a new patch set on July 6 that allows BPF programs to create and use UDP kernel sockets, instead of using netpoll.

Comments (4 posted)

Kitty chases the mouse

Kitty is a terminal emulator that runs on Linux, macOS, and the BSDs, which is notable for its speed and features such as image support and advanced font handling. It is under active development; a recent major release adds a new level of mouse support. Here, we will look at some of those features and show how the program can also be used as platform for text-based applications. Kitty is free software, released under the GPLv3.

Features

Kitty is a GPU-based terminal emulator. As with several other currently popular terminal emulators, use of the GPU provides it with enough bandwidth to dump megabytes of text onto the screen in an instant. This is a welcome virtue when accidentally displaying a huge file using cat, but the GPU also enables smooth exploration of the scrollback buffer. Kitty is designed to have low latency, which refers to the time between striking a key and the appearance of the corresponding character on the screen. This is a property that users are often not consciously aware of, but that has an important effect on the satisfaction and well-being of the terminal jockey.

Kovid Goyal created kitty when he became dissatisfied with the speed of the terminal emulator he was using [YouTube interview video] and decided to make a faster one. He then began to add features, starting with colored underlines, that he wanted but existing programs didn't support.

Kitty is thoroughly customizable through a configuration file. The user can define keyboard shortcuts or mouse actions for dozens of commands. The options go far beyond the usual appearance and keyboard shortcut settings; for example, there are several parameters exposed for performance tuning. In recent kitty versions, changes to the settings go into effect immediately upon saving the configuration file.

Font handling in kitty is flexible—as long as "fonts" does not include bitmap fonts, which kitty does not and will not support because they do not scale arbitrarily. Kitty can use any monospace TrueType or OpenType font and change its size on the fly with a keyboard shortcut.

Ligatures are supported (but, thankfully, can be disabled), as are alternative glyph shapes in OpenType and variable fonts. Separate fonts for normal, italic, and boldfaced text can be defined. In the configuration file, the user can also tell kitty which font to use for which Unicode codepoints, or range of codepoints. This is useful if, for example, a favorite coding font doesn't have good Unicode coverage, or if different fonts do a better job at rendering certain glyphs. There are parameters available to adjust font metrics and things like the position and thickness of underlines and strikethroughs; even how kitty composites text onto the background color can be adjusted.

If the user permits, kitty can override the colors chosen by an application running in the terminal when they would lead to insufficient contrast between foreground and background. I have this turned on, as it protects me against, for example, carelessly defining too similar foreground and background colors for a Vim highlight. As the documentation points out: "Terminal aficionados spend all day staring at text, [so] getting text rendering just right is very important."

Kittens

Kitty comes bundled with a handful of auxiliary programs called "kittens" that provide extra functionality and customization to the program. They are invoked with keyboard shortcuts or by executing the kitten command with an argument to select the desired kitten in the shell. The kitten command with no argument will display the list of available kittens. For brief documentation of any of them, a -h flag is provided; for example, kitten ssh -h for a guide to using the ssh kitten.

Kitty is designed to be extendable by the user. To this end kittens are implemented as external Python programs using an API that allows them to control kitty and gives them access to the text in a kitty window and in its scrollback buffer.

There are many kittens available; here I will mention some of the especially useful ones that come bundled with kitty. The ssh kitten wraps the system's ssh command to ensure that the remote connection plays well with kitty, mainly by transmitting the needed terminfo setting and enabling shell integration (more on that below) on the remote machine; it also brings other conveniences, such as reusing existing connections. There is a kitten that lists and previews any usable font installed. A theme picker allows interactive selection of a preset color theme from among far too many contributed options. The icat kitten uses the "terminal graphics protocol" (described below) to display an image file in any common format (including animated GIFs) directly in the terminal. Image files can be local or fetched with HTTP(S) or FTP; it will even work directly over SSH, as long as the remote machine has kitty installed.

The show-key kitten displays keycodes corresponding to key presses, similar to the xev program. A clipboard kitten enables copy and paste of various data types (e.g. images) over SSH. There is also a unicode input kitten for interactive searching and selecting Unicode characters, as can be seen in the screen shot below.

Kitty has a "hints mode" where any URL visible on the screen can be opened in the user's web browser by pressing a single key. It works like the popular Vimium browser extension, and uses a kitten under the hood. Remote control of kitty is provided by a kitten with many options allowing the user to send text to specific windows, change their properties, open and close windows and tabs, and more.

The kitty program features what the documentation calls "shell integration". This works for most popular shells such as Zsh, fish, and Bash, and third-party integrations are available for more exotic ones such as Nushell and Xonsh. Shell integration is a set of functions and shortcuts that make working at the command line more convenient. There are commands for opening the default editor with the contents of the scrollback buffer, the output of the most recent command, or the current command. Shell integration also supplies tab completion of kitty commands and various utility functions that are inserted into the shell environment, such as clone-in-kitty, which clones the current shell environment, including all environment variables, into a new window or tab. Another useful utility is edit-in-kitty, which enables editing remote files over SSH with the local editor.

New protocols

Kitty has created several new terminal protocols, implemented as sets of escape codes. Some of these protocols enable various kitty tricks, such as the terminal graphics protocol, which powers the icat command mentioned above. (Those familiar with the older sixel protocol may wonder why kitty didn't use that; the reason is that kitty's creator wanted to support a handful of features that are not supported by sixel graphics.) Additional kitty protocols improve on other legacy protocols, such as one that implements more flexible keyboard handling.

Other new protocols or protocol extensions implement colored and styled text underlines, copying of various types of data (images, text styles) to the clipboard, changing the shape of the mouse pointer, and mouse drag-and-drop of files to and from GUI file navigators. That feature is made possible by the enhanced mouse support that arrived with the latest major release.

Some of these protocols have been adopted by other projects, so, for example, users of Ghostty or Wezterm, along with several other terminal emulators, enjoy the benefit of kitty's terminal graphics protocol. It is also used by a long list of terminal-based applications, in addition to terminal emulators, and by plugins for editors. One such enhancement that I use with Neovim is the image component of the snacks plugin. It embeds graphics into Markdown buffers where it sees the Markdown syntax for inserting an image, or, optionally, displays them in popup windows. It also calls out to Typst and LaTeX (which, of course, need to be installed) to process math input and displays the result directly in the buffer. The plugin also understands Typst and LaTeX syntax for including images, and embeds those as well. Because Typst is so fast, the user gets realtime preview of the math when editing Typst documents.

Ι'll mention two final applications of kitty's terminal graphics protocol that I've found useful: both gnuplot and Julia can use it to embed their graphical output directly in their read-eval-print loops (REPLs), the former using a built-in "terminal", and the latter using the KittyTerminalImages Julia package. It can sometimes be useful to have, in the REPL, a record of plots alongside the commands that created them, an association that is lost when graphics are displayed in a separate window.

Interface builder

The kitty window can be split vertically or horizontally into any number of sub-windows, each with its own title, background color, and other properties. The resulting collection of component windows can be rearranged into a variety of layouts, and arbitrarily resized, similar to a tiling window manager such as dwm. The window arrangement can be saved as a "session" with a keyboard shortcut. The session information is saved in a text file that contains the layout details and also records what program each window was running at the time the session was created. These sessions can largely replace terminal multiplexers such as tmux.

In earlier kitty versions, arranging and resizing the windows in a session was done with the keyboard. While I generally prefer the keyboard to the mouse, this was not ideal. The new mouse handling in kitty v0.47.0 enables resizing and moving windows visually with a nicely thought-out interface that allows dragging window partitions and dragging and dropping windows to rearrange them. Session windows can even be dragged out of the session to become separate windows.

A session can loaded with the --session flag on the kitty command. The user can also load or switch sessions when using kitty with keyboard shortcuts. When a session is loaded, all of the programs that had been running in each window are started. Any shared SSH connections made with the ssh kitten mentioned above, which were active when the session was saved, will be reactivated.

An obvious use of kitty sessions is to define environments needed for various projects or tasks. A system administrator might want a session with several windows connected to different servers. A programming project could include a code editor, a REPL, and a window displaying documentation.

When sessions are combined with kitty's remote-control abilities they become the basis for simple text-user-interface (TUI) applications. To make such an application with kitty, it suffices to create the desired layout and save it as a session. The programs running in the component windows can handle user input and send and receive text and commands from each other in response. Considered this way, kitty becomes a kind of TUI construction kit.

Above is a screenshot from a simple image browser that I built using this machinery. When the user selects an image file from a list, it displays the image and some information about it. The list is filtered and displayed by fzf in one window, the image is displayed in another window with kitty's icat kitten, and the image information is displayed in a third window by processing the output from the exif command.

Kitty is generally available from distribution repositories, but for those who desire a release more recent than that, the installation instructions describe methods for binary installation and for building from source. Kitty's GitHub page shows over 400 contributors, with Goyal making the vast majority of commits. It's written in Python, C, and Go, with a smattering of a few other languages.

Conclusion

I decided to try out kitty a few years ago because I was intrigued by some of its features, such as advanced image display. It has since become indispensable. Kitty strives to be more than a terminal emulator; it provides a rich environment for textual human-computer interaction. But its panoply of features stay out of the way of the user who simply wants a fast terminal with good font support. They can be explored gradually, and incorporated into workflows as desired. I'm still discovering kitty features that make my life in the terminal more convenient.

Kitty does have at least one problem, which sometimes creates confusion: its name, which is shared with a combination networking client and terminal emulator for Windows.

Finally, because undoubtedly some readers are wondering: yes, you can play Doom in kitty.

Comments (50 posted)

QBE 1.3: metaprogramming, performance, and cross-platform support

QBE, a compact compiler backend developed by Quentin Carbonneaux, is a lightweight alternative to larger compiler backends such as LLVM and GCC. Designed to be small enough for a single developer to understand, QBE uses a static single-assignment (SSA) intermediate representation (IR), supports the C ABI, and serves as the backend for projects such as Hare and the cproc C11 compiler. Frontends emit the textual form of QBE's IR directly; QBE then takes care of register allocation, optimization, and native-code generation, producing assembly for the target architecture.

QBE is MIT-licensed, and has had 38 contributors, although Carbonneaux is by far the most prolific. On June 2, 2026, version 1.3 was released. Described in the release notes as the most significant update since 1.0 in 2022, the release adds roughly 7,000 lines of code while removing 1,500. It introduces a number of architectural changes aimed at improving the performance of code compiled with QBE, simplifying backend development via metaprogramming, and expanding platform support.

The minimalist ecosystem

QBE occupies a niche alongside other minimalist compiler projects, such as Vladimir Makarov's Medium Internal Representation (MIR) and Dmitry Stogov's IR. In discussions following the 1.3 release, participants noted that these systems share a common technical lineage rooted in traditional C programming paradigms. However, they differ sharply in their priorities.

Where MIR frequently focuses on aggressive optimization through techniques like function inlining, QBE prioritizes compilation speed and a small, maintainable code base. This design philosophy leads to a dense, traditional Unix C implementation for QBE itself that can be challenging for newcomers, but, once comprehended, has a simple structure and minimal added complications. Both the resulting binary and the external API remain minimal.

Closing the CoreMark performance gap

A primary motivation for the 1.3 release was a gap in the performance of code generated by QBE, relative to the project's stated goal of reaching roughly 70% of the performance of "gcc -O2". Earlier measurements using the CoreMark suite placed QBE closer to 40%.

According to the release notes, the push to address this gap was sparked by contributor Roland Paterson-Jones, who suggested utilizing CoreMark as a concrete baseline for optimization. Profiling identified a small number of hot functions from the CoreMark benchmark itself, most notably ee_isdigit() and crcu8(), as major contributors to the overhead. While these functions represent edge cases—CRC, for example, is best implemented with a pre-computed table rather than raw arithmetic—they are indicative of the compact, CPU-bound code sections where low-level workloads spend their execution time.

To address this, version 1.3 introduces several classic optimization passes. The additions include global value-numbering (GVN) and global code-motion (GCM), which allow the compiler to eliminate redundant computations and move them out of hot paths. Alongside GVN/GCM, QBE 1.3 includes targeted loop optimizations and if-elimination for control-flow graph simplification.

Function inlining was heavily discussed but ultimately deferred, Carbonneaux said in the release notes. QBE operates on a strict streaming, per-function compilation model, parsing and compiling functions individually to maintain a minimal memory footprint. Carbonneaux explained that inlining was postponed to avoid breaking this streaming architecture.

Despite this omission, the new passes improve the performance of the generated code to over 63% of other compilers' results on unmodified CoreMark, reaching the 70% target when the benchmark is manually adjusted to inline key routines. The new optimizations also carried over to more realistic workloads: Carbonneaux measured a 33% improvement in the runtime performance of code compiled for the Hare language test suite compared to QBE 1.2. Note that this is a measure of how fast the compiled Hare test programs run, not how fast QBE itself compiles them—QBE cannot be self-hosting, so the speed of QBE's own compilation process is not influenced by any optimizations it implements.

Metaprogramming for instruction selection

The release also revisits instruction selection. QBE has historically relied on a bottom-up tree-numbering algorithm inspired by Ken Thompson's Plan 9 C compiler. While functional, implementing this directly in pure C required intricate, hand-written logic to manage the associativity and commutativity of arithmetic operators, making the addition of new architecture backends tedious.

In the release notes, Carbonneaux stated that implementing a metaprogramming solution to this problem has been a longstanding goal. Version 1.3 introduces mgen, an OCaml-based metaprogramming tool that generates C code from declarative Lisp-like instruction patterns. Backend developers write these patterns directly in source comments; during QBE's build process, mgen scans the backend source for them and inlines the generated C code right below.

That generated code processes the directed acyclic-graph of instructions in a function in a bottom-up way. Each unique shape that a sub-graph can take is assigned a numeric code during the tree-numbering pass. Previously, turning those numbered codes into target assembly instructions required hand-written C code. QBE 1.3 replaces the bulk of that with code generated by mgen. At build time, the tool calculates which patterns could possibly match a given shape code, and creates a bit set to store them. When QBE is compiling a function, it checks the associated bit set for a given tree number to quickly see which patterns match.

If a selected pattern contains variables that need to be captured, mgen generates a dedicated matcher program for it. These programs are compiled into a simplified bytecode language and interpreted at compile time. This approach significantly reduces manual complexity, and Carbonneaux expects it may enable more advanced pattern recognition in the future, such as identifying complex bit-rotation idioms within optimization passes.

Windows ABI and shared objects

Beyond performance and maintainability, QBE 1.3 expands its practical platform support. It adds a backend for the Windows x64 ABI, contributed by developer Scott Graham.

Targeting Windows is now accomplished by passing the "-t amd64_win" flag to QBE. The System V AMD64 ABI used on Unix and Linux systems differs substantially from the Windows x64 ABI in areas such as register usage, stack alignment, and shadow-space management. Because these differences are foundational, the Windows implementation is largely separate. As Graham noted in on Hacker News, this isolation allows the code to support Windows in QBE to use Graham's slightly different coding style without polluting the System V implementations.

Furthermore, QBE 1.3 introduces support for position-independent code (PIC), enabling the generation of native shared objects on Executable and Linkable Format (ELF) systems.

The primary hurdle for shared-library generation was handling indirect access to global variables via the global offset table (GOT). QBE 1.3 solves this at the IR level by introducing a flag for external symbols that marks them as being dynamically loaded. Accessing such a variable implicitly uses a helper function that provides a location for the dynamic linker to perform relocations.

This addition allows languages built on QBE to produce modular, dynamically-linked programs in addition to the statically-linked executables that were previously possible.

Ongoing limitations and ecosystem impact

Community discussion following the release highlighted both the project's progress and its ongoing limitations. The addition of Windows support removes a key barrier for developers who previously viewed the lack as a non-starter for desktop-targeted projects.

However, QBE's strict adherence to minimalist C continues to present a steep learning curve. A major unresolved limitation is the lack of native source-level debugging-information generation, such as DWARF or PDB. While projects like Hare have taken on the complex task of building comprehensive DWARF support on top of QBE, the backend itself remains unable to generate it.

Other longstanding debates also persist, including the fact that QBE alone cannot compile itself. QBE is not self-hosting because it is just a compiler backend; it would need to add a C frontend to the project in order to become self-hosting. But, as one commenter argued in the discussion following the release, a self-hosted backend is forced to rely on its own code generation for everything. That is a constraint some compiler engineers see as useful pressure to keep pushing the backend's own optimization quality, since the project would have no other compiler to fall back on.

Despite these limitations, QBE continues to offer a highly distinctive approach: a backend with comparable performance and a simple external interface. With proven optimizations, a more flexible instruction-selection pipeline, and broader platform support, version 1.3 makes QBE a more viable option for projects that need a small, understandable backend without giving up too much performance.

Comments (4 posted)