Leading items

Welcome to the LWN.net Weekly Edition for April 2, 2026

This edition contains the following feature content:

• The many failures leading to the LiteLLM compromise: a revealing look at the weak links in the software-supply chains we depend on.
• Objections to systemd age-attestation changes go overboard: an attempt to help Linux distributions comply with age-attestation laws generates a shockingly hostile response.
• The role of LLMs in patch review: kernel maintainers object to a requirement to respond to feedback from an LLM-based review system.
• Vibe-coded ext4 for OpenBSD: an attempt at copyleft laundering is shot down.
• Rust's next-generation trait solver: the attempt to rewrite Rust's trait solver is making progress.
• Pandoc: a workhorse for document conversion: a look at the most recent release of the Pandoc document-conversion program.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

The many failures leading to the LiteLLM compromise

LiteLLM is a gateway library providing access to a number of large language models (LLMs); it is popular and widely used. On March 24, the word went out that the version of LiteLLM found in the Python Package Index (PyPI) repository had been compromised with information-stealing malware and downloaded thousands of times, sparking concern across the net. This may look like just another supply-chain attack — and it is — but the way it came about reveals just how many weak links there are in the software supply chains that we all depend on.

A Trivyal pursuit

This story starts with a different project, Trivy, which is a widely used security scanner, distributed under the Apache-2.0 license. As is often the case with these scanners, Trivy releases normally include tests for newly discovered vulnerabilities, so projects that depend on Trivy to detect potential security problems in their code have every reason to want to rerun their scans when a new release is made. Many projects hosted on GitHub thus set up an action so that, when a new release tag shows up for Trivy, new scans are automatically run, just in case Trivy has any new problems to point out.

On March 20, Paul McCarty announced that Trivy had been compromised; Philipp Burckhardt wrote up a detailed report on what was done. Somebody had managed to obtain credentials giving write access to the Trivy repository. This attacker then placed commits with malware in that repository, but they did so without changing any existing branches, avoiding the notifications that would have normally gone out in response to such changes. Instead, a large number of release tags were force-pushed to point to the new commits, which was sufficient to cause other projects to perform automatic Trivy runs.

The Trivy malware was of the information-stealing variety; whenever it ran within a project's context, it would sweep up everything it could get its virtual hands on and send it back to the attackers. Trivy itself was never the primary target; it was just a stepping stone giving access to other projects of interest. Just how many projects this exploit was able to hit may not become clear for some time.

On to LiteLLM

One of the projects that fell into that trap was LiteLLM; specifically, the PyPI credentials of one of its developers (krrishdholakia) were harvested. Those credentials were then used to upload compromised versions of LiteLLM (1.82.7 and 1.82.8) to PyPI, where they would subsequently be downloaded by the large number of people who install LiteLLM each day. Specifically, according to futuresearch, there were 47,000 downloads of the compromised LiteLLM package in just 46 minutes.

The word went out quickly, but the distribution of information was hampered by a set of hostile bots that targeted the LiteLLM GitHub issues (#24512 and #24518) with hundreds of spam comments, creating vast amounts of noise. At one point, people were advised to follow the Hacker News discussion rather than the GitHub issues themselves. The malware on PyPI was taken down quickly, but a lot of damage had been done by then.

The second compromised version of LiteLLM, in particular, used an insidious method to run the hostile code on systems where it was installed. Normally, if one installs a compromised Python program, that program will not be able to do anything until somebody runs or imports it. But when the Python interpreter starts up, it looks in the site-packages directory for files with a .pth suffix, and automatically executes the contents of every one it finds. Normally these files are used to configure Python's import paths and such, but they contain ordinary Python code. The litellm_init.pth file added by the compromised LiteLLM package was thus run anytime a Python interpreter started in the install environment (which may be anywhere in the compromised system if LiteLLM was not installed in a virtual environment), regardless of whether anything from LiteLLM itself was run.

That code, once again, gathered up all of the information it could find, including environment variables, SSH keys, Git credentials, Kubernetes secrets, shell histories, crypto wallets, and more. The whole thing was encrypted, packaged into a tar file, and pushed to a system behind the attacker-controlled litellm.cloud domain. There is no way to know how many of these uploads were done or what was in them — until the attackers make use of what they have learned.

Anybody who may have installed the compromised packages clearly has a significant cleanup job to do, including changing any credentials that might have been exfiltrated. The compromised code, for both Trivy and LiteLLM, has been taken down and should not attack any others, but this episode is far from over.

There are going to be vast numbers of compromised systems out there. It seems unlikely that LiteLLM was the only project that was compromised by the Trivy exploit. We have not heard about the others (with the exception of the telnyx compromise reported just as this article was going out); perhaps the attackers have not yet acted on other credentials that they may have acquired. But the attackers are still out there, and they seem unlikely to sit on any such material for long; expect to hear news of other compromised projects. There is also the matter of all those systems that installed the bad LiteLLM packages; again, there is a wealth of information that will have been collected by the attackers, and they seem likely to want to make use of it at some point.

In the end, this episode represents a multi-level cascading failure. A branch-change notification mechanism was bypassed. A software system meant to improve security instead compromised it. Important credentials were stored where automated exploits could read them, and two-factor authentication was evidently not in use. The protections meant to prevent the uploading of hostile packages failed. The ability of Python packages to add configuration code allowed an exploit to reach beyond the compromised package itself. Any of these failures on their own would have been enough to cause a lot of damage; taken together, then enabled a catastrophic security failure that may have only begun to play out.

Comments (42 posted)

Objections to systemd age-attestation changes go overboard

In early March, Dylan M. Taylor submitted a pull request to add a field to store a user's birth date in systemd's JSON user records. This was done to allow applications to store the date to facilitate compliance with age-attestation and -verification laws. It was to be expected that some members of the community would object; the actual response, however, has been shockingly hostile. Some of this has been fueled by a misinformation campaign that has targeted the systemd project and Taylor specifically, resulting in Taylor being doxxed and receiving death threats. Such behavior is not just problematic; it is also deeply misguided given the actual nature of the changes.

Age-attestation and -verification laws that place requirements on operating-system providers have passed in California and Brazil, and are being discussed in many other states and locations. This has led a number of Linux distributions to consider ways that they can comply with the laws if necessary. One idea that is being worked on is to add parental controls to the XDG Accounts portal which is a mechanism that allows applications to query for information about users, such as their name or avatar image. It can gather this information from various data sources, including systemd.

Taylor's changes would allow a system administrator to set and modify each user's birth date stored as birthDate in the YYYY-MM-DD format. Unprivileged users would not be able to modify the date, but could query it, so applications could retrieve the user's age via the Accounts portal. Note that this change does not automatically require or even prompt a user to provide their birth date; it only provides a way for applications to store the date. In other words, Taylor has only implemented one feature that could be used in a larger age-attestation system. His work does not implement an attestation system on its own, nor would such systems be prevented if the change were reverted; the birth date could simply be housed in another data store.

There have been some reasonable objections to the feature; as one example, Jeremy Soller, who works for System76, said that the company had been in talks with legislators, and it might be that open-source operating systems would be exempted from these laws in the end. Even if that does not happen, he worried that a dependency on systemd "which is decidedly unportable to non-Linux" systems, would require two implementations to exist.

Zbigniew Jędrzejewski-Szmek replied that while it was possible California's law would be changed, "similar ideas are popping up in other contexts and it's unlikely that they'll all go away". Ultimately, Luca Boccassi merged Taylor's changes after a bit of back-and-forth about the implementation.

Age attestation vs. verification

It is worth calling out that there is also a substantial difference between age attestation and age verification; attestation allows users or administrators of a system to provide information about their age. The person controlling the device is supplying the information. What is being discussed, for Linux systems, is a way for the user or administrator to supply the user's birth date. How that will be implemented for various desktop environments and distributions is still up in the air; the systemd birthDate field would be a small piece of any attestation systems that wind up being implemented.

The most important distinction is that an attestation system is under the control of the local administrator of a system; no documents are being checked or handed to a third party. Nothing prevents a person controlling the Linux system from supplying a random date rather than the accurate date. The services used for age attestation could also be disabled entirely by the administrator of the system, of course. Just because a Linux distribution ships a feature does not mean that users of the distribution are obliged to keep it turned on; but it should serve to satisfy the legal requirements placed on operating-system suppliers.

Age verification, on the other hand, generally requires furnishing a third party with some kind of proof—such as government identification—about one's birth date. There are many reasons to strongly object to age-verification schemes; privacy concerns and the potential for identity theft when third-party providers inevitably suffer data breaches are two that quickly come to mind. However, age verification is not what is being considered here. The fact that the two are being conflated, though, is not surprising—especially since Taylor used the term "age verification" in his pull request.

Controversy

As a rule, it does not take a great deal to cause a controversy in the open-source community. We are, to put it mildly, an opinionated bunch. Many people choose to use Linux because they wish to have full control over their operating system and applications; a change that offers exactly zero real benefit to the user other than compliance with a poorly thought-out law is not going to spark joy. One could predict a certain amount of objection and opposition to implementing age attestation no matter what.

However, the controversy appears to have been helped along by a few actors looking to stir the pot. In a discussion on the Debian developer mailing list about age-attestation systems, Otto Kekäläinen pointed to a site called The TBOTE Project that has been publishing information on the lobbying campaign to push an age-assurance law that has been introduced in a number of states across the US. Age assurance means that a device provides attestation of a user's age that has been verified, rather than self-supplied. For example, New York's Senate Bill S8102A requires "manufacturers of internet-enabled devices to conduct age assurance to determine a user's age category". This means that a device owner will likely need to perform some kind of age verification, and then the result would be stored locally.

The site also dedicates a section to the systemd project. To say it is conspiratorial in nature would be an understatement. It attempts to cast the systemd change as a sinister plot that has "altered the identity infrastructure of every major Linux distribution". While the site does not shy away from casting a spotlight on individuals in the systemd project, it does not disclose the person or persons who are behind The TBOTE Project. Aaron Rainbolt said that the site had been "used as part of a doxxing/harrassment attack" against an individual, which seems to refer to Taylor. "Given that the person running this repo actively helps doxx people, I consider them (semi-)malicious".

The TBOTE Project site is not alone in trying to fan the flames. Sam Bent, who describes himself as a YouTuber and content creator on his LinkedIn profile, published a hit piece against Taylor on March 20. It would also be best described as "malicious"; the less said about its contents the better, but it seems to have inspired some number of personal attacks against Taylor well beyond the usual objections in a GitHub pull-request comment thread. In an interview with the "It's FOSS" web site, Taylor described some of the backlash he has received; it is far beyond what an open-source contributor (or anyone else) should ever have to put up with:

I understood that the change was not going to be popular, but I was expecting civil discourse and a level-headed response. Things like death threats and harassment are not okay, especially when it negatively impacts unrelated third parties. However, the doxxing (and I am NOT just talking about my name, email and resume – that stuff is on my website, and is reasonably public. I don't commit with a pseudonym and I think it's reasonable to critique my contributions), hate mail, racism, homophobia, antisemitism, editing of my photo, turning my profile picture into memes and making fun of my appearance, etc. made me lose a bit of faith in the FOSS community. I'm really disappointed at the reaction. We should do better than this.

The unpleasant truth

The frustration and anger over government and industry pushes for age verification is justified; taking it out on open-source maintainers or projects is not. Reasonable people can disagree whether it is likely that, say, California would choose to prosecute Linux vendors or projects for non-compliance. We can certainly hope that logic would prevail, but one only needs to look around at the world today to determine that logic wins the day much less often than it should. Hoping that legal authorities will choose to overlook a project, or that the stars will align to provide a positive legal outcome, is not an effective strategy when it comes to avoiding real legal risk.

If a person or project wishes to take on that risk to challenge such laws, that is commendable; but demanding that another person or project take on such risks is unreasonable. The monetary costs of mounting a defense against a state government, even if ultimately successful, would be substantial and more than many projects could afford.

The efforts to push age-verification laws and similar unpleasantness did not show up on the open-source community's doorstep without warning. These political winds have been gathering strength for years; if they are to be challenged, they need to be dealt with by movements and organizations designed to take them on. Rather than harassing individual maintainers, for example, concerned people would do better to put their weight behind organizations like the Electronic Frontier Foundation (EFF) that have spent decades fighting exactly this sort of thing.

It is possible to counter legislative threats to open source and privacy without lobbing death threats at open-source maintainers (or anyone else). The European Union's Cyber Resilience Act (CRA) is a somewhat recent example. The CRA sets automatic security update and incident report requirements for products with digital elements; in non-legislative speak, this means software and devices that run software. The original drafts of the legislation would have saddled freely available open-source software with the same requirements as commercial products.

One might consider it reasonable and even desirable, for example, to require the manufacturer of a router, mobile phone, or car-entertainment system to provide security updates. But the same requirements would have applied to open-source maintainers and projects even when no money had changed hands. That, of course would have been a disaster for open source. FOSS organizations, such as the Eclipse Foundation, Linux Foundation Europe, Open Source Initiative, as well as several others pushed back politely, but firmly, and succeeded in having the CRA draft modified to exempt open-source software.

Attacking open-source maintainers, projects, or Linux distributions is not going to stop government and industry pushes for age verification. In fact, it will not be effective at anything except contributing to maintainer burnout, weakening the larger community, and perhaps encouraging developers to do more behind closed doors—or to stop contributing entirely.

Comments (105 posted)

The role of LLMs in patch review

Discussion of a memory-management patch set intended to clean up a helper function for handling huge pages spiraled into something else entirely after it was posted on March 19. Memory-management maintainer Andrew Morton proposed making changes to the subsystem's review process, to require patch authors to respond to feedback from Sashiko, the recently released LLM-based kernel patch review system. Other sub-maintainers, particularly Lorenzo Stoakes, objected. The resulting discussion about how and when to adopt Sashiko is potentially relevant to many other parts of the kernel.

Morton began by saying that the current way Sashiko integrates into the memory-management workflow isn't working. He merges patches to his tree, and "then half a day later a bunch of potential issues are identified." Morton stated that he was going to further increase the lag between seeing a patch set on the mailing list and merging it to his tree, to give Sashiko time to produce feedback and patch authors time to respond to it. He also wanted its reviews distributed to a wider audience — partly to better determine how useful its comments are, which he is "paying close attention to".

Stoakes said that he would look at the Sashiko reviews for the patch set, but asked Morton to hold off on incorporating it into the subsystem's workflow. He said that he appreciates the tooling, but that it is currently too noisy to use in that way. Stoakes referenced his message in the thread introducing Sashiko (that began on March 17) where he expressed the opinion that its false-positive rate was higher than his own experience using Chris Mason's kernel-review prompts. David Hildenbrand agreed that the false-positive rate was too high to be useful.

Roman Gushchin, Sashiko's creator, told Morton that he was actively working on integrating Sashiko with the kernel's email-based workflow, and that he hoped to have it sending reviews to appropriate recipients within the next week. Morton took the opportunity to ask about another problem with the tool — that many patch sets sent to the mailing list fail to apply in Sashiko's environment. In a follow-up message he expressed his intention not to apply patches to his tree that the system couldn't. Gushchin explained that Sashiko tries to apply patch sets to several bases, in order. For memory-management patches, it uses the patch set's base commit (if specified), then the mm-new tree, followed by mm-unstable, mm-stable, linux-next, and finally Linus Torvalds's tree. The review system evaluates the code in the first tree where the application attempt is successful. He didn't address why mailing-list patches would be failing to apply to these trees, however.

Stoakes asked Morton to hold off on integrating Sashiko so deeply into his workflow:

Andrew, for crying out loud. Please don't do this.

2 of the 3 series I respun on Friday, working a 13 hour day to do so, don't apply to Sashiko, but do apply to the mm tree.

I haven't the _faintest clue_ how we are supposed to factor a 3rd party experimental website applying or not applying series into our work??

He went on to say that he was not attempting to disrespect Gushchin or his efforts, but that even Gushchin had agreed that the tool was not ready to become a blocking component of the development process. Gushchin replied to say that working on Sashiko had increasingly shown him the subjectivity of reviews, and the importance of social context in providing good reviews. He acknowledged that it wasn't "perfect in any way" but suggested that some level of false positives (for example, 20%) was acceptable from a tool that catches the majority of bugs before they're merged. He suggested that this might be a reasonable lens through which to view Sashiko's current performance and future development.

Stoakes replied to clarify that he was objecting to Morton's unilateral demand that "every single point Sashiko raises is responded to". He was emphatically not blaming Gushchin for failures of the memory-management subsystem's review model, and thought the tool was promising. He reiterated his perception that the tool's false-positive rate was much higher than other people were claiming, and that — given its inexhaustible ability to produce new reviews that require human attention — it was important to think critically about what role it can play in the review process. Incorporating the tool in its present state, as anything other than a simple advisory, would increase the workload on the already overworked memory-management maintainers, he said.

This sentiment resonated with Pedro Falcato, who agreed that Sashiko should remain optional for the time being. Morton disagreed:

Rule #1 is, surely, "don't add bugs". This thing finds bugs. If its hit rate is 50% then that's plenty high enough to justify people spending time to go through and check its output.

[...]

Look, I know people are busy. If checking these reports slows us down and we end up merging less code and less buggy code then that's a good tradeoff.

Avoiding bugs is important, Falcato agreed, but: "I simply don't think either contributors or maintainers will be particularly less stressed with the introduction of obligatory AI reviews." He suggested that simply codifying the memory-management review process (as the netdev review process has been) would be more helpful than mandating the use of Sashiko (a suggestion that Mike Rapoport later supported). Falcato also pointed out that Sashiko is experimental, untested software, and it should probably not be made critical to the process yet on those grounds.

Morton responded by looking at actual replies to Sashiko's reviews on the linux-mm mailing list. Out of about 35 emails, 22 received replies indicating that alterations were definitely needed, with the rest being more ambiguous, being false reports, or not being responded to. He expressed the opinion that such a hit rate of finding actual problems in patches was worth the pain of figuring out how to integrate Sashiko into the process. "That's a really high hit rate! How can we possibly not use this, if we care about Rule #1?"

Stoakes disagreed with Morton's interpretation of the data, pointing out that those 22 emails indicate cases where the tool was correct in at least one individual observation. Since it normally sends multiple suggestions and questions per review, the actual rate of false positives for individual comments must be substantially worse than that.

Stoakes again reiterated that he found Sashiko useful, and was using it in his own reviews to some degree. The problem was in making it a mandatory part of the process. He suggested that Morton should delegate the decision of whether and how to use Sashiko to the sub-maintainers, avoid requiring that its automation cleanly applies a patch before accepting it, avoid requiring that every element of its reviews be responded to, and trust sub-maintainers to discard any parts of the reviews that are not both valid and important.

Sashiko is, at the time of writing, not even a month old. According to its statistics page, it has written over 10,000 reviews in that time, with an average of approximately 3,500 words of output (counting quoted source code) per patch. It is, quite literally, producing words faster than any individual person could reasonably read. But approximately half of those words, according to various different statistics, are about bugs that no human reviewer spotted ahead of time — either due to the difficulty of reviewing complex kernel code, or just due to a lack of infinite time to dedicate to the prospect. And several kernel contributors are finding the reviews useful.

As is the case unfortunately often, the problem posed by the use of Sashiko is a social one, not a technical one: how much extra reading, hallucinated problems, delays in the review process, robotic gatekeeping, and reliance on proprietary models is acceptable in order to make sure the kernel accepts less buggy code? It's a question that will eventually touch every subsystem of the kernel and beyond, not just the memory-management code, and one that undoubtedly deserves a lot of discussion. There are no easy answers, but hopefully the kernel community will eventually be able to reach a consensus.

Comments (17 posted)

Vibe-coded ext4 for OpenBSD

A number of projects have been struggling with the question of which submissions created by large language models (LLMs), if any, should be accepted into their code base. This discussion has been further muddied by efforts to use LLM-driven reimplemention as a way to remove copyleft restrictions from a body of existing code, as recently happened with the Python chardet module. In this context, an attempt to introduce an LLM-generated implementation of the Linux ext4 filesystem into OpenBSD was always going to create some fireworks, but that project has its own, clearly defined reasons for looking askance at such submissions.

It all started on March 17, when Thomas de Grivel posted an ext4 implementation to the openbsd-tech mailing list. This implementation, he said, provides full read and write access and passes the e2fsck filesystem checker; it does not support journaling, however. The code includes a number of copyright assertions, but says nothing about how it was written. In a blog post, though, de Grivel was more forthcoming about the code's provenance:

No Linux source files were ever read to build this driver. It's pure AI (ChatGPT and Claude-code) and careful code reviews and error checking and building kernel and rebooting/testing from my part.

There were a number of predictable concerns raised about this code, many having to do with the possibility that it could be considered to be a derived product of the (GPL-licensed) Linux implementation. The fact that the LLM in question was almost certainly trained on the Linux ext4 code and documentation does not help. Bringing GPL-licensed code into OpenBSD is, to put it lightly, not appreciated; Christian Schulte was concerned about license contamination:

I searched for documentation about that ext4 filesystem in question. I found some GPL licensed wiki pages. The majority of available documentation either directly or indirectly points at GPL licensed code. In my understanding of the issue discussed in this thread this already introduces licensing issues. Even if you would write an ext4 filesystem driver from scratch for base, you would almost always need to incorporate knowledge carrying an illiberal license.

Theo de Raadt, however, pointed out that reimplementation of structures and algorithms is allowed by copyright law; that is how interoperability happens. One should not conclude that De Raadt was in favor of merging this contribution, though.

From the OpenBSD point of view, the copyright status of LLM-generated code is indeed problematic, for the simple reason that nobody knows what that status is, or even if a copyright can exist on that code at all. Without copyright, it is not possible to grant the project the rights it needs to redistribute the code. As De Raadt explained:

At present, the software community and the legal community are unwilling to accept that the product of a (commercial, hah) AI system produces is Copyrightable by the person who merely directed the AI.

And the AI, or AI companies, are not recognized as being able to do this under Copyright treaties or laws, either. Even before we get to the point that the AI's are corpus-blenders and Copyright-blenders.

So as of today, the Copyright system does not have a way for the output of a non-human produced set of files to contain the grant of permissions which the OpenBSD project needs to perform combination and redistribution.

Damien Miller said something similar:

Who is the copyright holder in this case? It clearly draws heavily from an existing work, and it's clear the human offering the patch didn't do it. It's not the AI, because only persons can own copyright. Is it the set of people whose work was represented in the training corpus? Was the it the set of people who wrote ext4 and whose work was in the training corpus? The company who own the AI who wrote the code? Someone else?

We don't know. The law hasn't caught up to the technology yet and we can't take the risk that, when it does, it will go in a way that makes use of AI-written code now expose us to legal risk.

These words did not resonate entirely well with de Grivel, who refused to retract his copyright claims on the machine-generated code. He also is clearly pleased with the kinds of things one can do with LLMs:

We can freely steal each other in a new original way without copyright infringment its totally crazy the amount of code you can steal in just 1h. What took 20 years to Bell labs can now be done in 20 hours straight.

The conversation went on for some time, but the result was never really in doubt; De Raadt made it clear when he said: "the chances of us accepting such new code with such a suspicious Copyright situation is zero". In the above-mentioned blog post, de Grivel added a note on March 23 that he would respond by removing all of the LLM-generated code, leaving only code that he has written himself. After this episode, though, convincing others that he really did write any subsequent versions on his own may be an uphill battle. He acknowledged that "forking OpenBSD" might be easier.

The number of people who have concluded that they can have an LLM crank out thousands of lines of code and submit the result to the project of their choice is growing quickly. Needless to say, these people are not always diligent about documenting the provenance of the work they are submitting in their own names. There may well come a time when it turns out that even the sharp eyes of OpenBSD reviewers are unable to keep all of it out of their repositories.

All of this code is setting some worrisome potential traps for the future. As Tyler Anderson pointed out, the price of these tools is unlikely to go down as development projects become more dependent on them. Who will maintain this code, when its original "author" does not understand it and has no personal investment in it, is unclear at best. And if there is, in fact, a potential copyright problem inherent in this code, there will have to be a lot of scrambling (or worse) when it comes to light. Given all of that, it is unsurprising that many projects, especially those with longer time horizons, are proving reluctant to accept machine-generated submissions.

Comments (105 posted)

Rust's next-generation trait solver

Rust's compiler team has been working on a long-term project to rewrite the trait solver — the part of the compiler that determines which concrete function should be called when a programmer uses a trait method that is implemented for multiple types. The rewrite is intended to simplify future changes to the trait system, fix a handful of tricky soundness bugs, and provide faster compile times. It's also nearly finished, with a relatively small number of remaining blocking bugs.

Trait solving

Traits in Rust are similar to typeclasses in Haskell or interfaces in Java: they are a set of associated functions that can be implemented for different types. Library code can use the functions from a trait on any type that implements the trait, even if that type is foreign to the library. When it comes time to generate code, however, the compiler needs to know which specific type is being used, and where its trait implementation lives, so that it can call the correct function. Trait implementations can be generic, and therefore apply to multiple concrete types. Figuring out which trait implementation is relevant to a particular type is the job of the compiler's trait solver.

Most of the time, this is straightforward: the compiler just needs to find a trait implementation of the form impl Trait for Type {...}. The problem grows more complex when considering generic types. For example, a Vec can be cloned if and only if the type of elements it contains can be cloned. In the standard library, there is a trait implementation block that is equivalent to the following (after expanding some syntactic sugar and disregarding the irrelevant A parameter):

    impl<T> Clone for Vec<T>
    where T: Clone
    { ... }

This declaration can be viewed as an instruction for the trait solver: in order to find the Clone implementation for a Vec<Foo>, first find the Clone implementation for Foo. That implementation can be used to construct a working implementation for Vec<Foo>. The current trait solver follows these chains of instructions in a straightforward way. For each trait implementation that must be found, it determines whether it is best resolved through finding an impl statement or through using a where-bound (i.e. a where clause asserting that some generic type implements a trait). These can depend on other trait implementations existing, as in the Vec example, in which case the compiler adds them to a work list. In logic programming, these kinds of prerequisites that must be satisfied for another goal to be reached are called obligations; the compiler keeps taking obligations from the work list and processing them until they have all been eliminated. If they cannot be, an error is indicated.

This approach breaks down with more complicated types, however. It is possible to construct a situation where the obligations to be discharged form a loop. Doing so is a bit complicated, however, so even a minimal example is somewhat long:

    // Create a trait that has an associated generic type. Types that implement
    // Wrap correspond to a particular choice of Wrapper.
    trait Wrap {
        type Wrapper<T>;
    }

    // A singly linked list structure, with the payload omitted for brevity.
    // Each 'next' pointer goes to another list node wrapped in the Wrapper type
    // for some generic type W that must implement Wrap
    struct Link<W: Wrap> {
        next: Option<Box<W::Wrapper<Link<W>>>>
    }

    // Link structures can be formatted for debug print statements when their
    // contents can be — the omitted implementation calls the Debug
    // implementation for next's type, hence the second requirement.
    impl<W> Debug for Link<W>
    where W: Wrap,
          Option<Box<W::Wrapper<Link<W>>>>: Debug {
        ...      
    }

    // My Wrapper is a structure that wraps some other type T as its only field.
    struct MyWrapper<T = ()>(T);

    // And its associated wrapper type is just itself.
    impl Wrap for MyWrapper {
        type Wrapper<T> = MyWrapper<T>;
    }

    // Finally, wrappers can be formatted for debug print statements when the
    // wrapped value can be.
    impl<T> Debug for MyWrapper<T>
    where T: Debug { ... }

In order for the compiler to prove that Link<MyWrapper> implements Debug, it needs to show that a whole chain of requirements holds:

    Link<MyWrapper>: Debug ->
    Option<Box<MyWrapper::Wrapper<Link<MyWrapper>>>>: Debug ->
    Box<MyWrapper::Wrapper<Link<MyWrapper>>>: Debug ->
    MyWrapper::Wrapper<Link<MyWrapper>>: Debug ->
    Link<MyWrapper>: Debug

... but that chain eventually leads the compiler right back to where it started. Attempting to compile a Rust program that uses the Debug implementation of Link results in the error: "overflow evaluating the requirement `Link<MyWrapper>: Debug`".

In this case, however, it should be possible in theory for the compiler to generate an implementation of Debug for the same reason that assemblers can handle instructions that refer to their own address: the compiler can start generating the implementation, leaving a placeholder for the address, and then fill the address in once it knows it. At the stage where the trait solver runs, it isn't dealing with actual addresses, but the same idea applies; writing the Debug implementation for Link<MyWrapper> is a matter of "tying the knot" in a data structure representing the code to produce a structure that refers to itself.

The above example is contrived, but that's because this limitation of the language tends to come up in complex, highly generic code. Self-referential types are nothing new; it's only trying to abstract over them with traits that causes the problem. Code like that is not something that every programmer will need to write, but it is something that can make Rust libraries more flexible. Currently, crate authors need to work around this trait resolution problem by restructuring their interfaces to avoid creating loops in the trait solver.

A new approach

The trait solver's behavior isn't an inherent aspect of the problem, however. It's an implementation choice that was made essentially by default as the language's trait system slowly became more complex. The project to rewrite the trait solver began in 2015 with Chalk, an attempt to formalize Rust's type system. It compiles the rules of the type system — such as the mechanics of trait solving — into a Prolog-like system of logical inference rules that can then be solved generically with well-known logic-programming techniques. Chalk has two internal implementations of logical inference with their own strengths and weaknesses: the SLG solver (which uses an approach based on Prolog tabling) and the recursive solver, which more closely matches the operation of Rust's original trait solver. The SLG solver can handle some kinds of loops that the recursive solver cannot.

Rust developers determined that Chalk wasn't a good fit for the Rust compiler, however. One major concern was with maintaining the usefulness of error messages; the current implementation produces good error messages for simple mistakes, but Chalk's error messages have lagged behind. In 2023, the project decided to rewrite the trait solver within the compiler, using lessons from the first implementation and from Chalk.

The rewrite, called the next-generation trait solver, makes use of caching to both speed up trait solving and to defeat cycles. When called upon to start finding a trait implementation for a type, it starts by assuming that such an implementation will be found, and making a spot for it in the cache — it is marked as provisionally true, but cannot yet be fully relied on. Then the solver follows the chain of logical inferences in much the same manner as the current solver, with the exception that it looks results up in the cache when possible. When it derives a new piece of information, that is added to the cache; if deriving the new information relied on any provisional cache entries, the new entry is also marked as provisional.

If the trait solver ever reaches a point where the only remaining logical obligations refer to provisionally true cache entries, the entries are upgraded to actually true and can be referenced by other type-checking operations. On the other hand, if it doesn't reach such a point, the provisional cache entries are invalidated, and that branch of the computation is considered failed. It will either fail entirely (and show an error to the user), or continue trying some alternative potential solution.

In the case of the example above, when the new solver reached the state of needing to prove that Link<MyWrapper> implements Debug for the second time, it would see that the trait implementation had been cached as provisionally true by the previous exploration and (lacking other logical obligations) conclude that Link<MyWrapper> really does implement Debug.

Currently, the next-generation trait solver only extends this notion of provisional cache entries to a limited number of built-in traits, such as Send, but the plan is to eventually enable the behavior for all traits. The Rust developers are fairly sure that enabling this feature will just allow some code (like the above example) that doesn't currently compile to work, but since this is a logically intricate piece of code on which the correctness of the type system depends, they don't want to be too hasty. The next-generation trait solver itself is currently marked experimental for much the same reason.

One other interesting aspect of the next-generation trait solver is canonicalization. Since its design relies heavily on caching, the more types that can be cached, the faster compilation using the new trait solver will be. For that reason, logical obligations encountered in the course of trait solving are rewritten into a format that removes as many type variables and intermediate results as possible. This also has the nice effect of making it easy to reconstruct proof trees explaining how the compiler reached a particular answer. These are used to create more detailed error messages by explaining which options were tried and why they didn't work. Since the proof trees don't need to be generated unless an error message needs to be shown, being able to reconstruct them from the cache avoids the overhead of constructing them for code that does compile.

Trying it out

The next-generation trait solver is already used in stable Rust for coherence checking (that is, checking that trait implementations do not overlap), but is not yet used for general trait solving. In the nightly release of the compiler, the -Znext-solver=globally flag uses the next-generation trait solver for all trait-related type-system computations. The Rust developers have already used Crater to check that the flag does not cause compile errors for crates publicly released on crates.io, but would like to know if the flag causes problems for anyone's code prior to stabilization.

There are currently 76 open bugs related to the new trait solver, and 78 closed ones. The feature has made more progress than that suggests, however, since several of the closed bugs are rather large, and the remaining bugs are mostly internal compiler errors or performance problems that the Rust developers would like to see addressed. The new trait solver is intended to be both faster and more maintainable than the previous implementation, but whether that promise bears out remains to be seen. In either case, Rust developers may soon have the option to write more-complicated self-referential trait-based interfaces.

Comments (18 posted)

Pandoc: a workhorse for document conversion

Pandoc is a document-conversion program that can translate among a myriad of formats, including LaTeX, HTML, Office Open XML (docx), plain text, and Markdown. It is also extensible by writing Lua filters that can manipulate the document structure and perform arbitrary computations. Pandoc has appeared in various LWN articles over the years, such as my look at Typst and at the importance of free software to science in 2025, but we have missed providing an overview of the tool. The February release of Pandoc 3.9, which comes with the ability to compile the program to WebAssembly (Wasm), allowing Pandoc to run in web browsers, will likely also be of interest.

Problems solved

Writers, scholars, scientists, and many others often need to create versions of their documents in various formats. For example, a mathematician, after preparing a paper formatted with LaTeX, may want to generate an HTML version to share on the web, and later create a set of slides for a conference. Sections of the paper might be useful in a grant application, but the funding agency might require a Microsoft Word format. Creating these various incarnations would seem to require a great deal of redundant work, especially if the paper contains equations, figures, and references.

This is the problem that Pandoc was created to solve. Using Pandoc, our hypothetical mathematician can write the paper once and have it quickly and automatically converted to any of over 40 output formats. Others may simply have a document in some format and want it translated to a different format to use it in some other way. Pandoc also lurks behind other projects and online services that require document conversion, such as the popular Quarto "scientific and technical publishing system". The list of Pandoc input and output formats can be found in its README file.

A program that directly translated among N formats would have to incorporate N² translators. Instead of implementing the resulting hundreds of translation routines, Pandoc relies on a modular architecture. It ingests documents using "readers" specialized for each input format, whose job is to translate into Pandoc's internal abstract syntax tree (AST). Output is emitted by "writers" that translate the internal representation into the desired output format.

This strategy turns the N² problem into a 2N problem. It also makes it easier to enlarge Pandoc's repertoire: a contributor who would like the system to be able to read a new format need only write a Lua program to convert that format into the internal representation, after which Pandoc will be able to translate the format into any of its many outputs. Similarly, a writer module for a new format gives Pandoc the ability to convert any of the document types that it already understands into the new one. (I've simplified the situation somewhat, in that the set of document formats that Pandoc can read is not identical to the set of formats that it can write, but they have a large intersection.)

In practice, the most effective way to start using Pandoc for a new document is to write it in Pandoc's extended dialect of Markdown. This is because the readers for different formats vary in their abilities, but the Markdown reader is fully featured and complete. That variability is necessarily the case, because not every document format can be reliably translated into an AST. For example, LaTeX and Typst can embody arbitrary programs, whose output cannot be determined without running their respective compilers.

If a Markdown document is saved in a file called example.md, this command would translate it into a Typst document:

    pandoc example.md -o example.typ

The output format is inferred from the file extension. To translate it into HTML, a .html extension can be used instead. Another option is to specify the output format with the "-t" flag, for example "-t html".

Pandoc's Markdown dialect incorporates additions to the original specification that make it more suitable for expressing complex documents, while retaining the signature advantage of being pleasant to parse by eye. In addition to everything in traditional Markdown, it can handle ordered lists with letters or Roman numerals, task lists, definition lists, various styles of tables, LaTeX mathematics, footnotes, bibliographic citations, and quite a bit more.

Authors can also insert raw markup into the Markdown source, which is passed directly to the output file, bypassing Pandoc processing. Pandoc supplies syntax for raw markup that allows different fragments to be inserted in the output, depending on the format. For example, this code listing renders the name of Donald Knuth's typesetting engine in plain text, HTML, and (La)TeX; the appropriate fragment is used depending on the output format passed to Pandoc on the command line (or implied by the output file extension).

   The `TeX`{=plain}
     `T<span style="position:relative;top:0.2em">E</span>X`{=html}
     `\TeX\ `{=latex} typesetting system.  

The figure below shows all three versions, as they appear in a terminal, a web browser, and a PDF viewer, respectively. The PDF version was generated by LuaLaTeX processing the LaTeX output.

Citations

Pandoc has sophisticated support for citations and bibliographies. This feature is what makes it possible for scientists and scholars to write in Pandoc's extended Markdown rather than directly in LaTeX. Pandoc borrows the relevant syntax and mechanisms from LaTeX, using similar markup to indicate citations, the same BibTeX text-file databases of references, and the same standard Citation Style Language (CSL) bibliography style files.

As a simple example of how this works, here is a fragment of a document written in Pandoc's Markdown:

   Free software [@fsfs] is important for physics [@heisenbergBeyond] and
   other sciences.  

The bits inside the square brackets indicate references to resources defined in a BibTeX database. This a text file containing the bibliographic data for papers, books, websites, and anything else that an author might want to refer to. Here is the relevant part of the database for this document:

   @article{fsfs,
    author = {Lee Phillips},
    journal = {LWN},
    month = {June},
    title = {{T}he {I}mportance of {F}ree {S}oftware to {S}cience},
    url = {https://lwn.net/Articles/1023299/},
    year = {2025}
   }
   
   @book{heisenbergBeyond,
    address = {New York},
    author = {Werner Heisenberg},
    isbn = {06-131622-9},
    publisher = {Harper {and} Row},
    title = {{P}hysics and {B}eyond},
    url = {https://www.amazon.com/Physics-Beyond-Encounters-Conversations-Perspectives/dp/0061316229/},
    year = {1971}
   }  

The fragment can then be processed with Pandoc to create output, in any of the formats that it supports, with references and a bibliography. The command for doing so is:

   pandoc --citeproc --bibliography=refs.bib refsExample.md -o refsExample.html

In this command "--citeproc" flag tells Pandoc to use its citation machinery and "--bibliography=" points it to the BibTeX file. The input file is refsExample.md, the file extension indicating Markdown. The output will be in HTML5 because of the extension for the output file. The result will be formatted using the default bibliography style, which happens to be the Chicago Manual of Style author-date format. Different styles can be used by downloading the desired CSL file (a good source is the Zotero style repository)

The figure below shows how the result appears in my web browser, first using the default bibliography style and, below that, using a numerical superscript style. The Pandoc command line is the same in both cases; the only change was the installation of a different CSL file in the default location in the filesystem (another option is to use a "--csl" flag to point Pandoc at the desired file).

In a real project the bibliography would be set off from the main text by a header and, probably, a different font, both of which can be conveniently added by Pandoc.

We can even have plain text output with citations, for those occasions when our emails or website comments require maximum pedantry. Here's the output of the same command with the "-t plain" flag added to request plain text:

   Free software¹ is important for physics² and other sciences.
   
   1. Phillips, Lee. (2025). The Importance of Free Software to Science.
   LWN. https://lwn.net/Articles/1023299/
   
   2. Heisenberg, Werner. (1971). Physics and Beyond. Harper and Row. New
   York. ISBN 06-131622-9.

Because it is a command-line program, Pandoc can be flexibly incorporated into various workflows and used in concert with other tools. I have a keyboard shortcut defined in my editor that processes any selected text through Pandoc, using "--citeproc", and places the plain-text output in the clipboard, ready for pasting into emails or comment boxes.

Pandoc filters

A Pandoc filter is a Lua program that acts directly upon the AST. Filters can make changes to document elements and can perform arbitrary computations based on the contents of those elements. For example, a user might create a filter to number all of the paragraphs in a report, or to print a timestamp under an article's title.

There is also an older type of filter that manipulates a JSON serialization of the AST. These filters can be written in any language, but they have extra dependencies and are slower, as they require serialization and deserialization steps. The current recommendation is to write filters in Lua.

Using Lua filters involves no dependencies, as Pandoc has a Lua interpreter built in. In fact, entering "pandoc lua" in the terminal opens a standard Lua read-eval-print loop (REPL).

Here's an example of a simple filter that changes all underlined elements to Strong elements, which are rendered in most output formats as boldface:

   function Underline(elem)
       return pandoc.Strong(elem.content)
   end

To apply this filter, the program can be invoked with the argument "--lua-filter filter.lua" if filter.lua in the current directory contains the code above.

The function doesn't make much sense outside of the Pandoc context, but filters have access to an API that, somewhat magically, turns function names into routines that walk the AST and process nodes matching the name of the function. Another example is a filter that I wrote that allows me to input equations using the more concise Typst math syntax, rather than LaTeX, even when I need to produce a LaTeX document.

Filters are a powerful means to bend Pandoc to serve nearly any document-processing task, limited only by the imagination and patience of the author/programmer. Other potential applications might be to retrieve information from the Internet to insert into documents, to invoke external programs to create illustrations, or to assemble outlines.

Custom readers and writers

Although Pandoc can handle an impressive variety of document formats, it would not make sense to include every highly specialized or ad hoc markup system in existence. For more arcane applications, Pandoc provides the ability to create custom readers and custom writers. A custom reader is a Lua program that parses input text and translates it into Pandoc's AST; a writer does the reverse. To make this more convenient, Pandoc includes Lua's LPeg parsing library.

Some examples of custom readers are pandoc-fountain, which ingests screenplays written using Fountain markup, the IDML Pandoc reader that understands the markup language used in Adobe InDesign projects, and Lean.lua that can handle Lean files. Interesting custom writers include Pandoc Terminal Writer for colorful pretty-printing on the terminal, Pandoc to PreTeXt that creates markup in the PreTeXt format, which is a structural markup language for textbooks and papers, and a custom writer that emits the specialized format used in a discussion forum.

Changes for 3.9

Pandoc 3.9 was released in early February. The most prominent new feature is the ability to compile Pandoc to Wasm, allowing it to run entirely in a web browser. The project has provided an example page that allows users to do general document conversions via a web form. Pandoc in the browser leads to some interesting possibilities. For example, it should be practical to construct pages that accept user content in nearly any markup format, either in comment boxes or as wiki-style editable text, and convert it in realtime to valid HTML5 for inclusion in the page. Another possibility might be educational sites that, for instance, teach Typst or LaTeX with live feedback.

The new release brings with it dozens of small improvements and bug fixes. Most of these are technical minutiae having to do with corner cases, layout subtleties, or obscure markup formats.

There are two enhancements that may be of wider interest. One is wider support for specifying PDF standards (PDF/A, etc.), as long as LuaLaTeX is used to produce the output. The other is a significant improvement to citation handling that allows the author to reset citation history when needed, typically at the beginnings of book chapters. Standard practice is to abbreviate citations after the first mention (omitting publication details, etc.). But at the beginning of a new chapter, there may be a desire to see the full citation again.

Installations and origins

Pandoc is a free program, released under the GPLv2 (or later). It is available in the package repositories of most Linux distributions, but those versions may be behind the recent release. To get the latest release, the source, as well as binaries for Wasm, Windows, macOS, and Linux for both major architectures, are available on the web site. Those who want to use Pandoc to create PDFs must also have LaTeX, Typst, or another program that can accept one of Pandoc's output formats for conversion to PDF, installed. Pandoc will use LaTeX by default, but the user can specify another program.

The Pandoc documentation is quite good, and is all I've ever needed to support my extensive and daily use of the system.

Pandoc was created by John MacFarlane, a professor of philosophy at UC Berkeley specializing in the philosophy of language. The project is distinguished by being one of the few widely used Haskell programs. In an interview from 2023 its creator said:

I think Pandoc is, in general, written in fairly simple Haskell. I don't use too many complicated things in there. And that's partly due to the fact that I wasn't a very sophisticated Haskeller when I started, and I'm still not that much more sophisticated now.

Conclusions

Pandoc is indispensable for anyone who needs to create different kinds of written documents. With Pandoc, I can write for a publication that insists on Word documents without ever touching a GUI program, turn a set of notes into a web page, or have a book's illustrations automatically generated. Pandoc is fast and never, in my experience, fails to do exactly what it is supposed to do.

The program is actively developed and continually improved in its GitHub repository. Contributions are welcomed, and those interested in helping are not necessarily excluded by a lack of intimacy with Haskell. As MacFarlane remarked in the interview: "it's possible for people who are generally familiar with computer languages to look at some Pandoc code sometimes, even if they don't know Haskell, and figure out what might be needed".

Pandoc is most powerful as a command-line tool, where it can be incorporated into scripts and adapted to various document-processing pipelines. However, its new ability to be embedded in web pages is an interesting development that may lead to a wider variety of uses and, perhaps as a secondary effect, to an expanded interest in Haskell.

Comments (11 posted)