LWN.net Weekly Edition for February 1, 2024

Welcome to the LWN.net Weekly Edition for February 1, 2024

This edition contains the following feature content:

• OpenBSD system-call pinning: a new OpenBSD system call deepens its defense against ROP attacks.
• Looking ahead to Emacs 30: what's coming in the next release of the Emacs editor.
• Better handling of integer wraparound in the kernel: unexpected overflow and wraparound can lead to bugs and vulnerabilities; efforts are being made to prevent unpleasant surprises.
• The things nobody wants to pay for: the dark areas of our community that nobody feels the need to fund.
• Defining the Rust 2024 edition: the process of deciding what to include in this year's Rust edition.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

OpenBSD system-call pinning

Return-oriented programming (ROP) attacks are hard to defend against. Partial mitigations such as address-space layout randomization, stack canaries, and other techniques are commonly deployed to try and frustrate ROP attacks. Now, OpenBSD is experimenting with a new mitigation that makes it harder for attackers to make system calls, although some security researchers have expressed doubt that it will prove effective at stopping real-world attacks. In his announcement message, Theo de Raadt said that this work "makes some specific low-level attack methods unfeasable on OpenBSD, which will force the use of other methods."

Return-oriented programming is one of a family of techniques that use indirect jumps to call bits of code that already exist in a process's address space in an attacker-controlled order. The original attack involved overwriting the stack with carefully chosen addresses so that a function would "return" to a new location. Since the original discovery, other related attacks that use jumps through function pointers, signals, and other indirect jumps have been developed.

In December, De Raadt sent a patch to the OpenBSD mailing list expanding OpenBSD's restrictions on the locations from which a process can make system calls. A previous commit added code that declares a new ELF section which specifies where particular system calls are located within a program, so that the kernel can detect when a program tries to call a system call from the wrong location. Since OpenBSD does not have a stable system-call interface (instead suggesting that programs go through the C library for a stable interface), the new sections will not need to be explicitly added to most binary programs. Now that patch has been merged, finishing a process which De Raadt said has taken five years.

Background

OpenBSD already restricted where programs can make system calls. In 2019, De Raadt added code to ensure that system calls could only be made from four locations: in the text of a static binary (that links the C library statically, and so doesn't have a separate section at runtime), in the signal trampoline (where a system call is required to return from a signal handler), in the text of ld.so (the dynamic linker, that needs to make system calls to set up the process's address space), and in the text of libc.so (where the OpenBSD system call stubs live).

This code relied on a new msyscall() system call to let the linker inform the kernel of where libc.so (the shared object for the system's C library) is mapped within the address space. In February 2023, De Raadt extended these protections with the introduction of pinsyscall(), which is used to say where in the binary a process is allowed to call execve(). Both of these system calls can only be invoked once by a given process, which is done by the dynamic linker.

    int msyscall(void *addr, size_t len);
    int pinsyscall(int syscall, void *start, size_t len);

Despite its more generic signature, pinsyscall() only supports specifying a location for execve() calls.

These mechanisms were also intended to make it harder for ROP attacks to gain a foothold. Requiring the address from which a system call is made to be within the msyscall() block ensures that an attack cannot make use of any ROP gadgets ending in a system call that may be present outside of the specially designated areas. Requiring that execve() calls come from one specific location is also intended to make it harder for an attack to figure out where to make a call from, since calling execve() to execute an attacker-controlled program is a common stepping stone in an attack.

The new work obsoletes both of these mechanisms, with De Raadt suggesting that once the new code had been adopted for a release or two, they could "turn msyscall() and the less powerful pinsyscall(2) into NOPs, and eventually remove them".

The patch

The new work adds a new pinsyscalls() system call:

    int pinsyscalls(void *start, size_t len, u_int *pintable, int npins);

pinsyscalls() sends a "pintable" specifying from where in the process's address space each possible system call is expected to be made. The kernel uses the information in the table to check on entry to the kernel whether it is being invoked from a specified location. This check is intended to prevent a ROP attack from setting the system call number and then jumping directly to a system-call CPU instruction corresponding to a different system call. For example, an attack wishing to make an execve() call would need to jump to the specific instruction in the C library that has been added to the allowlist for that call, not another stub or the middle of an unrelated instruction which simply happens to decode as a system call instruction.

When setting up a new process, the dynamic linker uses pinsyscalls() to inform the kernel about from where the process expects to make system calls. The new work adds an "openbsd.syscalls" ELF section to select programs: ld.so, libc.so, and libc.a. The new ELF section contains an array of program offsets and system call numbers, indicating which system call is expected at each location. This section is read by the dynamic linker and used to provide a suitable pintable to the kernel. Programs that link against the C library can therefore benefit from the new protection immediately, without requiring changes to their build process. Unlike Linux, OpenBSD develops the kernel and user-space together, so the user-space components of this work are already in place.

Security researchers have expressed doubt about how useful this check is at preventing compromises. One researcher, "stein", noted that "an attacker able to perform ROP can simply use the libc stub, instead of issuing raw syscalls", referring to the possibility of an attack jumping directly to the instruction which has been added to the allowlist for a particular system call. Another researcher, Saagar Jha, commented on the new patch, saying "if you take this to its logical conclusion it's just 'applications should specify which system calls they use' which is literally just what pledge does and it’s enforced by the kernel and not in some weird ad-hoc IP to syscall number lookup scheme".

OpenBSD does have existing mitigations designed to make it difficult for ROP attacks to determine the location of the C library system call stubs. One such protection is address-space layout randomization (ASLR), which has been standard in many operating systems for a long time. OpenBSD takes randomization of a program's address space a step farther by also re-linking the sections of the C library in a random order on boot, meaning that an attack must determine not only the offset of the C library in memory, but also the offset of the specific code to which the attack wishes to jump within the library. Unfortunately, dynamically linked programs have to have this information in the symbol relocation table in order to allow for calls to the shared object. Therefore attacks that can construct a way to read memory can frequently leak enough offset information to circumvent these protections. De Raadt gave a talk (with slides) about ROP mitigations in OpenBSD at CanSecWest in 2023, including several other protections designed to make leaking information about the contents of a program harder.

Unlike pledge(), this patch has the advantage of securing an application even if the developer does not make any special effort. However, this protection is most useful to programs that statically link OpenBSD's C library; programs that use dynamic linking will still have all of the system calls used by the C library in their address space. pledge() also permits dropping unnecessary permissions after startup, which allows applications to use a more restrictive set of permissions than a static defense like pinsyscalls() can permit.

This work was difficult to bring to completion. One of the largest obstacles were programs written in Go. In his announcement that the new work had been merged, De Raadt said: "The direct-syscalls-inside-the-binary model used by go (and only go, noone else in the history of the unix software does this) provided the biggest resistance against this effort". He thanked Joel Sing specifically for his work to make the Go ecosystem compatible with the changes.

Since Linux permits programs to make system calls directly, without going through a wrapper from a blessed C library, and is unlikely to change this policy, additional steps would be needed to incorporate a similar mechanism there. Some Linux programs make system calls directly in order to avoid depending on a specific C library, but others make system calls directly in order to use new features which have not yet been wrapped by the system's C library; OpenBSD doesn't have this problem since its C library and kernel are developed in lockstep.

Conclusion

OpenBSD has a long history of adding novel mitigations, some of which are adopted by other projects and some of which are not. This work seems unlikely to be adopted elsewhere, given the doubts around the practical benefit and the costs of adding additional complexity to how system calls are performed. This work does add another barrier to constructing a ROP attack on OpenBSD, however, and seems especially beneficial for statically linked programs that use only a few system calls and have not yet made use of pledge.

Comments (2 posted)

Looking ahead to Emacs 30

EmacsConf 2023 was, like its recent predecessors, an online conference with lots of talks about various aspects of the Emacs editor—though, of course, it is way more than just an editor. Last year's edition was held in early December. One of the talks that looked interesting was on Emacs development, which was given live by John Wiegley. In it, he briefly described some of the biggest features coming in Emacs 30, which is the next major version coming for the tool.

Talk

Wiegley's talk is something of a yearly tradition at EmacsConf, apparently. He noted that he is not an Emacs maintainer, so he gathered the information for the presentation by talking for an hour with Eli Zaretskii, who is a longtime maintainer. Zaretskii told him that Emacs 29, released in July 2023, had lots of new features; some of those were "quite radical", Wiegley said, but it has been successful and well-received. The Emacs 29.2 release was imminent at the time of the talk; it came out in mid-January. The plan is to make a new release branch for Emacs 30 sometime fairly shortly after that point, he said.

Emacs 30 looks like it will "be a little less interesting that Emacs 29 was" because there will likely be fewer big changes going in, but that does not mean there is nothing planned. To start with, it will have Android support, which is a feature that LWN tried out last year. "If you have ever wanted to have native Emacs on a tablet", which is something Wiegley has "always wanted", it will be available in the new release. There is also much better touchscreen support coming in Emacs 30 for both tablets and laptops.

Emacs "Grand Unified Debugger" (GUD) mode is getting support for the LLDB debugger that is used with the LLVM compiler suite. It will be particularly useful on macOS and is another feature he is personally looking forward to. Emacs perl-mode is being deprecated in favor of C Perl mode because the project does not want to support two Perl major modes with its "meager resources".

One of the headline features for Emacs 29 was optional support for the Tree-sitter parsing library that brings better and faster syntax recognition and highlighting, code indentation, and so on. Tree-sitter will generate parsers for languages that are specified using JavaScript to describe the grammar; the output is a concrete syntax tree that can be used by other tools. Emacs 29 shipped with more than 20 new major modes for various programming (and other) languages, which can be used when Emacs is built with Tree-sitter. Many different programming languages (C, C++, Rust, Python, Go, Java, Ruby, ...), configuration languages (YAML, TOML, JSON), and other language types (Bash, CSS, CMake, Dockerfile, ...) got new major modes (e.g. python-ts-mode) as part of the new feature.

Tree-sitter uses the grammar libraries, which must be built from the JavaScript grammar and be present on the local system, in order to parse the structure of the language. The Tree-sitter modes avoid using Emacs Lisp and regular expressions for syntax recognition as is done now, Wiegley said. Over time, more and more languages will receive Tree-sitter-based major modes. For Emacs 30, three new Tree-sitter modes have been added: Lua, Elixir, and HTML.

The last new feature that he mentioned is better byte-compiler warnings for "many more questionable constructs". These include things like empty macro bodies and missing expressions in places where they are expected; "just silly stuff that might litter the code". The new warnings will "help you clean up the code and get rid of those potential sites of error". There is a long list of new warnings in the NEWS file for Emacs 30.

Wiegley also noted that Zaretskii had asked him to mention that Stefan Kangas has been added as an Emacs maintainer. Kangas also gave a talk at EmacsConf this year.

As part of a Q&A session after the talk, Wiegley discussed his favorite recent features, with native compilation being near the top of the list. He uses a lot of different modes within his Emacs buffers, including Org Mode, Gnus, and Eshell so native compilation has made a big difference. It has "brought the user experience much closer to a modern app than some of the lagging and slowness that I might have experienced in the past". He also touched on topics like using machine-learning tools such as ChatGPT (including for writing Emacs Lisp code), support for Emacs on macOS (he is a longtime user and has not encountered any major problems), Rosetta Code, and more.

Other features

Looking at the NEWS file shows lots of other bits that are coming in Emacs 30. For example, native compilation will now build automatically with Emacs if libgccjit is available on the build system. On Linux systems, Emacs will now be the default application for handling org-protocol URIs; the emacsclient.desktop file has been changed to register as the handler of the protocol. X selection requests (for selecting and yanking, also known as copying and pasting) are "now handled much faster and asynchronously", so they should work better over slow networks.

There are entries for new and changed features for many different parts of Emacs, including Dired, Grep, Network Security Manager, Version Control, Compilation Mode, and more. The NEWS file is nearly 2,000 lines of text, in an Org-format document naturally, which should give some sense for the amount of changes; another data point of interest is that the file for Emacs 29, which Zaretskii and Wiegley said was a much larger release, is more than double that length.

One change that was made was partially unmade after a lengthy debate about changing the existing behavior of the jump-to-register and point-to-register commands. A change was proposed in October 2023 for the behavior of the register-name prompt. At the time, the participants in the bug thread settled on making the new behavior the default, but once it got merged into the main branch, complaints arose. As shown in the NEWS file, the complaints were addressed by making the default be to preserve the existing behavior, while allowing others to customize the prompt behavior.

So the Emacs 30 release will be a hodge-podge of features, bug fixes, user options, and more throughout the huge footprint of the tool. The release cadence seems to have picked up over the last few major releases, which can be seen on the release history page, so we may well see Emacs 30 before the end of the year.

Another thing that seems likely in that time span is another EmacsConf, which seem to land near the end of each calendar year. The report from the 2023 edition describes a lively conference with lots of interesting talks, all of which are available for viewing in a variety of formats. The report also details the free-software tools that were used to organize and host the virtual conference as well as the process improvements tried and the finances for the conference, all of which will be of interest to other conference organizers.

As noted five months ago, I am a relative newcomer to really trying to get the most out of Emacs, though I have used it for a long time. Of course, that simply means that there are lots of opportunities to learn new things—too many in truth. The EmacsConf talks look like they provide even more of such opportunities, should time allow. There certainly does not seem to be much of a bottom to this particular rabbit hole.

Comments (10 posted)

Better handling of integer wraparound in the kernel

While the mathematical realm of numbers is infinite, computers are only able to represent a finite subset of them. That can lead to problems when arithmetic operations would create numbers that the computer is unable to store as the intended type. This condition, called "overflow" or "wraparound" depending on the context, can be the source of bugs, including unpleasant security vulnerabilities, so it is worth avoiding. This patch series from Kees Cook is intended to improve the kernel's handling of these situations, but it is running into a bit of resistance.

Cook starts by clarifying the definitions of two related terms:

• Overflow happens when a signed or pointer value exceeds the range of the variable into which it is stored.
• Wraparound happens, instead, when an unsigned integer value exceeds the range that its underlying storage can represent.

This distinction is important. Both overflow and wraparound can create surprises for a developer who is not expecting that situation. But overflow is considered to be undefined behavior in C, while wraparound is defined. As a result, overflow brings the possibility of a different kind of surprise: since it is undefined behavior, compilers can feel free to delete code that handles overflow or apply other unwelcome optimizations. To avoid this outcome, the kernel is built with the -fno-strict-overflow option, which essentially turns (undefined) overflow conditions into (defined) wraparound conditions.

So, in a strict sense, overflows do not happen in the kernel, but wraparounds do. If a wraparound is intended, as is often the case in the kernel (see ip_idents_reserve(), for example), then all is fine. If the developer is not expecting wraparound, though, the results will not be so good. As a result, there is value in using tooling to point out cases where wraparound may happen — but only the cases where it is not intended. A wraparound detector that creates a lot of false-positive noise will not be welcomed by developers.

In the past, the tooling, in the form of the undefined behavior sanitizer (UBSAN) and the GCC -fsanitize=undefined option, has indeed generated false-positive warnings. As a result, this checking was disabled for the 5.12 release in 2021 and has remained that way ever since. Cook is now trying to re-enable UBSAN's wraparound checking and make it useful; the result was an 82-part patch set making changes all over the kernel.

The key to making this checker useful is to prevent it from issuing warnings in cases where wraparound is intended. One way to do that is to explicitly annotate functions (generally of the small, inline variety) that are expected to perform operations that might wrap around and that handle that situation properly. The __signed_wrap and __unsigned_wrap annotations have been duly added for this purpose; they work by disabling the checking of potential wraparound conditions in the marked function.

The most common place where intentional wraparound is seen, though, is in code that is intended to avoid just that behavior. Consider this code in the implementation of the remap_file_pages() system call:

    /* Does pgoff wrap? */
    if (pgoff + (size >> PAGE_SHIFT) < pgoff)
	return ret;

Normally, the sum of two unsigned values will be greater than (or equal to) both of those values. Should the operation wrap around, though, the resulting value will be less than either of the addends. As a result, wraparound can be reliably detected with a test like the above. It is worth noting, though, that this test detects wraparound by causing it to happen; wraparound is an expected result that is properly handled.

To a naive wraparound detector, though, that code looks like just the sort of thing it is supposed to issue warnings about. The resulting noise makes such a detector useless in general, so something needs to be done. In this case, Cook adds a pair of macros to explicitly annotate this type of code:

    add_would_overflow(a, b)
    add_wrap(a, b)

The first returns a boolean value indicating whether the sum of the two addends would wrap around, while the second returns that sum, which may have wrapped around. These macros are built on the kernel's existing check_add_overflow() macro which, in turn, uses the compiler's __builtin_add_overflow() intrinsic function. Using these, the above remap_file_pages() test is rewritten as:

    /* Does pgoff wrap? */
    if (add_would_overflow(pgoff, (size >> PAGE_SHIFT)))
 	return ret;

This code now clearly does not risk an unwanted wraparound, and so no longer triggers a warning. The patch set rewrites a large number of these tests throughout the kernel. Along the way, Cook also had to enhance check_add_overflow() to handle pointer arithmetic so that pointer additions can be easily checked as well.

With all of this work in place, it is possible to turn on wraparound checking in UBSAN again. Eventually, the warnings generated should be accurate enough that it can be used to detect code that is not written with wraparound in mind. First, though, this work has to find its way into the mainline. In the best of times, a series that changes over 100 files across the kernel tree is going to be challenging to merge, though Cook has gotten fairly good at that task.

A more difficult challenge may be the opposition expressed by Linus Torvalds. He complained that the changelogs do not properly describe the changes that are being made, that the new annotations cause the compiler to generate less-efficient code, and that the tooling should recognize wraparound tests in the above form without the need for explicit annotation: "if there's some unsigned wraparound checker that doesn't understand this traditional way of doing overflow checking, that piece of crap needs fixing". He added some conditions for merging these changes.

Cook answered that he would rewrite the changelogs, which was one of the things Torvalds demanded. Another one of those demands — "fix the so-called 'sanitizer'" — might prove to be a bit more challenging, since it will require work on the compiler side. The advantage of such a fix is clear; it would remove the need for hundreds of explicit annotations in the kernel. But that would come at the cost of delaying this work and dealing with the bugs that enter the kernel in the meantime.

The history of the hardening work in the kernel suggests that these little obstacles will indeed be overcome in time and that the kernel will eventually be free of wraparound bugs (or close to that goal, anyway). Of course, as Kent Overstreet took pains to point out, this work would not be necessary if the kernel would just take the minor step of switching to Rust. Cook answered that any such change is not happening soon, so he will continue his work of "removing as many C foot-guns as possible". As this work is wrapped up, the result should be a more stable and secure kernel for all of us.

Comments (96 posted)

The things nobody wants to pay for

The free-software community has managed to build a body of software that is worth, by most estimates, many billions of dollars; all of this code is freely available to anybody who wants to use or modify it. It is an unparalleled example of independent actors working cooperatively on a common resource. Free software is certainly a success story, but all is not perfect. One of the community's greatest strengths — convincing companies to contribute to this common resource — is also part of one of its biggest weaknesses.

The GNU project, as described by Richard Stallman in the 1985 GNU Manifesto, looked hopelessly ambitious at the time. To many of us, it seemed that only large companies could build operating systems, and that a group of volunteers would never be able to aspire to such a goal. The volunteers got surprisingly far, to the point that, less than ten years after the GNU Manifesto was published, running a system on only free software (or something close to that) was possible. It was an impressive achievement.

Even then, though, that software not entirely devoid of corporate contributions. The X Window System, for example, was the product of a commercial consortium that predated Linux. The development of GCC was pushed forward by companies like Cygnus Computing. When the Linux kernel arrived on the scene, there was indeed a substantial body of GNU software that could run on it, but there was a nontrivial amount of company-contributed software as well.

Linux in the 1990s still lagged far behind the proprietary Unix systems in many ways, even though it was better in others. That began to change with the arrival of corporate funding, which supercharged development on Linux and on free software in general. Without it, we would not have the system we take for granted today. Companies, working in their own interest, have built up our body of free software hugely; it is almost as if this capitalism thing actually works.

The problem, of course, is that these companies have a tendency to interpret their own self-interest rather narrowly. They will happily pay to develop a driver for a hardware product, but they are less interested in supporting the maintainership and review that are needed to integrate that driver, the development of the subsystem into which the driver fits, or the support the driver will need over the years. Companies pay for thousands of developers to work on the kernel, but none pays for a single technical writer to create documentation. Work that is not seen as contributing to short-term revenue tends not to get much attention in the corporate world.

There are, needless to say, numerous other pathologies exhibited by corporations in the open-source community. These include license violations, free-riding, throwing code over the wall, and more. Projects that are controlled by a single company are often particularly problematic. These difficulties have often been covered elsewhere; today the focus is on the failure to support parts of the community that we all depend on.

Consider some recent examples of how this behavior affects the development and user communities.

In a recent linux-kernel discussion, Kent Overstreet complained about the lack of testing infrastructure for the kernel, calling it a failure of leadership. Greg Kroah-Hartman, instead, said that the problem lies elsewhere: "No, they fall into the 'no company wants to pay someone to do the work' category, so it doesn't get done." He pointed out that there is not much the leadership (such as it is in the kernel community) can do in a situation like this. So testing infrastructure for the kernel tends to languish.

Also recently, Konstantin Ryabitsev, the keeper of kernel.org and the author of the b4 tool, apologized for a lack of progress with b4, observing that he has not been able to get the go-ahead to put time into that work. He later clarified that post: "it just means that we haven't properly reallocated resources to allow me to prioritize tooling work". He has a lot of demands on his time, and b4 has not been able to rise to the top of the list.

This matters; anybody who has been watching kernel development over the last few years has seen that b4 has transformed the maintainer's task in many ways. It must have paid back the effort that went into its development many times over. The kernel community has never put the effort into tooling that it needs; the advent of b4 was a change in that pattern and one that we would all like to see continue.

One can easily criticize the Linux Foundation (LF) for not supporting this work at the level that we would like. But the fact of the matter is that the LF is about the only entity that has supported this work at all; without that support, we would not have b4. So, while encouraging the LF to more strongly support b4 is a good thing to do, it is also worth asking why no other company has seen fit to support that work, despite the fact that they have all benefited from it. The incentives that drive companies (and their managers) simply leave little room for this kind of work, even though the benefits from it would be real and immediate.

In a different part of our community, a discussion on the upcoming openSUSE Leap 16 distribution has led to fears that this version of the distribution will not support packages like KDE or Thunderbird — fears that are arguably getting ahead of the game, since they have not yet been confirmed by SUSE. Such a move would seem similar to Red Hat's decision to drop LibreOffice from Red Hat Enterprise Linux. Support from companies for the Linux desktop, it seems, is threatened.

Once again, it comes down to corporate priority setting, though with a slightly different driving force. Companies like Red Hat and SUSE (and a number of others) have supported Linux desktop development heavily for many years. But that investment is not seen as paying off. As Neal Gompa put it in the Leap discussion:

Have you ever wondered why KDE was deprecated in RHEL? It wasn't just because KDE Plasma 5 was so big that they couldn't make the jump for RHEL 8, it was also because people were not paying for RHEL for the desktop, so they had no budget for more people. It happened again with the layoffs at Red Hat last year. SUSE is no different. And Ubuntu? Canonical already did their big layoff in 2017 where they laid off hundreds of staff related to the desktop.

In many parts of the system, the work that leads to some company making money just happens to benefit all of us, even if we are not directly paying for it. But, seemingly, that path toward making money is more elusive in the desktop realm; those of us who are using desktop Linux are not generally paying for the privilege, and neither is anybody else. So the resources going into desktop development are reduced, and Linux desktop users will feel the effects of that.

Then, consider this ongoing discussion in the Python community about funding for the PyPI repository. As "fungi" put it:

Companies are unlikely to fund open source communities that are critical to their own business because “someone else will do it,” so it ends up falling on a handful of volunteers and people donating their own time after hours because they’re already being paid to work full time on other stuff.

There are few projects in our community that do not contain this kind of neglected area.

Solutions to these problems are not easy to come by. Criticizing companies for a failure to support the ecosystem they depend on can have results, but only to a point. Organizations like the LF can organize resources toward the solution of common problems, but they have to please the same companies that are paying their bills in the end. Governments can help to fund areas that the market has passed over; that path gets harder in the absence of a functioning government, which is the situation to varying degrees in many parts of the world at the moment.

If we cannot find a solution, we are likely to be forced back to our roots, depending on volunteers to do the work in areas that companies decline to fund. That can be successful, but often at a high cost to the people who are doing that work. Depending on volunteers is not an inclusive approach; there are a lot of people who do not have the luxury of giving many hours of their time to this kind of project. Progress in that world will be slow.

The community that has accomplished so much over the last few decades should be capable of doing better than that. We have solved many problems getting to the point we are at now — problems that few thought we would be able to overcome; we should be able to find a way around these difficulties as well. It will be interesting to see what we come up with.

Comments (77 posted)

Defining the Rust 2024 edition

In December, the Rust project released a call for proposals for inclusion in the 2024 edition. Rust handles backward incompatible changes by using Editions, which permit projects to specify a single stable edition for their code and allow libraries written in different editions to be linked together. Proposals for Rust 2024 are now in, and have until the end of February to be debated and decided on. Once the proposals are accepted, they have until May to be implemented in time for the 2024 edition to be released in the second half of the year.

Accepted proposals

Some proposals have already been accepted, the largest of which is a change to how functions (and methods) that use "Return Position impl Types" (sometimes also called "-> impl Trait") capture lifetime information. While this proposal is slated for inclusion in Rust 2024, it does leave certain questions open, and may end up not being included if those questions cannot be resolved in time. In Rust, one can write functions that return opaque types that can only be used by invoking methods of traits they implement. LWN covered previous work to improve the consistency of this feature late last year. An example of a function that uses Return Position impl Types is the following function definition that returns some type that implements the Foo trait, without committing itself to a specific type:

    fn foo<'a, T>(x: &'a T) -> impl Foo { ... }

One wart with this functionality is the rule for how the inferred hidden type of the function incorporates other types from its surroundings. In the given example, the type that foo() returns can reference the type T. In Rust 2021, however, it cannot reference the lifetime 'a. This asymmetry requires users to introduce awkward contortions in order to correctly write functions that capture references and return them as part of an opaque type. This is especially troublesome in the case of trying to convert an asynchronous function to use the impl Trait syntax, because asynchronous functions can capture lifetime information in this way.

The proposal corrects this asymmetry in the 2024 edition by permitting lifetime information and types to be captured using the same rules. The discussion about this feature proposal included some concerns about whether this could lead to "overcapturing", where a lifetime parameter that the user does not intend to be referenced is captured anyway, restricting how the returned value can be used. The proposal ends up leaving this question open, saying that these changes to the lifetime-capture rules should not be included in the final set of revisions for Rust 2024 unless the community can find "some solution for precise capturing that will allow all code that is allowed under Rust 2021 to be expressed, in some cases with syntactic changes, in Rust 2024."

Two other accepted proposals have to deal with how Cargo, Rust's package manager, handles dependencies. One proposal changes how packages specify optional dependencies. Currently, every optional dependency of a package implicitly creates a Cargo "feature" — Rust's solution for conditional compilation — with the same name. The proposal says that this makes it "easy for crate authors to use the wrong syntax and be met with errors" and leads to "confusing choices when cargo add lists features that look the same", because the names of optional dependencies appear alongside features that are defined in the crate. In Rust 2024, there will be no externally-visible feature with the same name as the dependency by default. Instead, enabling an optional dependency will be done using a feature named "dep:package-name" instead of "package-name".

The other proposal that impacts dependency handling is designed to make it easier for Rust to automatically detect and warn about changes that may break semantic versioning compatibility guarantees. In current editions of Rust, when one crate depends on another, the first crate can expose types from the second crate as part of its API. The following example, taken from the proposal, shows code that re-exposes types from serde and serde_json:

    #[derive(Clone, Debug, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
    pub struct Diagnostic {
        code: String,
        message: String,
        file: std::path::PathBuf,
        span: std::ops::Range<usize>,
    }

    impl std::str::FromStr for Diagnostic {
        type Err = serde_json::Error;

        fn from_str(s: &str) -> Result<Self, Self::Err> {
            serde_json::from_str(s)
        }
    }

If serde_json were to change the definition of serde_json::Error, then the return type of Diagnostic::from_str() would change, which could potentially be a breaking change. The proposal seeks to remedy this by turning an existing warning (lints.rust.exported_private_dependencies) into an error in the 2024 edition, and giving crates a way to mark dependencies that are deliberately exposed by adding a public flag to Cargo dependency declarations.

The last accepted proposal at the time of writing stands out by not modifying the language itself, but the policies around how future editions will maintain backward compatibility. Rust permits users to define syntactic macros (as opposed to procedural macros) that pattern-match given fragments of Rust syntax, and produce modified Rust code. When Rust adds new pieces of syntax, this impacts the meaning of existing macro definitions. This proposal seeks to clarify the policy around when and how the syntactic macro pattern-matching rules are updated to match new syntax, saying that when the syntax of Rust is changed in such a way that the patterns in syntactic macros no longer align with the actual grammar of the language that the project should: add a new pattern which preserves the behavior of the existing pattern; in the next edition, change the behavior of the original pattern to match the underlying grammar of the release of Rust corresponding to the first release of that edition; and have cargo fix replace all instances of the original pattern with the new pattern which preserves the old behavior.

Proposals under discussion

Currently, there is one unaccepted proposal that is in its final comment period before being approved, a proposal to add support to Cargo for respecting the minimum supported Rust version of a crate. Crates can already declare that they need a certain version of Rust using the rust-version field in the Cargo manifest. This proposal adds a new Cargo resolver that prefers versions of dependencies that have compatible rust-version fields.

Another proposal that has not yet been accepted, but that seems likely to be accepted because it has been part of Rust's overarching vision for some time, is a proposal that would reserve the gen keyword. The new keyword would be analogous to the async keyword, but permit the definition of generators — recently renamed in the internal documentation to coroutines to match use of the term outside of the Rust community — instead of asynchronous functions. It is already possible to write coroutines in nightly Rust, and they are part of how the compiler implements asynchronous functions, but they are not yet stabilized. The proposal doesn't suggest getting them stabilized by May, but only reserving the syntax so that automatic upgrades from Rust 2021 to Rust 2024 can update existing programs to avoid clashing with coroutines when they are introduced.

Migrating code from one edition to the next is always supposed to be possible using cargo fix. For newly reserved keywords, this means rewriting programs which use those keywords as identifiers. For example, "let gen = ...;" would be rewritten to "let r#gen = ...;". This uses a seldom-seen feature of Rust known as raw identifiers, which permit using otherwise reserved words as variable or function names by prefixing them with "r#".

The proposal prompted a lot of debate about the exact place that this keyword would have in the Rust parser, and whether it should be changed to reflect the new term coroutine, eventually prompting Oli Scherer — the contributor who submitted the proposal — to declare: "This RFC is on hiatus until we have an implementation and will then mirror the implementation". Work on an implementation is ongoing.

Another proposal intended to make writing asynchronous code slightly smoother is adding the Future and IntoFuture traits to the prelude — the section of the standard library available without explicitly importing a library. Alice Cecile expressed doubts that this was as useful as some of the existing functionality in the prelude, saying: "Generally I find that traits are most useful in preludes when you want to implicitly use their methods". She observed that the only method that this proposal would permit the use of is poll(). The proposal remains open for debate.

A proposal that hasn't seen much discussion since November is a suggestion to disallow directly casting function pointers to integers. Several supporters of the proposal noted that there are architectures where pointers to functions are different sizes than pointers to data, and expecting a usize value to contain a function pointer is incorrect. Others were concerned that changing this behavior would be making users jump through extra hoops for little benefit.

The last proposal currently under discussion would change how Rust represents ranges by adding new range types that would implement Copy, allowing them to be duplicated without boilerplate. The current range types do not implement Copy because of how this would interact with their use as iterators. The discussion of this proposal focused on how the introduction of new types might complicate the use of libraries that rely on the old types, for example by using the old types as indexes into collections. Peter Jaszkowiak, the contributor working on this proposal, did not believe this would be a significant problem, but said that he would adapt the proposal to include convenience features to ameliorate this if the language team thought that was warranted.

Conclusion

People who have been paying attention to Rust development since 2021 may find this list of changes for the 2024 edition a bit sparse, since it includes little mention of improvements to asynchronous code, to the standard library, or any of the hundreds of other small improvements Rust contributors have made in that time. These improvements are all already available in the 2021 edition. In Rust, only backward incompatible changes need to wait for the next edition. Therefore, the release of the 2024 edition heralds two things: the 2021 edition becoming more stable — in the sense that new syntax will now go in the 2024 edition — and the chance to fix various small issues that nonetheless hinder the continued evolution of the language, without breaking existing users.

Comments (191 posted)