LWN.net Weekly Edition for March 7, 2024

Welcome to the LWN.net Weekly Edition for March 7, 2024

This edition contains the following feature content:

• A sandbox mode for the kernel: a new proposal for increased isolation in the kernel seems unlikely to be accepted.
• Formalizing policy zones for memory: a patch set adds new memory zones to provide better support for transparent huge pages (THP).
• MySQL and MariaDB changes coming in Fedora 40: Fedora changes how it packages MySQL and MariaDB.
• An alternate pattern-matching conditional for Elisp: Emacs gains an alternative to pcase.
• Making multiple interpreters available to Python code: Python may soon include a new option for parallelism.
• Not so quickly extending QUIC: the QUIC working group considers four protocol extensions.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

A sandbox mode for the kernel

The Linux kernel follows a monolithic design, and that brings a well-known problem: all code in the kernel has access to the entirety of the kernel's address space. As a result, a bug in (for example) an obscure driver may well be exploitable to wreak havoc on core-kernel data structures. Various attempts have been made over the years to increase the degree of isolation within the kernel. The latest of these, "SandBox Mode" proposed by Petr Tesařík, makes it possible for the kernel to run some limited code safely, but it has encountered a bit of a chilly reception.

Sandbox mode

The intent behind this new mode is to allow the kernel to run a function in a way that it cannot affect the rest of the kernel. In its simplest form, sandbox mode is used by defining a function to be run in this isolated mode:

    #include <linux/sbm.h>

    static SBM_DEFINE_FUNC(untrusted_func, void *input_data, void *output_data);

That function would then be invoked with a sequence like:

    struct sbm sbm;

    sbm_init(&sbm);
    result = sbm_call(&sbm, untrusted_func, SBM_COPY_IN(&sbm, input_buffer, in_size),
			                    SBM_COPY_OUT(&sbm, output_buffer, out_size));

This code will result in a call to untrusted_func(). The input and output buffers will be allocated, and the input data copied, before that function is called with pointers to the new buffers. On a successful return, output data will be copied back, and sbm_call() will return the value returned by untrusted_func().

In the absence of architecture-specific support, that is about all that sandbox mode does; the associated documentation rightly describes this as "weak isolation". It might be enough to trap a simple overflow of the input or output buffers, but it still does not protect the kernel from any stray accesses that go further afield.

In a separate series, Tesařík provided a set of x86-64 architecture hooks that enhance the sandbox to provide stronger isolation. Specifically:

• The sandboxed function will be run with a separate set of page tables that limit its address space to the relevant code, the input buffer (mapped read-only), and the output buffer. As a result, the function will have no access to any other memory in the system. This change has some far-ranging implications; for example, it must be undone if an interrupt arrives so that the interrupt handler can run within the kernel's address space.
• The CPU is put into user mode, so that it cannot execute any privileged instructions; the function runs as if it were a user-space process.
• A separate kernel stack is allocated and the function is called on that stack so that it has no access to the normal kernel stack. There is also a separate exception stack that is used while sandbox mode is in effect.
• Any sort of CPU fault causes the immediate termination of the sandbox and an error return to the caller.

At this point, according to the documentation, sandbox mode provides "strong isolation" that should suffice to prevent the sandboxed function from accessing the rest of the kernel.

In search of users

But for what purpose has this mode been created? The documentation says that sandbox mode exists for "parsing data from untrusted sources, especially if the parsing cannot be reasonably done by a user mode helper", but there was no actual user included with the patch series, so there was no way to see what an intended user looks like. That, naturally, led to questions. Andrew Morton remarked that the API seemed overly restrictive and wondered how it would be possible to get any real work done; he asked for an example to clarify the situation, a request that Greg Kroah-Hartman echoed.

Tesařík answered that the framework is "quite limited" in its current form, but that he intended to expand it over time. A bit later, he posted a PGP-key parser that would run within the sandbox mode as an example user, but that did little to increase acceptance of this work. As Dave Hansen pointed out, the kernel does not currently contain a parser for PGP keys, so the new series just raised the question of why that needs to be added too. Hansen said it would be far better to move some existing kernel functionality into the sandbox to show how it could be made safer.

The response to that request was yet another patch series moving the parsing of AppArmor profiles into a sandbox. Supporting this use case required making a number of changes to the sandbox mode itself, including a new "fixup" mechanism designed to make it possible to call specific kernel functions from within the sandbox. So, for example, if code within the sandbox needs to allocate memory, it can call kmalloc(). That call will result in a fault, which will result in the execution of a proxy version of kmalloc() that will restore the kernel's full address space for the duration of the call.

Hansen responded that the "fixup" mechanism looked like a maintenance problem: "Establishing and maintaining this proxy list will be painful. Folks will change the code to call something new and break this *constantly*." He concluded that sandbox mode did not seem like a good idea overall: "I don't see any viable way forward for this approach". He did not even comment on the need to add a special "__nosbm" marker to all functions that might land in the same page as one that has been marked for calls from within sandbox mode — an extra step that seems almost certain to go wrong at some point.

The obvious conclusion is that sandbox mode is unlikely to make it into the mainline in anything resembling its current form. But there is clear value in isolating some kinds of kernel code, if there were only an acceptable way in which it could be done. One possibility is to use BPF, which is intended to provide isolation; non-trivial BPF programs can be tricky to get past the verifier, though, and the fact that they are loaded from user space may make some security-oriented people nervous.

Another possibility might be the user-mode blob feature that was merged into the 4.18 kernel nearly six years ago. It was intended for a similar purpose — the parsing of firewall rules for the BPF-based "bpfilter" subsystem — but has never seen use in the mainline kernel. In response to a query about using this feature instead of a new sandbox mode, Roberto Sassu said that "security people don't feel confident" about using it. The main concern seems to be that, since user-mode blobs run in a separate, user-space process, they would be subject to manipulation by user space; sandbox mode, being fully contained within the kernel, should be better protected.

If complete isolation from user space is also a requirement for this work, then it may be that there are no viable solutions for Linux at this time. Hardening the kernel is a worthy goal, but it is just one of many that have to be traded off in the creation of a kernel that is both useful and maintainable in the real world. In the absence of a better implementation, it would appear that sandbox mode does not offer enough to justify the tradeoffs it would require.

Comments (24 posted)

Formalizing policy zones for memory

The kernel's memory-management subsystem is built on the concept of "zones", which were initially added to describe the physical characteristics of the memory pages contained within them. Over time, zones have taken on more of a policy-related role as well. With a patch set called THP allocator optimizations, Yu Zhao has set out to better define the role of policy-related zones on the path toward adding two more of them, with the ultimate purpose of improving the kernel's support for transparent huge pages (THPs).

The history zone

A bit of background might help to set the context for this patch series.

The earliest x86 systems that ran Linux talked to peripheral devices via the ISA bus, which was only able to address 16MB of memory, using 24-bit addresses. Even in those days, computers often had more memory than that, meaning that some memory could not be used by ISA devices for DMA I/O operations. That made life inconvenient for any device drivers that might happen to allocate a DMA buffer in memory inaccessible by the devices they were trying to control.

To avoid this problem, the kernel added the GFP_DMA allocation flag in 1.1.68, in November 1994. That flag didn't actually do anything until 1.1.69, and was not used until the Buslogic SCSI driver added support in 1.1.73. This flag asked the memory-management subsystem to allocate memory within the physical address range that was reachable by ISA devices, so that driver authors no longer needed to worry when allocating buffers.

At that time, GFP_DMA simply caused the allocator to skip over memory that was not suitably located. That worked, but life, as is its wont, became more complicated. The kernel, in the early days, could only manage a bit less than 1GB of physical memory; the 32-bit address space, as partitioned by the kernel, simply could not contain any more. That nearly 1GB, of course, ought to have been enough for anybody. But users can be rather strident when it comes to being able to use all of the memory they paid for, so kernel developers came under pressure to find a solution.

That solution was "high memory", which could be allocated to user-processes but could not be directly addressed by the kernel without an explicit (and temporary) mapping step. High memory was added to 2.3.23pre3 in late 1999, adding another type of memory that had to be taken into account; memory used by the kernel usually could not be placed there. To deal with these constraints, the concept of "memory types" was added to 2.3.23pre5. That concept was then formalized into "zones" in the 2.3.27 release in November 1999. At that time, there were three zones: ZONE_DMA, ZONE_NORMAL, and ZONE_HIGHMEM.

An important aspect of the zone system is that the zones were organized in an order that allowed for a straightforward fallback mechanism. Any allocation that could be served out of ZONE_HIGHMEM could also be satisfied from either of the other two zones, and ZONE_NORMAL allocations could fall back to ZONE_DMA. That property still holds in current kernels.

The 2.6.15 kernel, released in January 2006, saw the addition of ZONE_DMA32, wedged between ZONE_DMA and ZONE_NORMAL, to meet the needs of devices that could only handle 32-bit DMA addresses. In 2.6.23 (October 2007), ZONE_MOVABLE was added, and ZONE_DEVICE showed up in 4.3 (November 2015) to describe CPU-addressable memory installed on peripheral devices.

To an extent, all of these zones describe physical characteristics of the memory involved, mostly relating to addressability. ZONE_MOVABLE is a bit of an exception, though. It describes memory where, with luck, all of the contents can be moved elsewhere if need be. User-space pages, for example, can be migrated elsewhere in memory; the page-table entries will be updated accordingly, and user space will never know that anything changed. This zone was added to support hotplug memory that could be added to (or removed from) a system at any time. If memory is to be removed from a system, it is somewhat important to shift all of the contents of the memory elsewhere first.

The hotplug use case still exists, but ZONE_MOVABLE has become more of an expression of memory-management policy. If all of the movable pages are put in the same region of memory, it is relatively easy to reclaim some of that memory to, for example, create large, physically contiguous ranges. So now ZONE_MOVABLE is used as a way of separating movable and non-movable allocations, and provides important support for, among other things, the contiguous memory allocator. The important point here is that ZONE_MOVABLE can be placed anywhere in physical memory; it is not driven by the physical characteristics of that memory, especially on the large majority of systems where hotplugging is not in use.

In current kernels, the zone hierarchy is:

• ZONE_DMA
• ZONE_DMA32
• ZONE_NORMAL
• ZONE_HIGHMEM
• ZONE_MOVABLE
• ZONE_DEVICE

Note, though, that not all zones are present on all systems; current 64-bit systems, for example, have no need for ZONE_HIGHMEM.

Explicit policy zones

Zhao has come to the conclusion that more can be done with the idea of a memory zone as an aid to allocation policy. Specifically, he argues that there may be a use for a couple of other policy zones (which are often called "virtual zones" in the code itself). To that end, his patch series adds:

• ZONE_NOSPLIT, which is a zone where contiguous blocks of pages cannot be split below a given size. It exists to help the system maintain large blocks of memory (to use for transparent huge pages, among other things) without having to go through a continual process of compaction.
• ZONE_NOMERGE also has the minimum block-size property, but also disallows the merging of blocks of pages into bigger groups; as a result, it can only hold chunks of a single size.

These zones, which are placed after ZONE_MOVABLE, maintain the fallback hierarchy: a ZONE_NOMERGE allocation can also be satisfied from ZONE_NOSPLIT, or from one of the lower zones if that fails. Allocations from these zones must be movable, so a fallback to ZONE_MOVABLE is possible as well.

The idea behind these zones is to make the allocation of transparent huge pages as efficient as possible. They will prevent the splitting of huge pages, which will keep the kernel from having to go through the effort of reassembling them later. Much of the work that currently goes into compaction could become unnecessary.

In a sense, this work can be seen as a sort of compromise between those who would like to see Linux use a larger page size overall and those who worry about the associated internal-fragmentation cost. ZONE_NOMERGE creates something close to a second native page size for the kernel, making larger pages available in situations where they make sense while still keeping smaller pages available.

Internal fragmentation can still be a problem with transparent huge pages, though; a process may have such a page allocated, but only be using a small portion of the memory within it. Current kernels will try to respond to this situation by splitting huge pages back into base pages, allowing the unused parts to be reallocated elsewhere.

Splitting clearly cannot happen with pages located in (or above) ZONE_NOSPLIT; that is exactly the policy that zone exists to enforce. Instead, Zhao's patch set introduces the concept of "shattering" huge pages. If a page is shattered, its contents are migrated (copied) to smaller pages located in a suitable zone; once that process completes, the original huge page, which remains intact, can be allocated to another use. Shattering is more expensive than splitting; Zhao sees it as an appropriate cost for a process not properly using its memory; from the changelog: "In retail terms, the return of a purchase is charged with a restocking fee and the original goods can be resold".

Another claimed advantage of ZONE_NOMERGE is that it facilitates huge vmemmap optimization (HVO), which was covered here in 2020. In short, this trick allows the kernel to recover the memory used to hold page structures for many of the pages in a huge page. In a system where a lot of huge pages are in use, this optimization can save a significant amount of memory. In current kernels, HVO can only be used with the hugetlbfs mechanism, which is not transparent and is normally only used in specialized situations. ZONE_NOMERGE pages are organized in fixed blocks, though, like hugetlbfs pages, so it becomes easy to use HVO with them.

The patch set is in an early stage; among other things, it does not have any sort of benchmark results showing the advantages of this new machinery. Review comments from other developers are just beginning to come in; it is a significant chunk of work and will take some time to digest. It is likely to be a discussion topic at the Linux Storage, Filesystem, Memory-Management and BPF Summit in May. The new zones will thus probably not land in the kernel in the near future, but their advantages might prove compelling in the longer term.

Comments (7 posted)

MySQL and MariaDB changes coming in Fedora 40

The Fedora Project switched to MariaDB as the default implementation of MySQL in Fedora 19 in 2013. Once a drop-in replacement for MySQL, MariaDB has diverged enough that this is no longer the case—and, despite concerns about Oracle and optimism that MariaDB would supplant MySQL, the reality is that MySQL and MariaDB seem to be here to stay. With that in mind, Fedora developer Michal Schorm proposed that the project revise the way MySQL and MariaDB are packaged in Fedora starting with Fedora 40.

How MariaDB became "mysql" for Fedora

A quick(ish) recap of MySQL's history (and ownership), and how MariaDB came to be, may be in order. MySQL was created by MySQL AB, with its first release in May 1995. It was a key part of the ubiquitous open-source LAMP (Linux, Apache, MySQL, and Perl, PHP, or Python) stack which powered (and still powers) much of the web. Early on, open-source databases like MySQL and PostgreSQL were dismissed as not suitable for real workloads. But steady improvements and rapid adoption meant that major players were taking notice and showing concern about the impact that open-source databases would have on their business.

MySQL was so popular that Sun Microsystems decided to buy MySQL AB in 2008, for the staggering (at the time) sum of $1 billion in "total consideration". Sun's open-source stewardship, and relationship with the Linux community, were a bit uneven at times, but Sun's ownership of MySQL was generally met with a "wait and see" attitude among the community. However, Sun's stewardship of MySQL was never truly put to the test—before the dust could settle, Sun was up for sale just 14 months after it had completed its acquisition of MySQL AB. After a bid by IBM was withdrawn, Oracle announced its intent to buy Sun in April 2009.

While Sun was mostly given the benefit of the doubt, Oracle received a much cooler reception from the open-source community, and more regulatory scrutiny. The US waved the sale through without much fuss, but the European Union (EU) decided to conduct a longer investigation to look more closely at the database giant acquiring MySQL. During the approval period, MySQL creator Michael "Monty" Widenius, who had left Sun some time before, campaigned heavily (and unsuccessfully) to keep MySQL away from Oracle; others joined in his efforts. Despite Widenius arguing that the ability to fork MySQL wasn't enough to counter potential harms of Oracle owning MySQL, he had already created the MariaDB "branch", and a company to continue its development, due to his dissatisfaction with Sun's management of MySQL.

Objections aside, the acquisition was approved in January 2010. Oracle moved on to sue Google over its Android Java implementation (a suit Oracle lost), and divest itself of OpenOffice.org to the Apache Software Foundation (ASF). It also presided over the split of the Hudson community which led to the creation of the Jenkins project, and generally failed to win hearts and minds in the open-source community at large.

By the end of 2012, the Wikimedia Foundation (WMF) was making plans to switch to MariaDB. Not for performance reasons, but because WMF viewed MariaDB as "the best route to ensuring a truly open and well supported future for mysql derived database technology". By 2013, a number of Linux distributions had switched or were making plans to switch to MariaDB as a replacement for MySQL due to changes in MySQL's security reporting, bug tracking, and the lack of a full regression testing suite.

Some distributions stopped packaging MySQL entirely, and switched entirely to MariaDB. Others, like openSUSE and Fedora, decided to take the approach of making MariaDB the default while offering MySQL's community edition under a new package name. In Fedora, MySQL was now community-mysql, while mysql would install MariaDB instead. At the time, Fedora's Honza Horak cited two reasons to continue packaging MySQL—Fedora users who needed MySQL for "certification or similar reason" and a future when MariaDB ceases to be a drop-in replacement. That future, it seems, has arrived.

MariaDB and MySQL today

Despite fears that Oracle might stop MySQL development, or close its source, the company continues to provide a community edition of MySQL under the GPLv2. Some features are held back for its enterprise edition, including transparent data encryption and high-availability features, to name just two, but the open-source version remains popular. MariaDB has not replaced MySQL, but it has gained its own following and diverged substantially enough that it's no longer a "drop-in" replacement.

In his change proposal, Schorm noted that it no longer makes sense for MariaDB packages to claim to provide mysql, and hasn't made sense since the MariaDB 10.5 release in 2020. In the early days of the split, one could literally "drop in" MariaDB (or MySQL) and expect it to work with existing databases. However, that hasn't been possible since MySQL 8.0. Migrating between the two is still possible, but requires more steps and users may run into problems when working with some data types, such as JSON.

Over the years, the projects have diverged in the way that they implement encryption for data at rest, the global transaction IDs (GTIDs), replication, and much more. MariaDB has also implemented some features, such as support for Oracle PL/SQL, that are unlikely to appear in MySQL. According to MariaDB's documentation on applications supporting MariaDB: "Every project we know of which works with MySQL also works with MariaDB". However, once an application has been deployed, migrating between the two is increasingly difficult. MariaDB maintains a detailed list of specific versions of MySQL and MariaDB and the ways they are known to be incompatible.

MariaDB continues to have a loyal, albeit smaller, following. The DB-Engines popularity index for databases currently ranks MySQL as the second-most popular database (after Oracle database), with MariaDB coming in at 13, and number nine among relational database management systems.

Changes coming to Fedora 40

Schorm proposed a number of changes for Fedora 40, some that have been completed and some that have been deferred to a later release. One of the biggest changes, recognizing the MySQL community edition as mysql, has already happened. Prior to Fedora 40, users who wanted brand-name MySQL would install the community-mysql package. Schorm noted in the change proposal that the package name community-mysql was unique to Fedora, and that reverting the name to mysql was not only good for users, it would benefit him as well:

This change will save me, the maintainer, [a] noticeable amount of time and energy when cherry-picking commits and patches from Fedora to CentOS Stream and RHEL.

And the more energy I save downstream, the more I can put into Fedora and upstream.

Schorm has also dropped "cross-installation functionality" that allowed users to install the server for one database and the client for another. For example, users could install MariaDB server and the MySQL client or vice versa. Schorm said he introduced this "in the hope of delivering a handy enhancement for the users", but it turned out to have too many drawbacks. This is also dropped in Fedora 40, and should not impact other packages, such as the ODBC, Python, or Java connectors for MySQL.

MySQL and MariaDB tend to have multiple major versions available and in wide use at the same time. For example, MySQL has moved to a model of offering long-term support (LTS) and "innovation" releases concurrently, and MariaDB has several overlapping releases with five years of support. Prior to Fedora 39, these could be packaged as Fedora modules to offer multiple major versions simultaneously. Fedora retired modularity in Fedora 39, leaving fewer options for packaging multiple versions at the same time.

From Fedora 40 on, MySQL and MariaDB packages with multiple versions will have an unversioned metapackage name (e.g., mariadb) that points to a specific version, such as mariadb10.11. This allows Schorm (or other packagers) to package, say, an LTS release as mysql and offer an innovation release as well. If any additional versions are packaged for a release, they will have a versioned package name (e.g. mariadb10.5) that can be installed instead. With the end of modularity, users will only be able to select one version of a package to be installed. It will not be possible to have two versions of MySQL or MariaDB installed at the same time.

Schorm had planned to package MariaDB 10.5 and 10.11 for Fedora 40, but in an update to the change proposal he indicated that he was only able to package MariaDB 10.11 in this cycle. Thus the default (and only) version of MariaDB in Fedora 40 will be 10.11. He also had proposed packaging MySQL 8.1 for Fedora 40, but was unable to complete that in time either. The default version of MySQL will be MySQL 8.0.36, which is also the only version of MySQL available in Fedora 40.

i686 builds spared, for now

Even though Fedora dropped support for installation on i686 with Fedora 31 in 2019, the project continues to build packages for i686 for various use cases—mostly for users of Wine and Steam. He proposed dropping builds of MySQL server and MariaDB server for i686 in Fedora 40, but that will have to wait until Fedora 41 or later while he investigates whether any wanted i686 packages might depend on MariaDB or MySQL unexpectedly. He updated the change proposal to say that he still plans to do the work, but found it "too complex" to complete in the Fedora 40 cycle. Schorm said that he was unable to "get a correct recursive list of all dependencies of packages intended to be removed, for a given architecture".

Onward to Fedora 40

Users upgrading from Fedora 39 to Fedora 40 with MariaDB or MySQL packages should not encounter any problems with their installed MariaDB or MySQL packages. Users will now need to explicitly ask for MariaDB if that's what they really want, or applications that pull in mysql as a dependency will actually get MySQL. The change proposal drew little commentary, which suggests that MariaDB has few proponents who see a distinct benefit to using MariaDB over MySQL. It is, however, good enough and (for some users) has the advantage of not being from Oracle. It will be interesting to see how or if the naming changes affect database choice going forward—whether users express a strong preference for one database or the other, or simply take the default when looking for "mysql".

Comments (15 posted)

An alternate pattern-matching conditional for Elisp

One of the outcomes of the (extremely) lengthy discussion about using Common Lisp features in Emacs Lisp (Elisp), which we looked at back in November, was an effort to start removing some of those uses from Emacs. The rewrite of some of the Elisp in Emacs that uses the Common Lisp library (cl-lib) was started by Richard Stallman as a way to reduce the cognitive load needed for maintaining Emacs itself. Since then, he has broadened his efforts to simplify Elisp by adding a new pattern-matching conditional that would be a competitor to pcase, which is a longstanding macro that he finds overly complex.

Complexity

Back in mid-November, Stallman noted that he found the "little language" that pcase defines to be "so concise it is downright cryptic". He recognizes that trying to solve the same set of problems combining simpler Elisp constructs, such as cond and let, is "long-winded and cumbersome", but pcase has taken the desire for conciseness to an undesirable extreme. That imposes a cost on all Emacs developers who have to maintain code using pcase, he said, so he decided to adapt some pcase features in other constructs.

Predictably, that led to a long thread—standard fare for the emacs-devel mailing list—discussing whether there is a need for a pcase alternative, what one might look like, and more. Stallman started a new sub-thread to investigate his ideas for a new macro, cond*, which would provide a simpler pattern-matching construct that is still more concise than using "old-fashioned Lisp". His new macro is meant to combine the conditional cond form, which handles checking for multiple different values—something like switch constructs in other languages—with let, which temporarily binds values to variables within a limited scope. pcase and cond* are both designed to provide an ML-style pattern-matching conditional mechanism for Elisp.

A simple pcase example may give the general flavor of these constructs, but there is a great deal more that both can do, including pattern matching and pulling lists and other data structures apart, which is known as "destructuring". This example, taken from the pcase documentation, handles several different types (e.g. string, symbol) for a return code, producing an appropriate message for each:

    (pcase (get-return-code x)
      ;; string
      ((and (pred stringp) msg)
       (message "%s" msg))

      ;; symbol
      ('success       (message "Done!"))
      ('would-block   (message "Sorry, can't do it now"))
      ('read-only     (message "The schmilblick is read-only"))
      ('access-denied (message "You do not have the needed rights"))

      ;; default
      (code           (message "Unknown return code %S" code)))

Stallman characterized his approach as trying to "avoid the kludginess of pcase's bells-and-whistles-for-everything approach". Naturally, that led to further arguments in favor of pcase. For example, Michael Heerdegen said: "In my opinion `pcase' comes very close to the optimal solution for its task."

By mid-December, Stallman was asking about features needed for cond* "so that it is rare to encounter a pcase that can't be replaced cleanly". That conversation took place in another branch of the original pcase-replacement discussion, which makes it a little hard(er) to follow. In that part of the thread, others were working with Stallman on cond*; once again, there were numerous helpful suggestions and clarification queries, amidst a few grumbles. Stallman was clearly making progress on the feature, however.

In mid-November, Alan Mackenzie had raised an issue that has seemingly lingered with pcase since it was added in 2010: documentation. He pointed to a post he had made in 2015 that described the problems that existed in the pcase documentation; many of those were addressed at the time, but there is an ongoing effort, led by Jim Porter, to improve the documentation for the macro.

Porter listed multiple areas that need attention, including moving the presentation of the backquote ("`") operator, which is used for pattern-matching and destructuring, earlier in the pcase doc string. As Mackenzie had noted, one of the confusing things about pcase is its use of two punctuation marks, backquote and comma (","), that already have established uses in Elisp macro definitions: "pcase complicated the meaning of ` and ,. Before pcase these had definite meanings. Afterwards, they became highly context dependent."

There were a few responses to Porter's message, largely in favor of his ideas; eventually, Emacs co-maintainer Stefan Kangas copied the post to Stefan Monnier, who is a former Emacs maintainer and the developer of pcase. Monnier was generally in favor as well; he thought that the doc string was not really the place for detailed backquote information, though some reorganization made sense. Stallman took exception to Porter's other suggestion to possibly mention pcase in "An Introduction to Programming in Emacs Lisp". "We should not encourage people learning Emacs Lisp to use pcase."

First draft

In mid-January, Stallman posted a first draft of cond*, asking for more testing, "constructive comments, bug reports, patches, and suggestions". Andrea Corallo asked about one particular feature of cond*:

what is the reason for some of these cond* clauses to keep the binding in effect outside the clause itself and for the whole cond* construct? At first glance it doesn't look very idiomatic in Lisp terms to me.

Corallo is referring to the bind* sub-clause that can appear anywhere in the body of the cond* to bind variables with a scope that does not end with the bind* that contains them. Instead, the scope of those variables is the body of the cond* from that point onward, which is a bit of an oddity from the usual expectation in Lisp.

    (cond*
         (CONDITION FORM)
         ((bind* (x 42)))  ; create a binding of 42 to x
         (CONDITION-using-x FORM-using-x)
         ...)

As João Távora noted, others had already asked about that behavior, but he is "not sure we eventually clarified it". He also wondered if cond* could be built using pcase; if it is a strict subset of the features of pcase, it might help to do so. It "could actually facilitate the adoption path for 'cond*' (as questionable as that path may still be, at least for some parties)". But Stallman does not see things that way; he listed the advantages he sees with cond* and said that he plans to add the new macro to Emacs:

cond* has four basic advances over pcase: making bindings that cover the rest of the body, matching patterns against various data objects (not forcibly the same one), use of ordinary Lisp expressions as conditions in clauses, and the [ability] to make bindings and continue with further clauses.

I'm going to do some more testing and then install cond*.

Adam Porter asked Stallman to reconsider installing cond* into Emacs proper, suggesting that he should consider enhancing pcase rather than add a whole new facility that developers will need to learn. One of the complaints about pcase is that it is a burden to learn; "How will that burden be helped by having to learn both Pcase and cond*?" (Adam) Porter pointed out that pcase could handle many of the advances Stallman had listed, so there may be a path to enhance pcase:

Your stated reasons for writing cond* were various shortcomings of Pcase. Some of those, e.g. the documentation, have already had volunteers step up to address. The others could also be addressed in various ways. I've suggested a few, but you haven't explained the reasons for rejecting them.

As might be guessed, Stallman did not agree:

If pcase lacked features for certain specific jobs, it would be [easy] to fix that by adding a few features. However, the problem with pcase is that it has too many features for the job it does. cond* does the same jobs with fewer features because they work together better.

ELPA?

Kangas said that he was not closely following the discussion, but was surprised to hear that "there was a plan to install `cond*', or I would have spoken up sooner". Adding cond* will necessarily make the job of maintaining Emacs harder, since pcase is not going away, thus there will be "not one, but two relatively complex macros" that he will have to know and understand. That might be worth doing if cond* offered substantial benefits, but he does not see that; "What I see instead is a mere _version_ of `pcase'." So, he recommended making a new package for the GNU Emacs Lisp Package Archive (ELPA), which will provide "a good way of exploring an alternative version of an existing macro".

The lack of a plan to wholesale replace pcase with cond* was seen as a good thing by Kangas. That avoids a bunch of code churn and bugs that would naturally result. But Mackenzie disagreed; he thinks that fully replacing pcase would be a good goal for "improving the readability and maintainability of our code at a stroke". He thinks that putting it into an ELPA library is "a way of ensuring it never comes to anything and just gets forgotten about"; a feature branch with an eventual merge into the Emacs mainline would be a better course.

Like several others, Monnier believes that cond* is simply further complicating Elisp. On the other hand, he said, it would be a bit hypocritical for him to oppose it:

So, I'm not super enthusiastic about adding such a new form, but being responsible for the introduction of several such new constructs in ELisp over the years, I do feel a bit like the pot calling the kettle black.

So, rather than oppose it, I'll just point out some things which I think could be improved (or which offend my taste, as the case may be).

What followed was an extended back-and-forth between Stallman and Monnier about cond*, its overlaps with pcase (and how the two could perhaps "meet in the middle"), some deficiencies in the regular cond form with respect to binding inside of the conditional, and more. Much of the discussion revolved around Monnier's concern that the two constructs had overlapping, but still different, pattern languages:

[...] I see a fairly extensive but hardcoded pattern language, which seems like a regression compared to Pcase where the pattern language is built from a few primitives only (with the rest defined on top via `pcase-defmacro`). The worst part, tho, is that the two pattern languages are very similar, and I can't see any good reason for the differences.

Monnier asserted that Stallman could just reuse the low-level pattern language of pcase for cond*, which would have a number of benefits: "Less work, less code duplication, less documentation duplication, less to learn for coders. And presumably you'd then improve Pcase, so everyone wins."

The discussion continues as of this writing, though it is unclear which direction Stallman will take with the pattern language. That language, which is implemented by the match* form that would be added for cond*, could easily be changed to use the pcase machinery, Monnier said; a patch to do so has already been sent. Mackenzie objected to cond* using the pcase pattern handling, in part because of a lack of documentation of the low-level pcase machinery. Alfred M. Szmidt hoped that eventually pcase could be written to use cond*, instead of the reverse, but Monnier said that would not be technically feasible.

Adding cond*

At the end of January, Emacs co-maintainers Eli Zaretskii and Kangas announced that cond* would be installed in the Emacs core as an alternative to pcase; there would be no effort to switch away from pcase, however, as it is "to be considered a matter of stylistic preference". In the post, Kangas makes it clear that political, rather than strictly technical, considerations were part of the decision-making process:

Our responsibility as maintainers is first and foremost to ensure that we can all work together, and unite under a common banner. Our success as a project depends on it. Thus, the last thing we want to do is to alienate any group of contributors, big or small.

We believe that this is a more important concern than the arguments for or against cond* or pcase. The simple fact is that we have different backgrounds and experiences, which have tended to land us on either side of this discussion. This diversity is a strength, and not a weakness.

Even that has not satisfied everyone as there are apparently still some scars from pcase being installed in Emacs 14 years ago, at least for Mackenzie. He believes that the middle path chosen by the maintainers will not help resolve the problems that he and others have in understanding some pcase uses; he would still advocate a wholesale replacement using cond*. Stallman, on the other hand, does not think that pcase should be replaced, but does "hope to discourage its use inside Emacs in favor of cond*". And, of course, the decision did not stop yet another discussion of "cond* versus pcase" from breaking out and, naturally, continuing at length.

At the end of January, Stallman said that he was close to ready to commit the code to the Emacs Git repository, though that has not happened as of this writing. Zaretskii asked that documentation be added to the Elisp reference manual at the same time, so that may have slowed the process some. Before long, though, cond* should make an appearance in Emacs, so there will be two options for conditionals with pattern-matching and destructuring capabilities—for good or ill.

Comments (13 posted)

Making multiple interpreters available to Python code

It has long been possible to run multiple Python interpreters in the same process — via the C API, but not within the language itself. Eric Snow has been working to make this ability available in the language for many years. Now, Snow has published PEP 734 ("Multiple Interpreters in the Stdlib"), the latest work in his quest, and submitted it to the Python steering council for a decision. If the PEP is approved, users will have an additional option for writing performant parallel Python code.

Snow's work on this topic began in 2015 with a post to the python-ideas mailing list. He followed that up in 2017 by writing PEP 554 (also titled "Multiple Interpreters in the Stdlib"). He later gave a talk at the 2018 Python Language Summit to gather support for the idea. By 2020, he was optimistic about the possibility of seeing PEP 554 approved for Python 3.10. Ultimately, it was delayed to focus on prerequisite work in the form of ensuring that each Python interpreter uses a separate global interpreter lock (GIL). In 2023, he gave a talk at PyCon about the status of the work so far, and what would be necessary to push it over the finish line.

Python already has several ways to run code in parallel. The threading library allows running tasks in different threads, although only one thread can run Python code at any one time because of the GIL. The multiprocessing library avoids contending for the GIL by running tasks in separate Python processes. These processes can communicate via shared memory, but they cannot share Python objects directly, instead relying on a system of queues that copy objects or by using proxy objects. The overhead involved in sharing complex objects can make multiprocessing a poor fit for some applications.

When circulating the initial version of PEP 554 for comment in 2017, Snow summed up the purpose of the work by saying: "The project is partly about performance. However, it's also particularly about offering [an] alternative concurrency model with an implementation that can run in multiple threads simultaneously in the same process."

PEP 734 proposes adding a new module — interpreters — that uses Python's longstanding support for multiple interpreters (sometimes called subinterpreters) to permit running independent Python interpreters, each with its own GIL, in different threads of the same process. It used to be the case that only one interpreter could run at a time, but a previous proposal from Snow, PEP 684 ("A Per-Interpreter GIL"), fixed that by giving each interpreter its own GIL in Python 3.12. This could allow multiple interpreters to offer a substantial performance boost, because separate threads could actually run in parallel.

Another PEP may have had much the same effect; PEP 703 ("Making the Global Interpreter Lock Optional in CPython"), which was proposed and accepted in 2023, makes the GIL optional entirely. That change could make the performance advantages of having multiple interpreters less competitive. On the other hand, the GIL remains enabled by default in CPython releases, since it permits better single-threaded performance and remains a requirement for many extension modules, which may make multiple interpreters a practical option for environments where only a stock Python interpreter can be used.

Sharing the same process allows data to be passed back and forth between the interpreters by sharing the underlying memory, without having to copy it into an area of shared memory or incur the cost of interprocess communication. PEP 734 introduces a new type of Queue for sending objects between interpreters. The PEP is clear that Python objects themselves are still not shared between interpreters. Instead, immutable types (such as str or bytes) can share their underlying storage directly. Small types such as int or float can also be shared directly, as can tuples of shareable objects. The only mutable objects that can currently be shared are Queue and memoryview objects, but the PEP promises that "there is no restriction against adding support for more types later".

Still, this new queue offers noticeably different semantics than multiprocessing queues, which can appear to "fork" objects, since objects are marshalled and copied when sent through the queue. When discussing the desired semantics of the newly introduced queue and the relationship between objects on one side of the queue and the other, Snow said: "My preference for that relationship is 'they may not be the exact same object, but they might as well be'."

Queue feedback

Despite extensive previous discussion, the updated PEP still drew additional comments. Antoine Pitrou expressed concern about the restrictions on what types can be passed between interpreters in order to preserve Snow's desired semantics: "I think interpreters.Queue deviating from the threading and multiprocessing queue semantics by only allowing shareable objects will be annoying [to] users." He went on to suggest that the queues could use the out-of-band buffers available in pickle (Python's object-serialization mechanism) protocol version 5 to send immutable types with the same efficiency the current design allows, while also permitting other objects. Snow agreed that this was an interesting possibility, but is "still ruminating over the potential consequences of using pickle by default".

Pitrou suggested separating out a LowLevelQueue that allows shareable objects only, and then a regular Queue built on top of that. Steve Dower concurred, but noted that "I think we're still at the stage where we want 3rd party packages to design the Queue object". Guido van Rossum agreed that it made sense to have a queue which only accepts some types:

I think there will always be a notion of shareable objects — though that's a poor name, it's really about things that have value semantics. And the interpreters module can have a Queue that only allows values. Over time the definition of "value" can be adjusted.

Shared memory

Ran Benita suggested that it might make sense to consider the design of transferable objects in JavaScript that can be sent between contexts and "'hollowed out' on the sending side". Transferable objects are mutable objects that are safe to send between threads, because they are made unusable by the sending thread in the process of being transferred. Code written using transferable objects can be sure that only one thread will try to mutate them at a time, but avoid the overhead of copying large objects between threads. Benita also said: "The reason I'm bringing up Transferable Objects is not that it should be a part of the PEP, but that I think it would be good to either make sure the design does not preclude it as a future enhancement, or that it explicitly does preclude it in case it's not relevant for Python." Snow agreed that it was a "cool idea" and, while it should not be part of the initial PEP, it had been part of previous discussions.

Benita also questioned what would happen if code in multiple interpreters wrote to a shared memoryview object without synchronizing that access via a queue. The answer would appear to be a data race. Pitrou noted that users already need to worry about this possibility when using multiprocessing shared memory.

Next steps

Now that Snow has submitted PEP 734 to the steering council for consideration, there is a good chance of actually seeing this work merged for Python version 3.13, expected in October 2024. The council is likely to make a pronouncement one way or another in time for the first beta release (and feature freeze) in May. Even if it does approve the PEP, however, there is still more work to be done before the interpreters module will be generally available.

Comments (9 posted)

Not so quickly extending QUIC

QUIC is a UDP-based transport protocol that forms the foundation of HTTP/3. It was initially developed at Google in 2012, and became an IETF standard in 2021. Work on the protocol did not stop with its standardization, however. The QUIC Working Group published several follow-up standards. Now, it is working on four more extensions to QUIC intended to patch over various shortcomings in the current protocol — although progress has not been quick.

QUIC serves as a replacement for TCP and TLS with several interesting benefits. Combining the handshakes involved in establishing a connection and encrypting it allows QUIC to reduce the number of round trips required before applications can send data — to zero, in some cases. Combining encryption with the transport layer also allows QUIC to hide details of connections from intermediate routers ("middleboxes"). QUIC also supports sending multiple independent streams over the same underlying connection. This ability is useful for applications like web servers that want to transfer related resources over the same connection without a dropped packet in one stream delaying the rest — a problem with TCP known as head-of-line blocking.

Load balancing

One interesting feature of QUIC is connection migration, which allows computers to seamlessly change IP addresses while maintaining a connection. QUIC packets all contain a connection identifier, which can be used to determine which connection the packet is part of, even if it comes from an unexpected source address. This poses a challenge for load balancers, which usually determine how to forward packets based on the IP addresses and port numbers of the source and destination. The connection ID is one of the few unencrypted parts of a packet — to allow servers to tell what encryption key should be used with a packet — so a QUIC-aware load balancer can use the connection identifier instead, but this interacts poorly with another aspect of QUIC: connection ID rotation.

QUIC permits a server to supply a client with a pool of connection IDs for a given connection, in order to reduce "linkability". If a QUIC connection remains open for a long time — for example because it is being used to place an ongoing phone call — an attacker could use the connection ID of the packets to track the user as they move between different networks. To mitigate this, a client that knows it is about to change networks (for example, because it is about to switch from a cellular connection to WiFi) can start using a new connection ID from the pool when it switches. The server, which knows what connection IDs it provided to the client, can continue serving the connection with no interruption.

Any non-participant in the connection sees the source IP address, port number, and connection ID change simultaneously, making it difficult to recognize the new flow of packets as being part of the same connection. Unfortunately, this includes any load balancers sitting in front of the server, which have no way to recognize that the packets involved ought to be routed to the same server that handled the initial request.

The QUIC Working Group is trying to enhance load balancing with a new standard for securely encoding load balancing information in the connection ID. Servers using this new standard create connection IDs by combining a server ID and a cryptographic nonce, and then encrypt the resulting string using a key shared with the load balancer to produce a connection ID. This key needs to be provided to the server and the load balancer by whoever is configuring the system. When the load balancer receives a packet, it attempts to decrypt the connection ID and then forward the packet to the relevant server. The standard also includes provisions for handling version updates or key migration.

In June 2023, Martin Duke, one of the authors of the new standard, said that the draft "is mostly ready, but we've put it on ice until it gets deployed somewhere". There have been several small changes since that time, but it is not clear when the draft will be ready to take the next step toward being an IETF standard: being proposed to the Internet Engineering Steering Group (IESG) for consideration.

Multipath

There are several areas where TCP still has an advantage over QUIC. One of those is multipath support. Multipath TCP connections can send data on different network paths simultaneously — for example, sending via both WiFi and cellular data — to provide better throughput than either path permits individually.

The in-progress QUIC multipath extension would adapt QUIC's connection-migration mechanism to allow multiple paths to be in use simultaneously. Currently, when a computer starts sending packets for an existing connection on a new path, that's taken as a sign that the previous path ought to be abandoned. If the connection is set up with the enable_multipath option, sending packets on a new path would instead add that path to the connection and permit both to be used simultaneously.

One complication is with network address translation (NAT) rebinding. QUIC identifies paths by the IP addresses and ports of the source and destination. Unfortunately, these parameters are not always stable for a given path. Some routers perform NAT, allowing multiple computers to share a single IP address. When a TCP connection passes through such routers, the router can watch the TCP session establishment, and keep the same external port mapped to the same internal IP address for the duration of the connection.

This approach doesn't work with QUIC for two reasons. Firstly, QUIC is a relatively new protocol, and many existing routers do not have code to handle it. Secondly, QUIC encrypts many details about the connection in order to avoid interference and reduce linkability. This means that NATs need to fall back to the same non-connection-aware approach used for other UDP protocols: establishing a mapping when they see an outbound packet, and then expiring that mapping after a timeout. An overloaded computer, lost packet, or misconfigured NAT can cause that timeout to expire even though there is an active QUIC connection. In turn, this causes the NAT to select a different outgoing port for the next packet, making the path appear to change.

In existing QUIC, the server would consider this to be a connection migration, and continue without problems. With the multipath extension, the server would instead add the new path to the connection, but still send some data on the old path, which would be dropped by the NAT. To overcome this, multipath QUIC requires that clients changing to a new path use a new connection ID, as described above. Therefore, if the server sees a path change with the same connection ID, it can identify the change as a NAT rebinding event and stop sending packets to the old address.

Another concern is packet numbering and encryption. QUIC uses the packet number of each packet in a connection as a cryptographic nonce when encrypting the contents of the packet. But with multiple simultaneous paths in use, packet numbering becomes more complicated. Using a single sequence of packet numbers for all the paths in a connection makes it difficult to tell when a packet has been dropped, or when the packet has simply been sent via another path, making calculating the reliability and congestion of different paths difficult. The authors of the multipath extension chose to simplify implementations by using a separate set of packet numbers for each path in a connection. This in turn requires changes to how packets are encrypted, in order to prevent any nonces from being used multiple times.

There are several other considerations that have needed hammering out in order to ensure that multipath QUIC is viable. The draft has gone through many rounds of revision, and is still under development by the working group. There is a session planned to discuss it in more detail at the upcoming IETF meeting in March.

Acknowledgment frequency

Since QUIC is designed as an alternative to TCP, it needs to take care of data integrity and congestion control itself. To ensure that all sent data is received, both ends of a QUIC connection send acknowledgment messages. Right now, these messages are sent for every second packet. This is a compromise between sending acknowledgments too quickly (wasting resources) and too slowly (preventing the congestion control algorithms from responding promptly to changes in the network).

Unfortunately, not every QUIC connection has equivalent needs. Some asymmetric internet connections have reduced receiving bandwidth when packets are being sent, making acknowledgments more costly. Some devices have constraints on battery power or transmission frequency. Some devices are connected via reliable paths that don't have noticeable jitter or loss. Devices in all of these situations would benefit from sending fewer acknowledgments.

Unfortunately, sending fewer acknowledgments cannot be done unilaterally, because the computer on the other end of the connection will interpret this to mean that the data ought to be retransmitted, or at least that the round-trip time of the path is much worse than it is. Omitting acknowledgments can still work if the other participant in the connection is expecting it, however.

The draft acknowledgment frequency extension allows the participants in a connection to request changes in the acknowledgment mechanism. Systems using the extension can set the maximum number of packets or amount of time that can occur before an acknowledgment is sent.

The extension also adds an IMMEDIATE_ACK message, to explicitly request that the other side of the connection send an acknowledgment upon receipt. A large section of the document is dedicated to detailing when implementations may want to use IMMEDIATE_ACK messages to ensure that delayed acknowledgments don't cause unnecessary slowdowns.

The extension is theoretically ready to move on to the next step of standardization, because its last call for comments ended in October 2023 without prompting another revision. The authors of the extension have not moved forward with the next step in the IETF's standardization process, however, prompting Gorry Fairhurst — one of the working group members — to ask on the mailing list in mid-February "Is this document finished or is it waiting for action based on issues?", but there has been no response.

Partial delivery

QUIC allows separate streams multiplexed over the same connection to fail independently — an eventuality that might occur in, for example, a video conference where one participant has problems while others do not. When only one stream experiences a problem, the server can send a RESET_STREAM message to signal to the client that a particular stream was affected. When this happens, QUIC says that the client should discard any data received for that stream so far, and the server should not respond to retransmit requests for the data.

This is a problem for protocols like the draft WebTransport standard, which send some initial data that must be received reliably on a stream, but that still wish to have streams be resettable. The QUIC Working Group is addressing this use case with a draft QUIC extension that defines a RESET_STREAM_AT message. This would allow a server to reset a stream, while specifying that data before a particular offset should still be retained or rerequested if it is lost in transmission.

The draft is nearly ready to be made a published standard, with the last call for comments ending on February 8. The IETF mandates some additional process before the standard is presented to the IESG for potential publication, so it still may be some time before it is adopted.

Conclusion

These potential improvements to QUIC promise to make the protocol more useful and performant, especially for devices with asymmetric or intermittently available links. Unfortunately, progress on these improvements has not exactly been quick. Christian Huitema, a long-time contributor to QUIC, expressed frustration with the working group's progress:

In the old days, the IETF was prone to find a solution that was good enough, ship it, gather experience, and then revise the standard later to fill in the gaps. If we had followed that process, we could probably have published a QUIC Multipath RFC last year, or maybe the year before — the draft 6 was definitely good enough, and the previous draft was probably OK as well. But we have decided instead to discuss all details before approving the draft. The result will probably be better, although gathering experience sooner would also have helped improve quality. In any case, the process is not quick.

It's uncertain how long it will take the IETF to finalize the new standards. In the meantime, QUIC adoption is growing, with support available in one form or another in every major browser. One of the key concerns of QUIC's designers was avoiding protocol ossification, but QUIC's anti-ossification design choices will be of little use if further improvements to the protocol are stymied by the slow-moving nature of the standardization process.

Comments (18 posted)