Leading items

Welcome to the LWN.net Weekly Edition for July 8, 2021

This edition contains the following feature content:

• Rust for Linux redux: support for writing kernel code in Rust is getting closer to merge-ready, but the development community is not yet fully sold on the idea.
• Core scheduling lands in 5.14: a new scheduling mode for Spectre protection and more.
• The first half of the 5.14 merge window: what the first set of commits for 5.14 brought.
• Bye-bye bdflush(): an ancient and unused Linux system call is about to go away.
• Python attributes, __slots__, and API design: how can we protect against silly mistakes in security-critical code?

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Rust for Linux redux

On July 4, the Rust for Linux project posted another version of its patch set adding support for the language to the kernel. It would seem that the project feels that it is ready to be considered for merging into the mainline. Perhaps a bigger question lingers, though: is the kernel development community ready for Rust? That part still seems to be up in the air.

Round 2

Miguel Ojeda, who has been hired to work on the project full-time, posted the patch set; in the cover letter, he listed a number of changes and updates since the RFC patch set was posted back in April. In particular, the allocations that would call panic!() when they failed have been replaced. A modified version of the alloc crate has been created for the kernel project, though the plan is for the changes to work their way into the upstream project so that the customized version can eventually be dropped. Incidentally, the patch adding the modified crate was apparently too large for the lore archive, though it does show up in the LWN archive.

There is more progress on Rust abstractions for kernel facilities, including "red-black trees, reference-counted objects, file descriptor creation, tasks, files, io vectors...", he said, as well as additions for driver support. Beyond that, the Rust driver for the Android Binder interprocess communication mechanism has more features and there is ongoing work on a driver for the hardware random-number generator available on some Raspberry Pi models.

But the lack of a "real" driver was one of the sticking points back in April, and it was raised again this time. Christoph Hellwig said that he was strongly against merging the code if that was the intent of the posting. In the cover letter, Ojeda said that the patch set was being added to linux-next, and he confirmed that it is being submitted for merging. Hellwig wants to see Rust prove itself before being included:

[...] prove it actually is useful first. Where useful would be a real-life driver like say nvme or a usb host controller driver that actually works and shows benefits over the existing one. Until it shows such usefulness it will just drag everyone else down.

Ojeda pointed at the Binder driver as a "non-trivial module" that "is already working", but acknowledged that a real hardware driver is an important (and, as yet, missing) piece. But the cover letter listed a number of organizations that are interested in or involved with the project already, and Ojeda believes that is also something that is helping to prove the project:

It is "proven" in the sense we are already starting to get users downstream and other interested parties have shown support.

However, others are more conservative and will only start investing into it if we are in mainline, i.e. if the decision of having Rust or not is already taken.

But the Binder "driver" is not really a good example to use for a few different reasons, Greg Kroah-Hartman said. It is missing a fairly large piece of functionality (binderfs) for one thing, but it also does little to help show how Rust will fit in with the rest of the kernel. As before, he strongly recommended working on something that would help clear up some of the questions that kernel developers have about Rust:

Not to say that it doesn't have its usages, but the interactions between binder and the rest of the kernel are very small and specific. Something that almost no one else will ever write again.

Please work on a real driver to help prove, or disprove, that this all is going to be able to work properly. There are huge unanswered questions that need to be resolved that you will run into when you do such a thing.

Kernel Summit topic

The theme of Rust proving itself was also present in a thread on the ksummit-discuss mailing list. At the end of June, Ojeda proposed Rust for Linux as a technical topic for the Kernel Summit track at this year's Linux Plumbers Conference. On July 6, Linus Walleij replied, agreeing that it was a topic that should be discussed. He noted that there are already quite a few languages that kernel developers need to be up on (e.g. C, assembly, Make, Bash, Perl, Python, ...), so a question in his mind is what Rust will bring to the table that makes it worth adding to the list.

That message started the ball rolling on a discussion of how Rust fits—and where the experiment leads. There is clearly some concern among kernel developers about having to know Rust to continue working on the kernel. Ojeda tried to allay those fears to a certain extent, but did acknowledge that most kernel developers would eventually need to understand the language. For now, the intent is for the project to work with maintainers that want to add a Rust API to their subsystem, Ojeda said. That will bootstrap support for Rust within the kernel and will help alleviate some of the concerns expressed by Leon Romanovsky, James Bottomley, and others.

Romanovsky was worried about refactoring, especially cross-subsystem refactoring, in a system with drivers in both languages. Bottomley suggested that there would be fewer reviewers for the Rust code, so bugs could more easily slip in:

Since most of our CVE type problems are usually programming mistakes nowadays, the lack of review could contribute to an increase in programming fault type bugs which aren't forbidden by the safer memory model.

In the near term, those are problems that will need to be dealt with, but the goal is to get to a point where Rust knowledge is more widespread among kernel developers. As Ojeda put it:

After all, if we are going to have Rust as a second language in the kernel, we should try to have as many people on board as possible, at least to some degree, within some reasonable time frame.

As Laurent Pinchart pointed out, that is an important point and one that needs to be more clearly highlighted in the discussion:

[...] adopting Rust as a second language in the kernel isn't just a technical decision with limited impact, but also a process decision that will create a requirement for most kernel developers to learn Rust. Whether that should and will happen is what we're debating, but regardless of the outcome, it's important to phrase the question correctly, with a broad view of the implications.

There are, of course, plenty of technical hurdles that need to be cleared as well. One of those areas is the Linux driver model and the object-lifetime handling that goes along with it. Roland Dreier suggested that interfaces like devres (i.e. managed device resources) could have avoided a lot of problems that he has seen in drivers regarding lifetime management, and error paths in particular, but that it has not been adopted widely. Walleij disagreed that it hasn't been widely used, but is not at all sure that a Rust switch is better:

I think it's a formidable success, people just need to learn to do it more.

But if an easier path to learn better behaviours is to shuffle the whole chessboard and replace it with drivers written in Rust, I don't know? Maybe?

For Kroah-Hartman, seeing real drivers in Rust for the kernel would help clear up how these problems are going to be solved in the language. There are some difficult problems that will need to be tackled:

This is going to be the "interesting" part of the rust work, where it has to figure out how to map the reference counted objects that we currently have in the driver model across to rust-controlled objects and keep everything in sync properly.

For the normal code, the fact that the memory was assigned to one specific object (the struct device) but yet referenced from another object (the cdev). devm_* users like this do not seem to realize there are two separate object lifecycles happening here as the interactions are subtle at times.

I am looking forward to how the rust implementation is going to handle all of this as I have no idea.

There is also the question of where all of this leads. Walleij wondered whether the "leaf nodes" of the kernel (i.e. drivers) are actually the best place to start from the perspective of showing the benefits of the language. He also asked about that in a lengthy message back in April. Part of his question is whether the decision to start with drivers was made for other reasons:

If the whole rationale with doing device drivers first is strategic, not necessarily bringing any benefits to that device driver subsystem but rather serving as a testing ground and guinea pig, then that strategy needs to be explicit and understood by everyone. So while we understand that Rust as all these $UPSIDES doing device drivers first is purely strategic, correct? I think the ability to back out the whole thing if it wasn't working out was mentioned too?

He also asked about writing whole subsystems in Rust. That would entail exposing Rust APIs to C code elsewhere in the kernel. It would also potentially allow an evolution toward more and more Rust in the kernel:

If we want to *write* a subsystem in Rust then of course it will go the other way: Rust need to expose APIs to C. And I assume the grand vision is that after that Rust will eat Linux, one piece at a time. If it proves better than C, that is.

Ojeda said that while it is possible to expose Rust APIs to the C part of the system, he does not recommend it. The problem is that the C callers lose much of the benefit that Rust brings to the table:

In general, I would avoid exposing Rust subsystems to C.

It is possible, of course, and it gives you the advantages of Rust in the *implementation* of the subsystem. However, by having to expose a C API, you would be losing most of the advantages of the richer type system, plus the guarantees that Rust bring as a language for the consumers of the subsystem.

In a similar vein, Johannes Berg asked about replacing parts of a subsystem with Rust, but leaving the drivers in C—the reverse of the existing plan, effectively. Once again, Ojeda said that its possible, but cautioned about losing Rust features and guarantees. In addition, there are architectures where there is no Rust compiler available at this point, so it may be premature to be looking at Rust-based subsystems.

Future

The project's eventual goal is not entirely clear. If all of the Linux drivers were written in Rust, there would still be lots of big important pieces that are running unsafe C; is the next step to replace those if Rust proves itself in the meantime? But it is also unclear what "proves itself" actually means in this context.

Learning a new language, with all of its different behaviors, quirks, and idiosyncrasies, is a pretty big ask for developers who are already juggling a fair amount of complexity in maintaining the existing kernel code. Not to mention the added complexity of providing new functionality on top of what's already there. The skepticism that seems fairly prevalent from commenters in both threads (and elsewhere) likely stems from that learning burden.

Adding Rust to the kernel requires a lot of work, for a lot of different people, without a clear and obvious benefit beyond promises that can only truly be fully fulfilled with a whole kernel written in Rust. With luck, the project can provide some clear "wins" in the early going that clearly demonstrate both the potential of the language and an ability to be utilized incrementally on a large and complex code base like the kernel. Without that, it may be difficult for the project to progress very far in the goal of "rustifying" the kernel.

Comments (305 posted)

Core scheduling lands in 5.14

The core scheduling feature has been under discussion for over three years. For those who need it, the wait is over at last; core scheduling was merged for the 5.14 kernel release. Now that this work has reached a (presumably) final form, a look at why this feature makes sense and how it works is warranted. Core scheduling is not for everybody, but it may prove to be quite useful for some user communities.

Simultaneous multithreading (SMT, or "hyperthreading") is a hardware feature that implements two or more threads of execution in a single processor, essentially causing one CPU to look like a set of "sibling" CPUs. When one sibling is executing, the other must wait. SMT is useful because CPUs often go idle while waiting for events — usually the arrival of data from memory. While one CPU waits, the other can be executing. SMT does not result in a performance gain for all workloads, but it is a significant improvement for most.

SMT siblings share almost all of the hardware in the CPU, including the many caches that CPUs maintain. That opens up the possibility that one CPU could extract data from the other by watching for visible changes in the caches; the Spectre class of hardware vulnerabilities have made this problem far worse, and there is little to be done about it. About the only way to safely run processes that don't trust each other (with current kernels) is to disable SMT entirely; that is a prospect that makes a lot of people, cloud-computing providers in particular, distinctly grumpy.

While one might argue that cloud-computing providers are usually grumpy anyway, there is still value in anything that might improve their mood. One possibility would be a way to allow them to enable SMT on their systems without opening up the possibility that their customers may use it to attack each other; that could be done by ensuring that mutually distrusting processes do not run simultaneously in siblings of the same CPU core. Cloud customers often have numerous processes running; spamming Internet users at scale requires a lot of parallel activity, after all. If those processes can be segregated so that all siblings of any given core run processes from the same customer, we can be spared the gruesome prospect of one spammer stealing another's target list — or somebody else's private keys.

Core scheduling can provide this segregation. In abstract terms, each process is assigned a "cookie" that identifies it in some way; one approach might be to give each user a unique cookie. The scheduler then enforces a regime where processes can share an SMT core only if they have the same cookie value — only if they trust each other, in other words.

More specifically, core scheduling is managed with the prctl() system call, which is defined generically as:

    int prctl(int option, unsigned long arg2, unsigned long arg3,
              unsigned long arg4, unsigned long arg5);

For core-scheduling operations, option is PR_SCHED_CORE, and the rest of the arguments are defined this way:

    int prctl(PR_SCHED_CORE, int cs_command, pid_t pid, enum pid_type type,
	      unsigned long *cookie);

There are four possible operations that can be selected with cs_command:

• PR_SCHED_CORE_CREATE causes the kernel to create a new cookie value and assign it to the process identified by pid. The type argument controls how widely spread this assignment is; PIDTYPE_PID only changes the identified process, for example, while PIDTYPE_TGID assigns the cookie to the entire thread group. The cookie argument must be NULL.
• PR_SCHED_CORE_GET retrieves the cookie value for pid, storing it in cookie. Note that there is not much that a user-space process can actually do with a cookie value; its utility is limited to checking whether two processes have the same cookie.
• PR_SCHED_CORE_SHARE_TO assigns the calling process's cookie value to pid (using type to control the scope as described above).
• PR_SCHED_CORE_SHARE_FROM fetches the cookie from pid and assigns it to the calling process.

Naturally, a process cannot just fetch and assign cookies at will; the usual "can this process call ptrace() on the target" test applies. It is also not possible to generate cookie values in user space, a restriction that is necessary to ensure that unrelated processes get unique cookie values. By only allowing cookie values to propagate between processes that already have a degree of mutual trust, the kernel prevents a hostile process from setting its own cookie to match that of a target process.

Whenever a CPU enters the scheduler, the highest-priority task will be picked to run in the usual way. If core scheduling is in use, though, the next step will be to send an inter-processor interrupt to the sibling CPUs, each of which will respond by checking the newly scheduled process's cookie value against the value for the process running locally. If need be, the interrupted processor(s) will switch to running a process with an equal cookie, even if the currently running process has a higher priority. If no compatible process exists, the processor will simply go idle until the situation changes. The scheduler will migrate processes between cores to prevent the forced idling if possible.

Early versions of the core-scheduling code had a significant throughput cost for the system as a whole; indeed, it was sometimes worse than just disabling SMT altogether, which rather defeated the purpose. The code has been through a number of revisions since then, though, and apparently performs better now. There will always be a cost, though, to a mechanism that will occasionally force processors to go idle when runnable processes exist. For that reason core scheduling, as Linus Torvalds put it, "makes little sense to most people". It can be beneficial, though, in situations where the only alternative is to turn off SMT completely.

While the security use case is driving the development of core scheduling, there are other use cases as well. For example, systems running realtime processes usually must have SMT disabled; you cannot make any response-time guarantees when the CPU has to compete with a sibling for the hardware. Core scheduling can ensure that realtime processes get a core to themselves while allowing the rest of the system to use SMT. There are other situations where the ability to control the mixing of processes on the same core can bring benefits as well.

So, while core scheduling is probably not useful for most Linux users, there are user communities that will be glad that this feature has finally found its way into the mainline. Adding this sort of complication to a central, performance-critical component like the scheduler was never going to be easy but, where there is sufficient determination, a way can be found. The developers involved have certainly earned a cookie for pushing this work to a successful completion.

Comments (17 posted)

The first half of the 5.14 merge window

As of this writing, just under 5,000 non-merge changesets have been pulled into the mainline repository for the 5.14 development cycle. That is less than half of the patches that have been queued up in linux-next, so it is fair to say that this merge window is getting off to a bit of a slow start. Nonetheless, a fair number of significant changes have been merged.

Some of the more interesting changes pulled so far include:

Architecture-specific

• Arm64 pointer authentication can now be configured independently for kernel and user space.
• The x86 split-lock detection was designed to kill processes that perform atomic operations that cross cache lines — operations that can severely affect performance. The 5.14 kernel adds a new command-line parameter (split_lock_detect=ratelimit:N) that can set a rate limit, expressed in lock operations per second. If that limit is exceeded (in the system as a whole), any process creating a split lock will be forced into a 20ms sleep rather than being killed.

Core kernel

• There is a new futex operation, FUTEX_LOCK_PI2, which uses the monotonic clock for timeouts rather than the realtime clock.
• The core scheduling functionality, which provides control over which processes can share a core, has been merged. Core scheduling can be used as a defense against some Spectre vulnerabilities, but there are other use cases for it as well.
• The burstable CFS bandwidth controller is now in the mainline; this feature allows bursty workloads to briefly exceed their CPU-time restrictions in some conditions.
• The initial infrastructure for BPF program loaders has been merged; this work will eventually allow the kernel to require BPF programs presented for loading to be signed by a trusted key.

Filesystems and block I/O

• There is a new I/O priority controller for control groups that can manage the priority of block-I/O requests (including writeback) generated by members of each group. This commit contains a bit of documentation on this feature. The mq-deadline I/O scheduler has been updated to support these priorities.

Hardware support

• Hardware monitoring: MPS MP2888 pulse-width modulators, Sensiron SHT4x humidity and temperature sensors, Flex PIM4328 power interface modules, and Delta DPS920AB power supplies.
• Media: Sony IMX208 sensors and Atmel extended image sensor controllers.
• Miscellaneous: Stormlink SL3516 crypto offloaders, PolarFire SoC (MPFS) mailbox controllers, Lenovo WMI-based systems management controllers, and Intel SkyLake ACPI INT3472 camera power controllers.
• Networking: Intel M.2 WWAN IPC-over-shared-memory controllers, Ingenic Ethernet controllers, Loongson PCI DWMACs, Sparx5 network switches, and Mellanox BlueField gigabit Ethernet interfaces.
• Regulator: Richtek RT6160 BuckBoost and RT6245 voltage regulators, MediaTek MT6359 power-management ICs, Silergy SY7636A voltage regulators, and Maxim 8893 voltage regulators.
• Removals: at long last, the old IDE block drivers have been removed; the libata subsystem is able to control any IDE devices that are still able to spin.

Networking

• There is an elaborate new mechanism allowing for custom configuration of hash policies for multipath IP traffic; see this merge commit for details.
• The networking layer almost gained support for NVMe/TCP offload adapters; see this commit for some details. Unfortunately, that support was not kept for long; it was reverted after a request from the NVMe developers who were surprised by the whole thing and did not feel that the code was ready for merging.
• The virtio virtual transport has gained support for SOCK_SEQPACKET sockets (which are described briefly in the socket() man page).
• The SO_REUSEPORT socket mechanism has been improved to give applications more control over how failover happens and to avoid spurious connection failures.

Security-related

• User-space handlers for seccomp() have a new operation that can create a file descriptor for the sandboxed task and return that file descriptor as a result of the system call being handled — all as a single atomic operation. This is a partial solution to the signal-related problems covered here in April.
• There is a new mechanism providing better control over resource limits within user namespaces.

Virtualization and containers

• KVM now supports the Arm64 memory tagging extension in guests.

Internal kernel changes

• The DISCONTIGMEM memory model, described in this article, has been removed since no architectures use it.

The 5.14 merge window can be expected to stay open through July 11, though the possibility of an early closing always exists. LWN will, naturally, post another article once the merge window closes describing the additional changes merged; watch this space.

Comments (1 posted)

Bye-bye bdflush()

The addition of system calls to the Linux kernel is a routine affair; it happens during almost every merge window. The removal of system calls, instead, is much more uncommon. That appears likely to happen soon, though, as discussions proceed on the removal of bdflush(). Read on for a look at the purpose and history of this obscure system call and to learn whether you will miss it (you won't).

Linux, like most operating systems, buffers filesystem I/O through memory; a write() call results in a memory copy into the kernel's page cache, but does not immediately result in a write to the underlying block storage device. This buffering is necessary for writes of anything other than complete blocks; it is also important for filesystem performance. Deferring block writes can allow operations to be coalesced, provide opportunities for better on-disk file layout, and enables the batching of operations.

Buffered file data cannot be allowed to live in memory forever, though; eventually the system must arrange for it to be flushed back to disk. Even the 0.01 Linux release included a version of the sync() system call, which forces all cached filesystem data to be written out. While the kernel would flush some buffers when the buffer cache (which preceded the page cache and was a fixed-size array at that time) filled up, there was no provision for regularly ensuring that all buffers were pushed out to disk. That task was, if your editor's memory serves, handled by a user-space process that would occasionally wake up and call sync().

There are advantages to handling this task in the kernel, though; it has a much better idea of the state of both the buffer cache and the underlying devices. As a step in that direction, the bdflush() system call was added to the 0.99.14y release on February 2, 1994. (This was a different era of kernel development; the preceding 0.99.14x release came out seven hours earlier, and 0.99.14z came out nine hours later). That implementation was not particularly useful, though; all it did was return a "not implemented" error. An actual bdflush() implementation was not added until the 1.1.3 development kernel in April 1994.

It must be said that bdflush() was a strange system call. It was defined as:

    int bdflush(int func, long data);

If func was zero, bdflush() would never return; instead, it would loop within the kernel, occasionally flushing out dirty buffers. In essence, a user-space process would become the kernel buffer-flushing thread by making that call; these were the days before proper kernel threads, after all. Passing func as one would cause some buffers to be flushed immediately. Higher values of func would either read or write the value of a control parameter for the flushing thread; these included the percentage of dirty buffers needed to activate flushing, the number of blocks to write in each cycle, etc.

While bdflush() was an improvement, there were a number of problems with it as well. One of those was that it relied on user space for a critical kernel function; if no process ever set itself up with bdflush(), or if that process were killed, bad things would happen. In the 1.3.50 development release (December 1995), the kernel was changed to automatically create a kernel thread (something it could do at that point) to do the flushing work. User space could still call bdflush() to tweak the various parameters, but an attempt to run as the flushing daemon would turn into an immediate call to exit(); that caused the update process started by older init systems to "work", avoiding boot-time unhappiness.

Another problem with bdflush() — or, more specifically, with the underlying implementation — since the beginning is that it was a single thread. As Linux grew in popularity and found itself on bigger systems, that single thread became an increasingly severe bottleneck. If you have a number of drives on a system, it will eventually take multiple threads to keep them all busy. Thus Andrew Morton replaced the remaining bdflush() infrastructure entirely in 2002 for the 2.5.8 development kernel; in its place was a new set of kernel threads called pdflush. Each pdflush thread was dedicated to a separate physical drive, providing a much-needed scalability improvement.

In December 2002, Morton merged a patch from Robert Love formally deprecating the bdflush() system call, promising that it "will be removed in a future kernel". The pdflush threads were removed in 2009 (for 2.6.32) in favor of a rather more elaborate, workqueue-based, writeback-control mechanism; those can still be seen in the form of kernel threads with names like kworker/u8:3-flush-259:0. Meanwhile, though, bdflush() lives on in current kernels, even though it has not done anything for many years.

Now, however, Eric Biederman is proposing to remove bdflush() entirely as part of a larger project he has to rework the kernel's exit() code. Given that this system call does nothing, was never widely used in the first place, and has been deprecated for nearly 19 years, one might confidently conclude that there are no users left. As it turns out, though, Geert Uytterhoeven has an old m68k image that he occasionally boots, presumably on days when he is overcome with nostalgia. Michael Schmitz demonstrated, though, that said image still boots successfully in the absence of bdflush(), so it is not an impediment to the system call's removal.

There are no other known users of bdflush() out there, so there would appear to be nothing preventing this removal from happening. At that point, it will be the first system call removed since late 2019, when sysctl() was deleted — by the same Eric Biederman. It would be surprising to see that happen in 5.14, though, given how recently this patch was posted. This system call has endured for almost 19 years after it ceased to be useful; keeping it for another two months until 5.15 does not seem like much of an imposition.

Comments (12 posted)

Python attributes, __slots__, and API design

A discussion on the python-ideas mailing list touched on a number of interesting topics, from the problems with misspelled attribute names through the design of security-sensitive interfaces and to the use of the __slots__ attribute of objects. The latter may not be all that well-known (or well-documented), but could potentially fix the problem at hand, though not in a backward-compatible way. The conversation revolves around the ssl module in the standard library, which has been targeted for upgrades, more than once, over the years—with luck, the maintainers may find time for some upgrades relatively soon.

Thomas Grainger posted about a problem he encountered when setting the minimum TLS version to use for a particular SSLContext using the following code:

    context.miunimum_version = ssl.TLSVersion.TLSv1_3

That was meant to ensure that the program would only use TLS version 1.3 (and TLS 1.4+ someday perhaps), but he observed the program using TLS 1.2. As sharp-eyed readers may have noticed, "minimum_version" has been misspelled, leading to the bug.

It is, of course, no surprise that Python happily accepts the attribute name, even though it is "wrong". In a dynamic language, there is nothing inherently wrong with setting an attribute with an arbitrary name, but this case is a little different. For one thing, SSLContext is, obviously, a security-sensitive object, so an API that requires setting attributes—correctly spelled—may be less than ideal.

One way to potentially fix the problem is by using the __slots__ class variable for the SSLContext, as Jonathan Fine pointed out. A Python class that has a __slots__ entry is restricted to attribute names that are listed in the class variable:

    >>> class Foo:
    ...     __slots__ = ('bar', 'baz')
    ... 
    >>> x = Foo()
    >>> x.bar = 3
    >>> x.qux = 9
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'Foo' object has no attribute 'qux'

Fine also noted that the use of __slots__ is not well-documented; he laid down a challenge, in fact: "Find a page in docs.python.org that describes clearly, with helpful examples, when and how to use __slots__." One of the links he suggested, the "Descriptor HowTo Guide", does have a section with some concrete examples on how __slots__ could be used, though it seemingly falls short of what he was looking for.

Under the covers, __slots__ shorts out the normal per-instance dictionary for attributes and replaces it with a fixed-sized array, which saves some memory and speeds up attribute lookup; it will also catch the kind of pesky spelling error that Grainger ran into by raising an AttributeError at run time. Fine pointed to the warning in red in the ssl description, which strongly suggests reading the "Security Considerations" section of the module documentation. "Given that context is important for security, perhaps it's worthwhile closing the door to spelling errors creating security holes."

"Bluenix" also suggested __slots__, but as Guido van Rossum said, that is not a backward-compatible solution. There could be "code out there that for some reason adds private attributes to an SSLContext instance, and using __slots__ would break such usage." Even if it were determined that a compatibility break was in order here, __slots__ could not be used for other reasons, as Christian Heimes, one of the ssl maintainers, pointed out:

Also __slots__ won't work. The class has class attributes that can be modified in instances. You cannot have attributes that are both class and instance attributes with __slots__. We'd have to overwrite __setattr__() and block unknown attributes of exact instances of ssl.SSLContext.

There are two class attributes, sslsocket_class and sslobject_class, that are specifically mentioned as being settable in an instance, but __slots__ does not work with attributes that can switch between class and instance attributes. Eric V. Smith thought that making a specific fix for SSLContext was not the right approach, though:

Isn't this a general "problem" in python, that's always been present? Why are we trying to address the problem with this specific object? I suggest doing nothing, or else thinking big and solve the general problem, if in fact it needs solving.

But the problem at hand is "that assigning attributes is a bad API", Oscar Benjamin said. He suggested adding a new class with a better interface as a backward-compatible way forward. Others agreed that because of the security-sensitive nature of SSLContext, it deserves "special" treatment; the general "problem" of misspelled attributes is not seen as something that needs to be addressed in the language, however.

Grainger is advocating using a frozen dataclass for SSLContext, though that would also break backward compatibility. Python dataclasses were added for Python 3.7 (in 2018) as a way to represent a collection of data items as the attributes on a object, similar to a C struct. A frozen dataclass gets initialized with a set of values that cannot be changed by setting the attribute directly; Grainger suggested having explicit methods to change the attributes of an SSLContext.

Most who commented in the thread seemed to agree that there is a problem to be solved; Marc-Andre Lemburg put it this way:

IMO, a security relevant API should not use direct attribute access for adjusting important parameters. Those should always be done using functions or method calls which apply extra sanity checks and highlight issues in [the] form of exceptions.

Steven D'Aprano thought that a compatibility break might be in order to try to resolve a "mildly troubling security flaw/bug/vulnerability" Others were less sure of that, though. If there is to be a compatibility break, it should create "a cleaner, more Pythonic API", Brendan Barnwell said. He had some suggestions for how that might look:

Why not have the class accept only valid options at creation time and raise an error if any unexpected arguments are passed? Is there even any reason to allow changing the SSLContext parameters after creation, or could we just freeze them on instance creation and make people create a separate context if they want a different configuration? I think any of these would be better than the current setup that expects people to adjust the options by manually setting attributes one by one after instance creation.

Heimes said that there will not be any incompatible changes made to SSLContext in the near future, however. If time is found to work on this problem for Python 3.11, changes along the lines of the configuration object in PEP 543 ("A Unified TLS API for Python") would be made. We looked at the PEP in early 2017, but it was withdrawn in mid-2020, "due to changes in the APIs of the underlying operating systems". There are still pieces of the PEP that could be used to address the problem that Grainger encountered.

The ssl module has always been a thin layer atop OpenSSL, which has undergone a number of API (and other) changes over the years. Support for TLS in the Python standard library has changed as well; up until Python 3.4 in 2014, TLS certificates were not able to be checked for validity using it at all, for example. The ssl module has seemingly always had a lack of available developer time, which is rather worrisome for a critical piece of security infrastructure. Hopefully some time can be found to at least resolve problems like this that can be caused by a simple misspelling.

Comments (14 posted)