Leading items

Welcome to the LWN.net Weekly Edition for April 2, 2020

• Three candidates vying to be DPL: what the three Debian project leader candidates want to accomplish.
• Avoiding retpolines with static calls: another technique to claw back some of the performance lost to the Spectre vulnerabilities.
• Per-system-call kernel-stack offset randomization: making life harder for attackers by moving the kernel stack frequently.
• Some 5.6 kernel development statistics: where the code in the 5.6 release came from.
• Reworking StringIO concatenation in Python: a debate over whether the StringIO class should support a concatenation operator.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Three candidates vying to be DPL

The annual Debian project leader (DPL) election is well underway at this point; voting begins in early April and the outcome will be known after the polls close on April 18. Outgoing DPL Sam Hartman posted a lengthy "non-platform" in the run-up to the election, which detailed the highs and lows of his term, perhaps providing something of a roadmap, complete with pitfalls, for potential candidates—Hartman is not running again this time. When the nomination period completed, three people put their hats into the ring: Jonathan Carter, Sruthi Chandran, and Brian Gupta. Their platforms have been posted and there have been several threads on the debian-vote mailing list with questions for the candidates; it seems like a good time to look in on the race.

After the call for nominations went out, Carter posted a self-nomination that was rather longer than the usual message of that type. Hartman was concerned that his "not running" message might have influenced the length of Carter's "thoughtful message"—and set too high of a bar for others who might want to nominate themselves. Carter assured Hartman that his posting (and its length) had no bearing on the self-nomination, however.

Carter's message is also referenced from his platform, not surprisingly. He sets down his reasons for running for DPL, including finding ways to highlight the many positive aspects of the project:

Debian Foundations

Chandran had a more traditional self-nomination, while Gupta revealed the sole plank of his platform in his self-nomination message:

The platforms page has links for each candidate's platform. Each candidate is given the opportunity to add "rebuttals" of their opponents' platforms once they have been revealed. Chandran did not choose to do so, but both Gupta and Carter did. In general, the rebuttals are meant to show differences between the candidates or possible problems with a platform in a polite and formalized way.

Gupta's quest for the creation of foundation organizations for the project garnered a generally favorable response, though some wondered why it required being DPL to advance that cause. As described in his platform, Gupta wants to free the DPL from their purely administrative duties: "Debian Project Leaders should have more time to lead rather than be buried in the set of administrative tasks they currently face". The existing trusted organizations (TOs), mainly Software in the Public Interest (SPI), are falling short in various ways in his view; Debian-focused foundations would better serve the project. He also explained why having two foundations, under different legal frameworks, will provide redundancy in the face of problems in a particular jurisdiction and allow local financial transactions within the two dominant regions for the project.

Hartman agreed that establishing foundations made sense for the project, though for somewhat different reasons. Once his DPL term ends, he would be willing to sponsor a general resolution (GR) as a referendum on the foundations idea; he thinks Gupta would be a good choice to lead that effort, so the GR would "include text delegating making it happen" to Gupta. "But I can't figure out why you'd need or want to be DPL to do that", Hartman said.

Gupta replied that his idea would need the explicit support of the DPL, so that would be easiest if he were the DPL. He also sees the DPL election as a somewhat lighter-weight alternative to a GR; those can be kind of contentious within Debian these days. He does not think a GR is required "as my current plan doesn't require any changes to the constitution". But the election provides a mechanism for project members to express their preferences:

Mixing the DPL election with another issue was questioned by several in the thread and, in particular, with Gupta's interpretation of what a vote for "none of the above" (NOTA) should "mean" with respect to the foundations idea. Enrico Zini put it this way:

In his rebuttal to Gupta's platform, Carter raised another point: it is likely to take more than a single term to make a change of this sort. In fact, the kinds of questions that will need to be asked could be problematic to resolve. He is concerned that could easily stretch past a single term:

Gupta is US-based, so he plans to start with a foundation for the US if elected, but he acknowledged that even that might well stretch beyond a year:

Diversity

Chandran, like Gupta in some ways, is a "single-issue" candidate; she would like to increase the diversity within the Debian project, particularly along gender lines.

As she noted, there were not a lot of specific plans in her platform, which is something that Carter and Gupta mentioned in their rebuttals. Beyond that, Chandran is leading the team bringing DebConf to India in 2022; that will keep her quite busy over the next two years, as both of the other candidates pointed out.

Hector Oron asked the candidates about the Debian Outreach team; in particular, he wondered whether there were flaws in the current outreach efforts (mostly based around Outreachy and Google Summer of Code) and what could be done about them. Part of the thread veered away from candidate answers to some extent, but covered some important ground.

There was some general questioning of the value that Debian was getting from its paid outreach efforts. Martin Michlmayr noted that Debian really might not be providing a path for interns once they complete their internship:

Ulrike Uhlig, who was a Debian Outreachy intern in 2015, pointed out that there is a feedback element missing from the outreach efforts. It could be added as part of the outreach evaluation:

Chandran agreed that it would be useful to come up with such a process, as did Outreach team member Pranav Jain. Both said they would be happy to work with Uhlig on that effort.

In answer to Oron's original question, Carter said that there was simply not enough being done in the outreach department. For now, it would seem that outside organizations are better placed to handle the management of internships for Debian, but that it does not have to be that way:

Chandran also thought that finding the right balance in terms of diversity spending is important:

Priorities

Carter's platform is the one with the most concrete ideas and plans, as he is not focused on one particular issue. He listed a half-dozen separate "community improvements" that he would like to help foster, including better onboarding for new contributors, increasing the number of local Debian groups, and promoting mentorship within the project so that those efforts are as highly regarded as, say, being a Debian Developer (DD) is. He also presented two plans under the heading of "improve reporting": more detailed and frequent financial reports, along with more "bite-sized updates" from the DPL that would get included into the usual monthly DPL report.

The item that drew the most attention, though, was the first community improvement Carter listed: "Initiate a public discussion on our membership nomenclature". The different types of project members have confusing names, he said; in particular, "Debian Developer" is used in a different way than terms like "Android developer". Other names, like "Debian Maintainer and non-uploading DD", are unwieldy and unhelpful to outsiders. He would like to see the project grapple with that, come up with new terms that make more sense, and have a GR to resolve the issue.

Gupta said that he was not really in favor of changing the DD name as part of his rebuttal. In addition, Sean Whitton felt that, since the nomenclature issue was the most concrete part of Carter's platform, it might give the impression that it was the only thing that he intended to pursue for sure as DPL. Whitton asked Carter to better describe his priorities, which Carter did at some length, starting with:

He also noted that another of his bullet points was about increasing online interaction possibilities, including perhaps having a regular online-only DebConf offset by six months from the usual in-person variety. He said that regardless of whether he is elected, he would like to pursue some of that:

There were, of course, other questions, ideas, and threads on debian-vote; interested readers (and voters) should poke into all of that further. One thing missing from the discussions is any real mention of the project harassment issue raised by Hartman in his non-platform; it would seem that the candidates are ignoring that particular elephant in the room, at least for now. This election cycle seems somewhat different than many, with two candidates focused on singular goals for the project and one who is, seemingly, looking to calm the waters within the distribution some. We will know fairly soon how the project feels about those options.

Comments (4 posted)

Avoiding retpolines with static calls

Indirect calls happen when the address of a function to be called is not known at compile time; instead, that address is stored in a pointer variable and used at run time. These indirect calls, as it turns out, are readily exploited by speculative-execution attacks. Retpolines defeat these attacks by turning an indirect call into a rather more complex (and expensive) code sequence that cannot be executed speculatively.

Retpolines solved the problem, but they also slow down the kernel, so developers have been keenly interested in finding ways to avoid them. A number of approaches have been tried; a few of which were covered here in late 2018. While some of those techniques have been merged, static calls have remained outside of the mainline. They have recently returned in the form of this patch set posted by Peter Zijlstra; it contains the work of others as well, in particular Josh Poimboeuf, who posted the original static-call implementation.

An indirect call works from a location in writable memory where the destination of the jump can be found. Changing the destination of the call is a matter of storing a new address in that location. Static calls, instead, use a location in executable memory containing a jump instruction that points to the target function. Actually executing a static call requires "calling" to this special location, which will immediately jump to the real target. The static-call location is, in other words, a classic code trampoline. Since both jumps are direct — the target address is found directly in the executable code itself — no retpolines are needed and execution is fast.

Static calls must be declared before they can be used; there are two macros that can do that:

    #include <linux/static_call.h>

    DEFINE_STATIC_CALL(name, target);
    DECLARE_STATIC_CALL(name, target);

DEFINE_STATIC_CALL() creates a new static call with the given name that initially points at the function target(). DECLARE_STATIC_CALL(), instead, declares the existence of a static call that is defined elsewhere; in that case, target() is only used for type checking the calls.

Actually calling a static call is done with:

    static_call(name)(args...);

Where name is the name used to define the call. This will cause a jump through the trampoline to the target function; if that function returns a value, static_call() will also return that value.

The target of a static call can be changed with:

    static_call_update(name, target2);

Where target2() is the new target for the static call. Changing the target of a static call requires patching the code of the running kernel, which is an expensive operation. That implies that static calls are only appropriate for settings where the target will change rarely.

One such setting can be found in the patch set: tracepoints. Activating a tracepoint itself requires code patching. Once that is done, the kernel responds to a hit on a tracepoint by iterating through a linked list of callback functions that have been attached there. In almost every case, though, there will only be one such function. This patch in the series optimizes that case by using a static call for the single-function case. Since the intent behind tracepoints is to minimize their overhead to the greatest extent possible, use of static calls makes sense there.

This patch set also contains a further optimization not found in the original. Jumping through the trampoline is much faster than using a retpoline, but it is still one more jump than is strictly necessary. So this patch causes static calls to store the target address directly into the call site(s), eliminating the need for the trampoline entirely. Doing so may require changing multiple call sites, but most static calls are unlikely to have many of those. It also requires support in the objtool tool to locate those call sites during the kernel build process.

The end result of this work appears to be a significant reduction in the cost of the Spectre mitigations when using tracepoints — a slowdown of just over 4% drops to about 1.6%. It has been through a number of revisions, as well as some improvements to the underlying text-patching code, and appears to be about ready. Chances are that static calls will go upstream in the near future.

Comments (45 posted)

Per-system-call kernel-stack offset randomization

The kernel stack will always be an attractive target. It typically contains no end of useful information that can be used, for example, to find the location of other kernel data structures. If it can be written to, it can be used for return-oriented programming attacks. Many exploits seen in the wild (Cook mentioned this video4linux exploit as an example) depend on locating the kernel stack as part of the sequence of steps to take over a running system.

In current kernels, the kernel stack is allocated from the vmalloc() area at process creation time. Among other things, this approach makes the location of any given process's kernel stack hard to guess, since it depends on the state of the memory allocator at the time of its creation. Once the stack has been allocated, though, its location remains fixed for as long as the process runs. So if an attacker can figure out where the kernel stack for a target process is, that information can be used for as long as that process lives.

As it turns out, there are a number of ways for an attacker to do that. Despite extensive cleanup work, there are still numerous kernel messages that will expose addresses of data structures, including the stack, in the kernel log. There are also attacks using ptrace() and cache timing that can be used to locate the stack. So the protection offered by an uncertain stack location is not as strong as one might like it to be.

Cook and Reshetova's patch set (which is inspired by the PaX RANDKSTACK feature, though the implementation is different) addresses this problem by changing a process's kernel stack offset every time that process makes a system call. Specifically, it modifies the system-call entry code so that the following sequence of events happens:

• The pt_regs structure, containing the state of the processor registers, is pushed onto the base of the stack, just like it is done in current kernels.
• A call to alloca() is made with a random value. This has the effect of "allocating" a random amount of memory on the stack, which is really just a matter of moving the stack pointer down by that amount.
• The system call proceeds with its stack pointer in the now randomized location.

In other words, the kernel stack itself doesn't move, but the actual stack contents shift around and are located differently for every system call. That makes any attack that depends on placing data at a specific location in the stack likely to fail; even if the attacker succeeds in figuring out where the stack is to be found, they won't know exactly where any given system call will place its data on that stack.

Pushing the pt_regs structure before applying the randomization is important. The ptrace() attack mentioned above can be used to locate this structure (and thus the kernel stack); if it were located after the offset is applied, such attacks would thus reveal the offset.

Currently, the randomization amount is obtained by reading some low-order bits from the CPUs time-stamp counter. Cook notes that other, more robust sources of entropy can be added in the future, but he doesn't think that needs to be figured out before the current patches can be considered. There are currently five bits of entropy applied to the stack offset on 64-bit systems, and six bits on 32-bit systems. That is not a huge amount of entropy, but it is enough that any attack that depends on precise kernel-stack locations will probably fail — and generate a kernel oops — on the first few tries. More entropy can be added, at the cost of wasting more stack space.

With this feature in use, Cook measured the overhead as being about 0.9% on a no-op system call; it would clearly be less on any system call that does real work. But for people who don't want to pay even that cost, there is a static label to turn the randomization off.

The end result is a relatively simple mechanism to further harden the kernel against attack. Cook noted that it's not perfect, adding that "most things can't be given the existing kernel design trade-offs". If other developers agree, per-system-call stack offset randomization is likely to find its way into the mainline kernel's arsenal of hardening techniques.

Comments (22 posted)

Some 5.6 kernel development statistics

As Linus Torvalds pointed out in the release announcement, the current coronavirus pandemic does not appear to have seriously affected kernel development — so far. One should not, though, lose track of the fact that the 5.6 merge window closed in early February, well before the impact of this disaster was broadly felt outside of China. Most of the work merged for 5.6 was done even earlier, of course. Given the delays involved in getting work into the mainline, the full effect may not be felt until the 5.8 cycle.

It goes without saying that we hope those effects are minimal, and that the people in our community (and beyond) come through this experience as well as possible.

Of the developers working on 5.6, 214 were first-time contributors. Many projects would be delighted to have that many new contributors in a nine-week period, but that is low for the kernel — the lowest since 3.11, which featured 203 first-time contributors and was released in September 2013. This dip does not appear to be part of a long-term trend:

It is possible that this drop is partly due to the current pandemic; a surprising number of first-time contributors show up late in the development cycle with bug fixes.

The most active developers contributing to 5.6 were:

Most active 5.6 developers

By changesets Takashi Iwai 406 3.2% Chris Wilson 306 2.4% Sean Christopherson 143 1.1% Jérôme Pouiller 125 1.0% Eric Biggers 122 1.0% Arnd Bergmann 114 0.9% Zheng Bin 110 0.9% Geert Uytterhoeven 103 0.8% Tony Lindgren 103 0.8% Masahiro Yamada 94 0.7% Colin Ian King 92 0.7% Ben Skeggs 91 0.7% Ville Syrjälä 90 0.7% Andy Shevchenko 88 0.7% Russell King 88 0.7% Alex Deucher 86 0.7% Krzysztof Kozlowski 82 0.6% Thomas Zimmermann 80 0.6% Jens Axboe 77 0.6% Jani Nikula 74 0.6%

By changed lines Kalle Valo 48483 7.2% Arnd Bergmann 29415 4.3% Jason A. Donenfeld 18664 2.8% Ben Skeggs 13471 2.0% Greg Kroah-Hartman 11931 1.8% Chris Wilson 10615 1.6% Srinivas Kandagatla 8739 1.3% Alex Maftei 8581 1.3% Maxime Ripard 7521 1.1% Peter Ujfalusi 6970 1.0% Tony Lindgren 6320 0.9% Helen Koike 5789 0.9% Takashi Iwai 5622 0.8% Shuming Fan 5604 0.8% Michal Kalderon 5445 0.8% Sricharan R 5065 0.7% Andrii Nakryiko 4857 0.7% Roman Li 4852 0.7% Thierry Reding 4845 0.7% Sunil Goutham 4762 0.7%

This time around, the developer with the most commits is Takashi Iwai, who did a bunch of cleanup and API-migration work in the sound subsystem. Chris Wilson worked exclusively on the i915 graphics driver, Sean Christopherson has, seemingly, been rewriting the KVM hypervisor from the ground up, Jérôme Pouiller worked on the wfx wireless network interface driver in the staging tree, and Eric Biggers contributed a lot of work to the filesystem and crypto subsystems.

Kalle Valo made it to the top of the "lines changed" column with just five commits; the one adding the ath11k network driver was large. Arnd Bergmann, among many other things, removed a set of obsolete ISDN drivers and more-or-less completed the task of readying the kernel for the year 2038. Jason Donenfeld added the WireGuard VPN subsystem, Ben Skeggs worked extensively on the nouveau graphics driver, and Greg Kroah-Hartman deleted the unloved octeon driver from the staging tree.

The credits for testing and reviewing patches look like this:

Test and review credits in 5.6
Tested-by Keerthy 61 7.6% Andrew Bowers 47 5.9% Aaron Brown 36 4.5% Peter Ujfalusi 21 2.6% Tero Kristo 20 2.5% Stephan Gerhold 20 2.5% John Garry 20 2.5% Brian Masney 18 2.2% Alexei Starovoitov 17 2.1% Steven Rostedt 15 1.9% Arnaldo Carvalho de Melo 15 1.9%	Reviewed-by Rob Herring 140 2.8% Alex Deucher 99 2.0% David Sterba 88 1.8% Andrew Lunn 87 1.7% Florian Fainelli 83 1.7% Tvrtko Ursulin 82 1.6% Linus Walleij 78 1.6% Chris Wilson 78 1.6% Tony Cheng 74 1.5% Laurent Pinchart 70 1.4% Andy Shevchenko 69 1.4%

Of the patches going into 5.6, 669 (5.3% of the total) carried Tested-by tags, a decline from 5.5. Reviewed-by tags, instead, appeared in 4,183 patches, 33% of the total.

There were 877 patches added for 5.6 that included Reported-by tags to credit the reporting of a bug; the most active reporters were:

Reporting credits in 5.6
Hulk Robot	178	18.6%
Syzbot	99	10.4%
kernel test robot	58	6.1%
Dan Carpenter	23	2.4%
Randy Dunlap	20	2.1%
Stephen Rothwell	15	1.6%
Linus Torvalds	7	0.7%
Marek Szyprowski	7	0.7%
Christoph Paasch	6	0.6%
Naresh Kamboju	6	0.6%
Dmitry Osipenko	5	0.5%
Ravi Bangoria	5	0.5%
Michael Ellerman	5	0.5%
Jann Horn	5	0.5%
Erhard Furtner	5	0.5%
Qian Cai	5	0.5%

We continue to see an increasing number of bug reports coming from automated testing systems; such reports now make up just over a third of the total.

The work on the 5.6 kernel was supported by 207 employers that we were able to identify, a significant decline from 5.5 (which had support from 231 employers). The most active employers were:

Most active 5.6 employers

By changesets Intel 1694 13.4% (Unknown) 904 7.1% AMD 781 6.2% (None) 778 6.1% SUSE 713 5.6% Red Hat 702 5.5% Google 558 4.4% Linaro 503 4.0% Huawei Technologies 483 3.8% Facebook 298 2.4% Mellanox 252 2.0% Renesas Electronics 247 2.0% IBM 232 1.8% Arm 231 1.8% Code Aurora Forum 222 1.8% (Consultant) 216 1.7% Texas Instruments 213 1.7% NXP Semiconductors 210 1.7% Oracle 147 1.2% Broadcom 143 1.1%

By lines changed Intel 78083 11.5% Code Aurora Forum 68538 10.1% Linaro 59492 8.8% AMD 44979 6.6% Red Hat 40553 6.0% (Unknown) 28591 4.2% (None) 27387 4.0% (Consultant) 23271 3.4% Google 20038 3.0% SUSE 19274 2.8% Facebook 17525 2.6% Texas Instruments 16561 2.4% Mellanox 14977 2.2% Linux Foundation 12289 1.8% Marvell 11678 1.7% Realtek 10968 1.6% Collabora 9491 1.4% NXP Semiconductors 8689 1.3% Solarflare Communications 8670 1.3% IBM 8586 1.3%

We have reached the point where a full one-eighth of the patches coming into the kernel originate from within Intel. For years, Red Hat was the top contributor of changesets, but its position has been slowly falling for some time; this may be the first time that SUSE contributed more patches during a development cycle. Otherwise, these numbers look about the same as they usually do.

If one looks at Signed-off-by tags applied to patches that were written by somebody else, the picture changes a bit:

Non-author signoffs in 5.6
Developers David S. Miller 1162 10.1% Alex Deucher 748 6.5% Greg Kroah-Hartman 653 5.7% Mark Brown 445 3.9% Paolo Bonzini 271 2.4% Kalle Valo 239 2.1% Herbert Xu 236 2.1% Andrew Morton 220 1.9% Mauro Carvalho Chehab 213 1.9% Alexei Starovoitov 188 1.6%	Employers Red Hat 2423 21.1% Linaro 1213 10.6% AMD 786 6.9% Intel 763 6.7% Google 746 6.5% Linux Foundation 701 6.1% Facebook 395 3.4% SUSE 390 3.4% (None) 351 3.1% Mellanox 296 2.6%

When a developer adds a Signed-off-by tag to somebody else's patch, it (usually) means that said developer is routing that patch toward the mainline, usually by applying it to a subsystem repository. These signoffs thus give some visibility into who the kernel's gatekeepers are. David Miller, the maintainer of the networking subsystem, has kept that top position for years. The presence of other developers indicate that there continues to be a lot of activity in the AMD graphics, device support, and KVM subsystems, among others.

In the right-hand column we see that, while the percentage of patches coming from Red Hat has dropped over the years, over 20% of the patches getting into the mainline still pass through the hands of Red Hat developers.

The first time that LWN looked at signoff statistics was for the 2.6.22 development cycle in 2007. At that time, the top gatekeepers were Andrew Morton and Linus Torvalds, neither of whom handles vast numbers of patches now; the third place was held by David Miller. Four of the top-ten maintainers in 2007 are still in the top ten now. Similarly, five of the top-ten companies were in the top ten 13 years ago too (if one deems the 2013 Novell to be the same as the 2020 SUSE).

All told, the picture that emerges indicates that it's mostly business as usual in the kernel community. The flow of patches continues at a steady rate and the number of developers remains large. The makeup of the community changes — slowly — but the process of cranking out kernels continues uninterrupted.

Comments (1 posted)

Reworking StringIO concatenation in Python

Python string objects are immutable, so changing the value of a string requires that a new string object be created with the new value. That is fairly well-understood within the community, but there are some "anti-patterns" that arise; it is pretty common for new users to build up a longer string by repeatedly concatenating to the end of the "same" string. The performance penalty for doing that could be avoided by switching to a type that is geared toward incremental updates, but Python 3 has already optimized the penalty away for regular strings. A recent thread on the python-ideas mailing list explored this topic some.

Paul Sokolovsky posted his lengthy description of a fairly simple idea on March 29. The common anti-pattern of building up a string might look something like:

buf = ""
for i in range(50000):
    buf += "foo"
print(buf)

buf = io.StringIO()
for i in range(50000):
    buf.write("foo")
print(buf.getvalue())

To make it easier for existing programs to be switched from one form to the other, he suggested adding a "+=" operator for StringIO as an alias for the write() method. Adding an __iadd__() method for the StringIO class would allow the write() call to be removed in favor of using +=. The buffer initialization and getvalue() call would still be needed, but those are each typically done in only one place, while the concatenation may be done in multiple places. So a code base could fairly easily be switched from the anti-pattern to more proper Python just by creating the buffer instead of a string and getting its value where needed with getvalue(), the rest of the code can stay the same; "it will leave the rest of code intact, and not obfuscate the original content construction algorithm".

As Sokolovsky noted, his performance benchmarking shows that CPython 3 has already optimized for the anti-pattern, though. So even though it is still considered to be a bad practice, there is no real penalty for writing code of that sort in CPython 3—but only for that version of the language:

The optimization, which is described by Paul Ganssle in a blog post, effectively allows CPython to treat the string as mutable in the case where there are no other references to it. In a loop like the one in the example, there is no other reference to the string object being used, so instead of creating a new object and freeing the old, it simply changes the existing object in place. CPython can detect that case because it uses reference counts on its objects for garbage collection; PyPy is not reference-counted, so it cannot use the same trick.

But Sokolovsky is trying to target the bad practice regardless of the (lack of a) performance impact. The practice is widespread; the optimization added to CPython is evidence that it needs addressing, he said. He suggested that other implementations can either follow the lead of CPython (if possible) or try to promote better practices: "This would require improving ergonomics of existing string buffer object, to make its usage less painful for both writing new code and refactoring existing." And, of course, he was advocating the latter.

He also noted that, since the performance problem does not really exist for CPython, it might be seen as an argument that there is nothing to fix. "This is related to a bigger [question] 'whether a life outside CPython exists', or put more formally, where's the border between Python-the-language and CPython-the-implementation." Beyond that, one could "fix" the problem by creating a new class derived from StringIO that has an __iadd__(), but that suffers from worse performance as well, which argues that the problem should be addressed in C in StringIO itself.

The overall reception to the idea was chilly, at best, perhaps partly fueled by Sokolovsky's somewhat aggressive tone in his original note and some of the followups. Andrew Barnert replied that the join() mechanism is really the better alternative:

Barnert said that StringIO is meant to be a file object that resides in memory, so it is appropriate that its API does not support +=. He concluded with a third option for alternative Python implementations beyond the two that Sokolovsky presented:

The problem with the join() mechanism is that it is somewhat non-intuitive, especially for those coming to Python from another language. As Barnert noted, though, it can use more memory as well. Sokolovsky attempted to measure the difference in memory use, but the technique he used was not entirely convincing. His focus would appear to be on embedded Python, such as his Pycopy Python implementation. Pycopy is descended from MicroPython, which he also worked on. For the embedded use case, StringIO may well be the better choice for building strings, at least from a memory perspective; is that enough of a reason to turn a file-like object (StringIO) into a string-like object, but only for concatenation (+=)? The consensus answer would seem to be "no".

There was some discussion of having a generalized mutable string type, though that was not at all what Sokolovsky was after; there are some good reasons why that idea has never really taken off for Python, as Christopher Barker described. "So I'd say it hasn't been done because (1) it's a lot of work and (2) it would be a bit of a pain to use, and not gain much at all."

The objections to the original idea are basically that += can be trivially implemented for a derived class of StringIO; if the performance of that is not sufficient, switching to join() would fix that problem. The existing "join() on a list of strings" idiom works well for most people and nearly all use cases; it is the preferred way to solve this problem in Python, so making another idiom more usable is muddying the water to a certain degree. As The Zen of Python puts it: "There should be one-- and preferably only one --obvious way to do it."

On the other hand, CPython is the dominant player in the ecosystem, as Steven D'Aprano pointed out; that means applications can be written to take advantage of CPython quirks. On the other hand, even if all of the other Python implementations agreed on a change, it will not really be used unless CPython follows suit.

That is something for the CPython community to keep in mind. The existence of the other implementations of the language may provide opportunities to make some changes that are meant to be CPython-only (or at least not mandated for Python the language). But those changes can still get baked into the language via the back door—because most Python code runs on CPython.

In the final analysis, it is a pretty miniscule change being sought. The existence of the string concatenation optimization indicates that there is interest in helping "badly written" code to some extent, but perhaps adding += to StringIO is a bridge too far. There definitely does not seem to be any kind of groundswell of support for the idea and there are costs, beyond just the (minimal) code maintenance required, including in documentation and user education. The benefits, which some find to be dubious to begin with, are seemingly not enough to outweigh them.

Comments (38 posted)