LWN.net Weekly Edition for October 31, 2019

Welcome to the LWN.net Weekly Edition for October 31, 2019

This edition contains the following feature content:

• Unifying kernel tracing: making peace among tracing projects with a common infrastructure layer.
• Rethinking the governance of the GNU Project: GNU developers are looking to change how the project is run.
• Redesigned workqueues for io_uring: a new low-level asynchronous work mechanism for the kernel.
• Type checking for BPF tracing: closing a big loophole in the BPF verifier.
• The return of Python dictionary "addition": a controversial Python improvement moves forward.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Unifying kernel tracing

Steven Rostedt has been a part of the Linux kernel tracing community for most of its existence, it seems. He was the developer of ftrace, which was one of the early mainline additions for tracing. There are now many tracing facilities in the kernel. At the 2019 Open Source Summit Europe in Lyon, France, Rostedt wanted to present an idea that he has been thinking about for a long time: a unified tracing platform to provide access to all of the kernel tracing facilities from user-space applications.

There are "many faces of tracing" for Linux, including perf, LTTng, SystemTap, DTrace, BPF, ktap, strace, GDB, ftrace, and "there's plenty more", he said. He began his presentation with a look at the history of Linux tracing, touching on a number of those projects.

Tracing history

Ftrace actually started back when Rostedt was working on his Master's thesis in 1998. He created "logdev" to print messages to a ring buffer in a way that was faster than printk(). He was investigating network quality of service (QoS) behavior and printk() was simply too slow. He got a job at Timesys after that and needed a faster way to get information out of that company's realtime kernel; he was easily able to get logdev working on that kernel. His first version of logdev was for the 2.0 kernel; he ported it to 2.2, 2.4, and so on as they became available. He actually used logdev to debug ftrace as he was developing it; logdev is one of the two parents of ftrace.

The realtime patch set had a latency tracer that is the other parent. It was created in 2004 by Ingo Molnar and Nadia Chambers. The latency tracer used a GCC profiling feature that would call an mcount() routine at the beginning of every function, which could be used to trace the execution of the kernel. The latency tracer was not dynamic, as function tracing could not be controlled at runtime; it was either compiled in or not.

In 2008, Rostedt was at Red Hat on the realtime team, which was working on getting the realtime patches upstream. Arnaldo Carvalho de Melo tried to port the latency tracer to the upstream kernel, but the tracer was quite complex and hard to follow; it called trace(), which called _trace() and so on up to _____trace() (five underscores). Rostedt was familiar with the code, so he cleaned it up, overhauled the latency tracer to use logdev, and turned all of that into ftrace.

Next came perf, which started as a "profiling utility done correctly"; it came out of the flame wars that surrounded oprofile and other profiling solutions, he said. In addition to profiling, perf had some tracing features that it got by hooking into the ftrace infrastructure. Molnar wanted Rostedt to stop developing ftrace and to move all of that effort into perf, but Rostedt resisted that push in part because he wanted to keep ftrace's debugfs interface. No user-space tool was needed to use ftrace via debugfs, which made it useful for embedded Linux systems that did not have build tools; just using echo from BusyBox was enough to enable tracing and a simple cat could show the tracing results.

Before LTTng, there was the Linux Trace Toolkit (LTT) that was created around 2000 by Karim Yaghmour. It was the first real attempt to get a tracing solution into the mainline, but it ran aground on objections from Molnar and Linus Torvalds.

The "next generation" (ng) of LTT was actually a complete rewrite done by Mathieu Desnoyers in 2006. Rostedt noted that Desnoyers has said that it was a mistake to keep LTT in the name since it shared little with that code base and was confusing to those who thought it was simply an extension of LTT. While LTTng never made it upstream and has been maintained over the years as an out-of-tree patch set, Desnoyers was able to get tracepoints into the mainline; the trace events used by ftrace are built on top of tracepoints.

DTrace was the one of the first tracing tools to allow scripting in the kernel. It was originally for Solaris and was officially released in 2005. After Oracle acquired Sun, the company ported DTrace to Linux, but did not make it available under the GPL until 2017. That was too late for most, so DTrace is only available on Oracle Linux (and perhaps a few others). It is interesting to note that some say DTrace came from IBM's dprobe work, which was much earlier and shipped with SUSE Linux Enterprise Server (SLES) kernels at that time, Rostedt said. Everyone builds on each others' work in the grand tradition of "on the shoulders of giants".

Red Hat's answer to DTrace was SystemTap, which was released in 2009. Like some of the other solutions, it never made it into the mainline kernel. Part of the problem with getting SystemTap, LTTng, and others upstream is that those projects focused on the tracer side but never sold other developers on the usefulness of those tools.

Ftrace was not written as a tracing tool, per se, but it was part of the effort to get the realtime patches upstream, which needed the ftrace facilities for specific purposes. He worked with Desnoyers to get tracepoints into the kernel, then started making it easy for other developers to add trace events to their subsystems. The macros used to create trace events are horrifying to look at but easy to use. So ftrace got its foot in the door, then expanded as other kernel developers started to see its utility. Now there are over 1000 trace events in the mainline kernel.

A developer could simply define a trace event using the macro and it would appear in the tracefs hierarchy. A simple echo 1 into that file would enable the event, which would start showing up in the tracing output. Once kernel developers realized how easy it was, they started being much more interested in tracing and wanted to see more tracing features added. At the 2008 Kernel Summit, Torvalds proclaimed that SystemTap was far too complicated and that the simple tracing tools already in the kernel should be expanded further instead. Simplicity was the goal of the early tools, Rostedt said.

Then came BPF. A just-in-time (JIT) compiler was added for BPF on x86 in 2011, which made customized network-packet filtering extremely fast. In 2014, Alexei Starovoitov introduced extended BPF (eBPF) that allowed BPF programs to be applied to other areas of the kernel, including tracing. Tracing with eBPF provides Linux with the capabilities that DTrace and SystemTap were targeting. And now, both DTrace and SystemTap are being reworked to use eBPF under the covers so that existing scripts for those tools will function on mainline kernels.

Status

He referred to a 2017 tweet from Julia Evans that described the state of tracing on Linux, which she turned into a lengthy blog post; all of that information is still pretty accurate today, Rostedt said. He is often asked "why can't we have just one tracer?" Some people seem to think that too much choice is a problem but his suggestion for those people is a switch to macOS

Part of the complaint about having multiple tracers is that it is splitting the development effort, but he believes that "diversity brings innovation". If the Bell System had not been broken up back in the 1980s, which brought choice and diversity to the telephone world, we might still all be using rotary phones—a device that many don't even recognize today.

There is no "one size fits all" in tracing (or much of anything else); he pointed to "TABs versus spaces" and "Emacs versus Vim" as two classic examples. Tracing users often have different objectives. A dump truck could be used to travel between cities, but a sports car or motorcycle might be more appropriate; there are different needs at different times so it is nice to have a diversity of tools to address them.

Some complain that diversity killed Unix, he said; there were too many flavors of Unix, which led to its demise. But those Unix flavors were all proprietary, so they could not share features. In the proprietary world, forks are bad, but in the open-source world, forks are good, he said. Those forks allow people to try things out in different, unexpected ways; if the outcome is useful, it can folded back into the original. "Diversity is the strength of open source", he said, "because we can always share".

Commonalities

There are a number of pieces that are shared between the tracers. Tracepoints are being used by all of the different solutions, as are kernel probes (kprobes) and user-space probes (uprobes). The ftrace function hooks are also being used by many of the tracers. He showed the diagram above (from his slides [PDF]) to demonstrate the pieces that are shared within the world of Linux tracers. The top half, above the dotted line, contains the user-space pieces, while the lower part has all of the various kernel pieces.

His idea is to have a single user-space library that can be used by all of the different tracers to interface to all of the different kernel facilities. It can be seen in the diagram below as the large green box in user space—with a bit of a typo in its label. Effectively, each of the tools would have access to the capabilities used by the others, so the dump truck, sports car, and motorcycle of the tracing world can "unleash the power of all the infrastructure that's in the kernel".

There has already been some progress in that direction, Rostedt said. Babeltrace is a library that came out of the LTTng project. It can convert between multiple different tracing-file formats and the Common Trace Format (CTF). The goal is to allow any tracing utility to read and use the data handled by any other.

Another entry is libtraceevent, which is now ready to be packaged and distributed, Rostedt said; he is looking for distribution package maintainers to help out with that. It lives in the kernel source tree (in tools/lib/traceevent). When raw trace event data is written by the kernel, the format of that data is also exported in tracefs; that information is used by perf, PowerTOP, trace-cmd, and others. All of those tools copied the code that he wrote to parse the format information and access the data accordingly. He suggested that those projects copy the code, but now that it is available in library form, the code copies should be removed in favor of linking with libtraceevent.

The libperf library is available to wrap the perf_event_open() system call. That call is effectively an ioctl() with "hundreds of commands" that is complex to use. Libperf provides a more sane interface to perf_event_open(). It is still a new project, but it will help applications gain access to the power underlying perf.

There are also some libraries "coming soon", he said. Libftrace is an interface to the tracefs directory; it will allow users to start and stop tracers, enable and disable events, create kprobes and uprobes, and read the raw tracing data. Libtrace-cmd is a higher-level library that will allow other applications to do anything that trace-cmd can do. Similarly, libkshark is a GUI library that gives applications the ability to do anything that KernelShark does. That means the capabilities of KernelShark are not limited to only handling ftrace data; other tracers can use the features of the GUI tool to work with their own data.

That new box in the diagram would encompass these libraries and others to form a unified tracing platform for Linux. It would be an abstraction layer over all of the disparate capabilities that the Linux kernel provides for tracing. He suggested that the "Unix way" of "do one thing and do it well" is evolving into the "Linux way" of "do one library and do it well".

The tracing tools are not competing with each other anymore, Rostedt said. There were the tracing wars of days gone by, which were good because they fostered innovation, but there is no reason to keep warring. The tracing projects can take the best ideas from each other and meld them into their tools in various ways, which might lead to electric sports cars, then electric motorcycles and dump trucks. He concluded by saying that the ability to do that kind of thing is what makes Linux the best operating system in the world.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to attend Open Source Summit Europe in Lyon, France.]

Comments (14 posted)

Rethinking the governance of the GNU Project

The GNU Project was created by Richard Stallman in 1983 to further his goal of developing an entirely free operating system — a goal that seemed impossibly ambitious at the time. Stallman has recently resigned from some of his roles, but as of this writing his personal site still leads off with this proclamation: "I continue to be the Chief GNUisance of the GNU Project. I do not intend to stop any time soon". Within the project itself, though, it has become clear that this intention lacks universal support. We appear to be seeing the beginning of a governance transition for this venerable project.

To many, Stallman's departure from the Free Software Foundation and MIT appears to be an abrupt development based on behavior outside of the technical or project-management areas. Those reasons are mostly out of scope for this article (and for any comments), but there is one thing that is worth pointing out: the concerns that led to these changes have existed for many years. As is often the case, they came to a climax quickly, but the situation had been developing for years.

While these concerns certainly play into why there is pressure for change from within the GNU Project, there is more to it than that. Some recent events highlight the fact that some maintainers feel that change is needed; they have more to do with Stallman's leadership within the project than his behavior outside of it.

The GNU C Library manual

In mid-2018, the glibc community endured a series of discussions and events regarding a joke in the documentation for the abort() function in the manual. A number of glibc developers felt that the joke was confusing to many, offensive to some, and unhelpful at best; after some discussion it was removed. Stallman, however, claimed the absolute authority to make decisions regarding changes like that, and he called for the patch to be reverted — which was duly done by Alexandre Oliva. The episode left bad feelings with many who were involved, and who thought that they were the developers and maintainers of glibc.

By common agreement, this discussion was allowed to go quiet for some time. But that ended in early October, when Florian Weimer posted a patch to the project's mailing list, once again removing the joke. Once again, Stallman opposed its removal, at least anytime in the near future:

As the conversation went on, Stallman made it clear that he still claimed authority over the glibc project (and all other GNU projects as well): "I directly appoint only package maintainers, who are then responsible directly to me". If those maintainers make a decision that he disagrees with, he feels that it is his right to override them.

That attitude appears to be increasingly unpopular within some GNU projects, though. In this case, glibc maintainer Carlos O'Donell replied to Stallman's demand for a delay with an outright refusal:

O'Donell told Weimer that he could go ahead and commit the change; that was done on October 11, bringing a long chapter to a close — and demonstrating a clear limit on how much authority Stallman has over the project at this point.

GNU going forward

Stallman has always been a divisive figure, both within the GNU Project and beyond. Some followers appear to have a nearly messianic devotion; others do not consider themselves to be followers at all. What seems undeniable is that his authoritarian approach has often led to this sort of conflict, and many see his decisions as having held the GNU Project back. It is telling that many developers, even those who are strongly committed to free software, do not wish to be associated with him.

In early October, a number of high-profile GNU maintainers signed a statement calling for new leadership for the project. In retrospect, perhaps the only surprising aspect of this statement is that it was so long in coming, given how long these problems have existed. More recently, O'Donell announced the designation of a moderated mailing list to discuss the governance of the GNU Project; that discussion is just getting started.

There can be no doubt that Richard Stallman has made many great contributions to our community. He did not invent free software, but his conceptualization of the free-software ideal and his determination that we could develop software that is under our control may well have saved us from an all-proprietary world in the last century. Even 36 years after the creation of the GNU Project, he undoubtedly has more to contribute. But it seems increasingly clear that the maintainers within the project he founded feel that the nature of his contribution should change, and that the GNU project should be something other than an autocracy. Whether the community can successfully navigate this change may have a huge effect on the shape of free software in the future.

Comments (36 posted)

Redesigned workqueues for io_uring

The io_uring mechanism is a relatively new interface for asynchronous I/O; it first appeared in the 5.1 kernel in May. Since then, though, it has quickly grown in capabilities and in users; now it appears that it is outgrowing some of the kernel infrastructure that supports it. Thus, we have a proposal from Jens Axboe (the io_uring maintainer) for a new workqueue subsystem for io_uring that hints at some interesting plans for the future.

Workqueues are used by many kernel subsystems to run work asynchronously in process context. Over the years, workqueues have been extensively tweaked to provide the features needed by the kernel and to keep queued work requests from running concurrently on the same processor and contending with each other for CPU time. They have been relatively stable for a while, indicating that they do what is needed most of the time.

The io_uring mechanism is all about allowing user space to create asynchronous threads of execution, so it's not surprising that workqueues are extensively used there. Over time, though, some of the limitations of workqueues have become apparent in this context. Workqueues are, to a great extent, about ceding control over where and when the work functions are executed, but io_uring would benefit from a higher degree of control over how that work is done. Thus, the new mechanism, called "io-wq".

One of the new workqueues (an "io_wq") is created with:

    struct io_wq *io_wq_create(unsigned concurrency, struct mm_struct *mm);

where concurrency is the maximum number of worker threads that can be running on any given NUMA node, and mm is the memory-management context associated with this queue. In io_uring, one of these workqueues will be created for each io_uring_setup() call, and mm will point to the calling process's mm_struct structure. Associating the mm_struct with the io_wq in this way makes a number of the memory-management issues easier; the existing workqueue mechanism does not maintain this association, even when private workqueues are created.

The io_wq may be destroyed by passing it to io_wq_destroy().

To defer work to an io_wq, one starts by filling out one of these structures:

    struct io_wq_work {
	struct list_head list;
	void (*func)(struct io_wq_work **);
	unsigned flags;
    };

The main thing to do is to set func to the function that should be called to actually do the work; flags should be set to zero. The item can then be queued with either of:

    void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work);
    void io_wq_enqueue_hashed(struct io_wq *wq, struct io_wq_work *work, 
			      void *val);

A call to io_wq_enqueue() adds the work to the queue for future execution. The io_wq_enqueue_hashed() variant, instead, is one of the reasons for the creation of new mechanism; it guarantees that no two jobs enqueued with the same val will run concurrently. If an application submits multiple buffered I/O requests for a single file, they should not be run concurrently or they are likely to just block each other via lock contention. Buffered I/O on different files can and should run concurrently, though. "Hashed" work entries make it easy for io_uring to manage that concurrency in an optimal way.

Passing an io_wq to io_wq_flush() will cause the calling thread to block until all pending work items have left the queue. Note that this does not mean that those items have completed, only that they have started.

Cancellation is another motivation for io-wq. The io_uring mechanism has to allow user space to cancel pending requests, meaning that it must be possible to cancel io-wq work requests in a predictable way. In current kernels, cancellation of requests on network sockets can occasionally lead to deadlocks; users tend to find this kind of behavior less amusing than one might think, so a better solution is needed. The new cancellation functions are:

    void io_wq_cancel_all(struct io_wq *wq);
    enum io_wq_cancel io_wq_cancel_work(struct io_wq *wq,
    					struct io_wq_work *cwork);

A call to the first function will cancel all outstanding operations on the given io_wq; the second one will cancel only the specified work request. Either way, this is done by sending a SIGINT signal to each running worker thread; the function will return after the signals have been sent without waiting for the worker threads to respond. For io_wq_cancel_work(), the return value will be IO_WQ_CANCEL_OK (the request was canceled before starting), IO_WQ_CANCEL_RUNNING (the request was running and the signal was sent), or IO_WQ_CANCEL_NOTFOUND (the request wasn't found, meaning it had already completed).

That is about it for the io-wq API. It is not clear that there would be benefits for any other kernel subsystem to move to this mechanism, so io_uring may remain the only user for some time. An improved io_uring will be enough for many users to celebrate, though.

That said, there may be more coming. Long-time LWN readers may remember a series of discussions in 2007 for an in-kernel mechanism called, at times, fibrils, threadlets, or syslets. Regardless of the name, this mechanism was intended to improve asynchronous I/O support, but there was another motive as well: to allow user space to run any system call asynchronously. None of those mechanisms reached a point of being seriously considered for merging, but it seems that they were not forgotten. In patch two of the series, Axboe notes that using io-wq in io_uring "gets us one step closer to adding async support for any system call". It thus seems that we can expect io_uring to develop the capabilities that were envisioned almost 13 years ago. Stay tuned for further developments.

Comments (5 posted)

Type checking for BPF tracing

The BPF in-kernel virtual machine has brought a new set of capabilities to a number of functional areas in the kernel, including, significantly, tracing. Since BPF programs run in the kernel, much effort goes into ensuring that they will not cause problems for the running system; to that end, the BPF verifier checks every possible aspect of each BPF program's behavior to ensure that it is safe to run in the kernel — with one notable exception. With a patch set titled "revolutionize bpf tracing", Alexei Starovoitov aims to close that loophole and eliminate a set of potential problems in a widely used class of BPF programs.

BPF is heavily used in tracing applications to gain access to useful kernel information and to perform data aggregation in kernel space. There are two variants of these programs. If a tracepoint has been placed in a useful location in the kernel, a BPF program can be attached there; otherwise, a kprobe can be placed at (almost) any kernel location and used to trigger a BPF program. Either way, the BPF verifier currently has little visibility into the data that will be passed to those programs.

Consider, for example, the trace_kfree_skb tracepoint placed in net_tx_action(). When this tracepoint triggers, any handlers (including attached BPF programs) will be passed two pointers, one to the sk_buff structure representing the network packet of interest, and one to the function that is freeing that packet. The type information associated with those pointers is lost, however; the program itself just sees a pair of 64-bit unsigned integers. Accessing the kernel data of interest requires casting those integers into pointers of the correct type, then using helpers like bpf_probe_read() to read the data behind those pointers. A series of bpf_probe_read() calls may be needed to walk through a data structure and get to the data the tracing program is actually looking for.

The problem is that a BPF program can cast one of these values into any type it likes; the result need not correspond to the actual type of that data. A mistake could cause a BPF program to go off into the weeds; in one worst-case scenario, the program could wander into a memory-mapped I/O area and cause some real damage. This isn't generally a security issue, since tracing is a privileged operation to begin with, but it is a safety issue — exactly the sort of issue that the BPF verifier is meant to prevent.

This problem has existed since the kernel first gained the ability to attach BPF programs to tracepoints and kprobes. Meanwhile, BPF developers have been working on an entirely different problem: the lack of binary portability for BPF programs. These programs go digging around in kernel data structures, but the layout and content of those structures varies depending on the kernel configuration, the underlying architecture, and more. The data of interest to any given program may be located 12 bytes into a structure on one kernel, but only 8 bytes into that structure on a different kernel. Without the ability to "relocate" these references, BPF users must rebuild their programs on every target system.

The "compile once run everywhere" effort has, over the last couple of years or so, worked to address this problem through the creation of a compact, machine-readable description of the kernel's data structures. This "BPF type format" (BTF) data is provided by the kernel itself, but it can be used by user-space support libraries to adjust a binary BPF program for a local kernel before loading it, mostly solving the binary portability issue. But it turns out that BTF information has other uses as well.

In particular, it is possible to annotate tracepoints with information about the types of the data values passed to handler programs. That allows the verifier to ensure that those programs are working with the correct data types. It also makes it possible for the C handler programs to follow pointers directly; when those programs are compiled to BPF and loaded into the kernel, the verifier can implicitly substitute the bpf_probe_read() calls where they are needed — after performing the necessary type checking, of course.

The end result of all this will be BPF tracepoint handler programs that are safer and far less error-prone to write. Whether tracing is "revolutionized" remains to be seen, but it is clearly improved in a significant way.

What is decidedly not revolutionized is data access within kprobe handlers. A kprobe can be set anywhere in the running kernel, and it is given access to the contents of the processor registers when the probe is hit. It is not, at this point, possible for the verifier to know what will be in those registers at that time, so this kind of checking cannot be done. That means that, especially in the parts of the kernel that are not amenable to the addition of proper tracepoints, the use of BPF programs without this sort of type checking will have to continue.

That said, progress is progress, and this work will increase the safety of much of the tracing code that is currently in use. It has been queued in the bpf-next tree so, barring some sort of last-minute hitch, it can be expected to show up in the 5.5 kernel.

Comments (11 posted)

The return of Python dictionary "addition"

Back in March, we looked at a discussion and Python Enhancement Proposal (PEP) for a new dictionary "addition" operator for Python. The discussion back then was lively and voluminous, but the PEP needed some updates and enhancements in order to proceed. That work has now been done and a post about the revised PEP to the python-ideas mailing list has set off another mega-thread.

PEP 584 ("Add + and += operators to the built-in dict class") has gotten a fair amount bigger, even though it has lost the idea of dictionary "subtraction", which never gained significant backing the last time. It also has two authors now, with Brandt Bucher joining Steven D'Aprano, who wrote the original PEP. The basic idea is fairly straightforward; two dictionaries can be joined using the "+" operator or one dictionary can be updated in place with another's contents using "+=". From the PEP:

    >>> d = {'spam': 1, 'eggs': 2, 'cheese': 3}
    >>> e = {'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> d + e
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> e + d
    {'cheese': 3, 'aardvark': 'Ethel', 'spam': 1, 'eggs': 2}
    
    >>> d += e
    >>> d
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

As can be seen, it is effectively an "update" operation (similar to using the .update() method) where the last value for a particular key "wins". That is why "cheese" is "cheddar" for d + e, but it is 3 for e + d. The example also shows that the operation is not commutative, which bothered some commenters even though there are already several such "arithmetic" operators that are not commutative; list "addition" using "+" isn't either, for example.

There were some objections to removing subtraction, some +1 and -1 responses, and others along those lines, but the biggest chunk of the thread was taken up by the question of how to "spell" the operator. The question seems to boil down to whether to use "|" instead of "+"; that was also part of the discussion back in March and is mentioned in the PEP as well. The operation is seen by some as being analogous to the set union operation, which uses "|".

Richard Musil kicked off a big sub-thread by making the argument for the set-union usage, though he suggested an entirely new operator ("|<") for it. He is concerned about the ambiguity of the + operator in Python and that choosing something completely new will ensure that users do not guess incorrectly about what it does. Chris Angelico did not see things that way, however:

But Paul Moore is unsure that there is any real need for a new dictionary addition operator:

He said that he has never needed that kind of operator and suggested that someone do a survey of real-world Python code to see if it would be improved using the new operator, though he did admit to not following the debate closely. It turns out that a big chunk of the PEP is taken up by examples of how the new operator might be used, taking examples from third-party code (including SymPy and Sphinx). Moore was not entirely impressed with them, however, saying that only four out of the roughly 20 examples were improved with the switch, though another few were arguable.

Andrew Barnert thought that Moore's observation actually made a good argument in favor of the proposal; if those who are not in favor of the proposal think that roughly a quarter of the examples are an improvement using it, that's a pretty strong vote in its favor. Beyond that, though, he thinks the need for + (which he calls "copying update") makes for a compelling case, more so than just for the += operator ("mutating update"):

With the "{**a, **b}" example, he is referring to using the dictionary unpacking operator, "**", which is specified in PEP 448 ("Additional Unpacking Generalizations"), to do the update operation. While that "works", it suffers from the drawbacks he mentions; it is also a fairly universally disliked language idiom. Most are fine with the "**" operator itself, but using it in that way is considered rather non-obvious and is quite unpopular.

D'Aprano pointed out that adding two dictionaries using + has come up frequently over the years, seemingly independently; "to many people, the use of + for this purpose is obvious and self-explanatory". The thread continued with some arguing for each spelling of the operator; in some sense, the arguments often came down to "taste". There were also some more exotic ideas (spellings other than + or |, providing a "did you mean ... ?" kind of error for + to lead users to |, and so on), but Guido van Rossum said he is "not crazy" about the "did you mean ... ?" idea; he indicated that he sees the field as already having been narrowed down:

Beyond that, his preference would be to use |, but he is not completely opposed to +:

While the discussion went on at length, no real consensus was reached. As is always the case in the Python world, though, it seems, the discussion was never heated or even contentious really; in the end it comes down to personal preferences. As D'Aprano put it, even if the PEP "fails", it will have succeeded at some level:

One would guess that the discussion will move from python-ideas to python-dev before too long and then likely to the steering council for some kind of decision. We know how one member of the council (Van Rossum) is leaning at this point, but we'll have to wait and see how the rest of that group feels as none have been active in the discussion. It seems like a reasonable "addition" to the language, however spelled, though using + seems more likely to head off newbie queries. Lists and dictionaries are much more integral to Python; those who are new to the language will probably see list "addition" well before they ever meet sets.

Comments (21 posted)