2015 Google Summer of Code organizations announced

Google announced the approved mentoring organizations for the 2015 Google Summer of Code (GSoC) program on March 3. Those organizations are the projects that will work with the 2015 batch of GSoC students, giving each student a paid internship—and giving the projects an extra full-time developer for a few months. One notable change in this year's lineup is that there is an overall reduction in the number of approved organizations—and some of the organizations absent from the 2015 list have been mainstays of previous GSoC programs. Still, more than a quarter of the included organizations have never been in any previous GSoC "class," which means that there will be the opportunity for interested students to get involved in some new parts of the open-source-software landscape.

The list of accepted mentoring organizations is available in an online spreadsheet-style document (which can be exported as a CSV file). 137 organizations are included for 2015, out of 416 applicants. The blog announcement notes that 37 of these organizations are new to GSoC.

Since GSoC has developed over the years into a reliable source for developers—more than 1,300 students participated in 2014—projects could come to view the program as a regular part of their development plans. But projects must re-apply to be mentoring organizations every year, and the 2015 list omits a few organizations that have been frequent participants. Mozilla, the Linux Foundation, and Tor have all made multiple appearances in the past (Tor often in concert with the Electronic Frontier Foundation), but are not on the 2015 roster.

The GSoC office does not publish any record of which organizations apply to be mentors but are not selected. However, Mozilla's Florian Quèze appeared to be at least a little surprised about Mozilla's lack of inclusion, and published a blog post on the subject on March 3. Quèze cited an email exchange with the GSoC office, in which he was evidently told that the decision to not include Mozilla this year was a difficult one, and that:

That assumption would, presumably, hold true for the LF, EFF, and Tor as well—all of those organizations are large and are on essentially stable footing. Furthermore, as Quèze pointed out, there are now multiple opportunities for students and other developers to contribute to open-source projects. Quèze pointed readers to a Mozilla-specific site designed to guide interested contributors toward any of several other coding opportunities connected to Mozilla projects.

A few other projects noted their lack of inclusion this year; it came up on the Tor mailing list, for example, but did not spark any discussion. Joomla was also left out, and reported in a blog post that the GSoC team said it had "decided to give newer organizations a little more visibility in the program this year."

But fixating on which organizations are not included in the 2015 mentoring group can distract from who is included. Most of the major open-source participants are returning, such as Apache, GNU, X.Org, GNOME, KDE, and a number of Linux distributions. Moreover, the list of new participants is interesting reading in its own right. It includes several well-known projects, such as CentOS, jQuery, and GNU Mailman. There are also newer projects that might not have had large enough communities in past years to run an organized GSoC mentoring program, such as the Tox encrypted instant-messaging program and the lowRISC project, which aims to produce open-hardware system-on-chip (SoC) designs.

Several university-sponsored projects are included for the first time, like MBDyn (a multi-body physics simulation engine from the Polytechnic University of Milan) and JdeRobot (a robotics development platform from Universidad Rey Juan Carlos in Madrid).

The largest group of new participants, however, seems to be from the biology and medical fields. That list includes:

• Bika, a laboratory-management system
• The Department of Biomedical Informatics at Stony Brook University
• The Encyclopedia of Life, a biological database
• The Global Alliance for Genomics & Health, a clinical data-sharing organization
• MEDES, a research group working on (among other things) space medicine
• OncoBlocks, a clinical "big data" warehouse project
• The Helikar Lab at the University of Nebraska, a computational biology project

As is often the case, the accepted mentoring organizations represent an intriguing slice of the wider open-source-software community. It can be quite fascinating to see just how many fields of study and areas of development now rely on open-source software.

As for the well-known organizations who were not included in this year's mentoring group, there is little cause for concern. Other projects have left and returned in years past, and although 137 organizations is certainly a hefty batch of participants, that number is still a reduction from last year's count: 190, the largest list in GSoC's ten-year history.

Plus, there are still numerous opportunities that an interested student could find to do work that benefits projects not on the list. Xiph.org, for example, is supported in its codec-development work by Mozilla, and there are multiple projects that offer kernel-related project ideas (BeagleBoard, QEMU, and the XIA networking project, to name just a few).

Now that the mentoring organizations have been announced, GSoC will soon enter the next phase: selecting the student participants from the pool of applicants. Students interested in participating can register starting on March 16. The deadline for applications is March 27, with the selections to follow a few weeks after that.

Unfortunately, there does not seem to be a robustly maintained "master list" of other open-source coding internship programs. The Opportunities page at the OpenHatch wiki comes the closest, but it relies on user contributions to keep its information and links up to date, with the usual caveats that accompany a wiki. There are a lot of coding opportunities out there in addition to GSoC, and GSoC's continued dominance on the topic in the online news space perhaps just serves to illustrate how much demand there is for programming internships such as these.

Comments (none posted)

Consumer ARM boards and the bleeding edge

There were several talks at SCALE 13x in Los Angeles that dealt with the practical side of embedded Linux development. One of the most well-received sessions was Stephen Arnold's look at the state of kernels and free-software graphics drivers on "open" consumer ARM devices. Arnold is a longtime Gentoo developer who also works with the Yocto project. He presented a rundown of the experiences other users can expect with ARM kernels and graphics on development boards and common single-board computers (SBCs), including the reliability of vendor-supplied drivers and the availability of Linux distributions.

Arnold's session was part of the "open hardware" track at SCALE, and one of the central questions addressed in his talk was how open many of the so-called "open hardware" ARM products really are. As most members of the community are aware, popular devices like the Raspberry Pi may be advertised as open, but the graphics chips on which they rely may have no free drivers at all. Those that do enjoy support from a free driver may have only partial functionality, or they may have an old, unmaintained driver that is difficult to port to a more recent kernel. If updating or rebuilding the vendor's kernel is impossible, after all, some of the key advantages of having an open device are all but lost.

ARM wrestling

Arnold led off with a quick tour through his personal embedded-hardware inventory, which he said he had amassed into "a collection of almost every ARM board ever." Hardware has changed so much over the years, he said, that today's boards exist in a blurry region between traditional embedded and desktop-class devices. Multi-core CPUs, more powerful GPUs, per-core floating-point units, and accelerated video processing are all becoming the norm.

Nevertheless, there are still significant differences to be found. Just within recent (ARMv7) generation products, he pointed out, boards like the Udoo and Cubox-i come with Vivante GPUs, Samsung Chromebooks and Sunxi "TV dongles" come with Mali GPUs, but the Acer Chromebook and Jetson TK1 board come with NVIDIA's Tegra K1—which is, essentially, a desktop-class graphics unit featuring up to 192 CUDA-capable cores.

Arnold also took a few minutes to explain some of the differences between the various instruction sets that users were likely to encounter on open-hardware ARM products and how that might affect writing software for them. The big divide, he said, is between devices that include the NEON instruction set and those that do not. Most ARMv7 boards include NEON, but a few (such as the Trim-Slice line of low-power boards) do not. As with ARMv6 devices (such as the original Raspberry Pi), the Trim-Slice boards use the older vector floating point (VFP) instruction set instead of NEON. The various video-driver projects often need to make use of NEON or VFP in order to provide responsive performance, so knowing which instruction set is supported is important.

The Five Blobs

Returning to the topic of graphics drivers themselves, Arnold looked at the five current GPU families for which ARM vendors are shipping binary-blob drivers: the iMX.6, the VideoCore IV, the Mali, the Tegra K1, and the OMAP SGX. There are projects working on open-source drivers for each GPU family, he said—in some cases, more than one. The Tegra family is the closest to having a complete solution, with both the OpenTegra/grate project and Nouveau support actively under development and producing working builds. Since the Tegra K1 is essentially a desktop GPU, this is perhaps not surprising.

The VideoCore IV (used in the Rapsberry Pi) has decent support with fbturbo, he said. That is what ships with the Raspbian distribution, but users may also want to look at the Weston for Raspberry Pi work that can take advantage of the GPU's hardware video scaler. The fbturbo driver also works with the OMAP, Vivante, and Mali GPUs, via the omapfb, lima, and etna_viv projects. Arnold noted that there is one other GPU family available, the Adreno, which has a free-software driver project, but he has not added the hardware to his collection yet, and thus has yet to test it himself.

If the user does stick with a vendor-supplied driver, though, there are still other factors to be considered—starting with what kernel the vendor ships. Typically, the kernels supplied by the vendor are a single release with a lot of patches applied—and the branch in question is likely to be old. Arnold said that the oldest kernel in his ARM collection is Linux 2.6.31.14, which shipped with a Fujitsu LifeBook and has never been updated. The newest kernel he has seen is 3.14. Usually, these kernels have had minimal back-porting done (if any), which means they may have security vulnerabilities, and attempting to forward-port the vendor's driver to a new kernel can take a long time.

Furthermore, the vendor kernels are often brittle: they may not use device trees, they may compile with lots of warnings (or even generate errors), or they may only compile with specific modules (or only when the kernel is built in monolithic form). Sometimes the kernel .config file is missing required configuration options, or the options included are clearly incorrect. A do-it-yourself kernel in most PC distributions is hard to mess up these days, he said, but if you change one thing in a vendor-supplied ARM kernel, you may find that USB stops working entirely, or that the firmware for the network or audio chips fails. Out of the current crop of ARM devices, he called the CuBox (with its iMX.6 GPU) and the Raspberry Pi the easiest to work with.

The other element to watch out for is the bootloader. Most vendors use their own fork of U-Boot; like the vendor-supplied kernel, it can be old and hard to work with. Some vendors that are "really on the ball" may have a newer U-Boot branch, since there has been considerable work upstream to consolidate the many U-Boot branches. If so, the user can take advantage of more recent bug fixes and optimizations.

Vendors tend to have only one supported bootloader configuration, he said, with the most common being booting from an SD card with two partitions (one for / and one for /boot). Manually changing this configuration is possible, but users who head down that path should be on the lookout for unusual boot options in UEnv.txt or boot.scr files. He also cautioned that he has seen different bootloader options even from the same vendor on two products that use the same system-on-chip (SoC), so an abundance of caution is warranted.

Status reports

Arnold then gave a rundown of the differences between the mainline kernel and the vendor kernels supplied for common ARM devices. Device trees are the main difference; whether or not a working device tree is available for a recent mainline kernel varies from device to device, and improving that situation is the subject of ongoing work by the community. He advised users looking to deploy a mainline kernel on their ARM device to check the Linux on ARM page at the EEwiki to see if there is a recent how-to guide. Robert Nelson from DigiKey maintains patches for a long list of vendor kernels and U-Boot forks at that wiki.

The Linux graphics stack itself is also in flux, Arnold said, with the migration from X to Wayland, the legacy driver model going away, and fluctuations in the OpenGL realm all complicating factors. He reported on his personal tests performed on his ARM hardware collection, calling out the Tegra 20/30 and Vivante hardware as working well with recent X.org releases. OMAP/PowerVR devices and Mali-based devices are also usable, although they depend on TI's open-source framebuffer code and fbturbo, respectively. He has recently given up entirely on the Efika MX product line, he added, which only works with "ancient kernels and closed-source drivers."

Even if the user decides to update from the vendor-supplied kernel and graphics drivers, it may be difficult to find a fully working Linux distribution for any particular ARM board. Arnold's final status report was a look at the distribution offerings for the various open-hardware ARM products in his collection. Typically, he said, the vendor will supply an Android OS option plus one other, traditional Linux distribution for each device. These distributions, of course, are typically limited to the vendor's legacy kernel and binary-blob drivers.

But some ARM products stand out from the crowd for their ability to support multiple distribution options. The Raspberry Pi offers the most, via its "New Out Of the Box Software" (NOOBS) system, which includes six distributions. The BeagleBone and BeagleBone Black officially support several flavors of Yocto and OpenEmbedded Linux, as well as Debian, Ubuntu, and Gentoo. Various Chromebooks support multiple mainstream distributions (Ubuntu, Debian, Gentoo, Arch, and Fedora being the most common). Udoo boards support an Ubuntu variant (udoobuntu), Android, OpenElec, and occasionally other distributions as well.

Pulling yourself up by your bootstraps

Ultimately, however, most users get interested in bootstrapping their own distribution at some point. For those people, Arnold said Yocto was the most reliable option. If there is a vendor board-support package (BSP), he said, Yocto can run on it. Yocto can also run on a modest desktop Linux machine, building a full image and handling build dependencies automatically. If there is not a BSP for your board, he said, you can probably create your own for Yocto.

Gentoo is the next option; Arnold maintains the Gentoo overlays for most of the ARM devices, and new stage3 builds are created every few weeks. For newcomers who may not be ready to migrate their desktop to Gentoo in order to do a native Gentoo build, there are other options, such as performing the build in a virtual machine. Last but not least, Debian and Ubuntu are always available: "if you can boot it, you can debootstrap it," he said.

In the brief question-and-answer period at the end of the session, Arnold agreed with one audience member that the pace of new ARM hardware devices was becoming problematic. There are new chipsets every year, but it typically takes the free-software community two or three years to fully implement software support. In the meantime, users are often left with just the vendor-supplied kernels, graphics drivers, and distributions to work with. So users can "run Linux" on their open-hardware ARM boards, but the situation is far from ideal.

Comments (9 posted)

A first look at the CANBus Triple

The CANBus Triple is a Kickstarter-funded open-hardware device intended to let car hackers read, write, and potentially modify the messages sent over their vehicle's Controller Area Network (CAN) bus. Although there are other hardware solutions for interacting with CAN traffic, the CANBus Triple offers some unique advantages. It is low-cost, the schematics and designs are all freely available, and it can serve in some roles that other, mass-produced devices cannot. That said, the software side of the project still has some catching up to do compared to the otherwise nice hardware.

The device itself is the work of Derek Kuschel, an independent hardware hacker from Detroit. In 2013, he built and sold a batch of prototype CAN bus devices after other readers on the MazdaSpeed discussion forum expressed interest in the personal CAN-hacking projects he had posted about. Based on the success of the prototype devices, he launched a Kickstarter campaign in mid-2014 intended to ramp up production. The fundraiser beat its target by a comfortable margin in September, and in February 2015, the first generation of production devices were mailed out to supporters—myself included.

The CANBus Triple takes its name from the fact that it provides three independent CAN bus channels (each including a controller chip and a transceiver). It also includes an Atmel ATMEGA 32u4 microcontroller, which allows it to be programmed with Arduino software tools for use in standalone mode, plus a USB serial port and a Bluetooth Low Energy module. The Bluetooth module does not offer sufficient bandwidth to log all of the CAN bus messages that a modern car might produce, but it would suffice for the Triple to be paired with a smartphone or tablet to, say, monitor specific message types or to send commands to the microcontroller's software. The USB port is capable of a high-speed connection to a laptop or other computer, and offers significantly more interaction possibilities.

The hardware included in the device is significantly more powerful than most of the other CAN bus peripherals that are available to consumers. For comparison's sake, USB peripherals can be difficult to find for less than 50 or 60 Euros, and that price range generally only provides a basic serial connection for a single CAN bus (see this vendor for one example). Modern cars often have at least two buses: one for high-priority components like engine sensors and braking modules, and one for monitoring events from low-priority components like the door locks and climate control system. At the other end of the spectrum, there are multiple Arduino shields that offer a CAN controller and transceiver (sometimes more than one), but those devices are difficult to make much use of in a non-Arduino software stack. At best, they can be adapted into logging tools, but few developers seem to succeed in doing much more.

To use the CANBus Triple, one needs the Arduino IDE and a copy of the project's Arduino code. This includes the configuration files necessary for the IDE to compile sketches and upload them to the device, plus a "basic" Arduino sketch (i.e., program) that lets the user connect to the device with a serial console and watch some CAN traffic. I had no trouble getting the Arduino sketch to compile and upload, and the device responded to the basic serial commands it is supposed to.

That said, there is more complexity to actually getting the device to do anything useful. The easiest way to connect to a car's CAN bus is through the OBD-II diagnostic port, which has a standard wiring configuration. Out of the box, the CANBus Triple comes with a custom cable so you can plug the device right into a OBD-II port. Plug in the Triple, start up the car, and one can see CAN bus messages over the serial connection.

For now, that is about the full extent of the device's functionality. Kuschel is working on a Cordova-based app that can be built for iOS, Android, and desktop systems (at least, any system for which Node.js is available). But the app is not yet in a workable state.

There is, however, a modest middleware layer in the Arduino code that lays the groundwork for more interesting development. It includes timers, hooks for catching and acting on specific CAN messages, and a channel-relay function to copy a CAN bus message heard on one of the CAN buses out onto one of the other CAN buses, among other things. The documentation here is currently quite sparse. Kuschel can hardly be blamed for that; as recently as two weeks ago he was still assembling, testing, and mailing out CANBus Triple units to Kickstarter supporters.

But there are a few developers on the discussion forum who have popped in already to announce that they are working on some carmaker-specific software using the CANBus Triple. Alternatively, Kuschel has released a more feature-rich Arduino sketch that is tailored to Mazda cars. Both of these developments highlight one of the challenges of car hacking: almost every manufacturer may use CAN bus, but there is such a wide array of diverse messaging formats that most software can quickly become vehicle-specific.

Nevertheless, I remain optimistic that CANBus Triple has a bright future. The vast majority of CAN-related hacking projects are simple data-logging or monitoring tools that use an Arduino; there has always been a large price-and-functionality gulf between those projects and the expensive USB CAN adapters that a full-fledged Linux box can, theoretically, do more with. The CANBus Triple basically sits right in between: it has an on-board Arduino-style microcontroller, but it has a USB serial port, too.

Furthermore, the Triple is the only device I am aware of that has the hardware configuration required to intercept, modify, and pass on CAN traffic. That functionality is the key to doing innovative things in the automotive environment—especially in the aftermarket arena. Without it, a car computer can eavesdrop on other CAN-connected components' messages or generate its own, but it cannot really override the car's existing modules in their factory configurations.

The possibilities for modifying CAN traffic are literally endless. From simple adaptations like having the audio unit raise the volume level when the car is traveling at a higher speed to more ambitious functions like having an electric car intelligently power-down non-essential components when the battery is running low, modifying CAN messages is a powerful tool.

The device has a few quirks. For example, the included cable (with a standard diagnostic-port connector) is only wired up on two pins (6 and 14, which are standard CAN pin locations), which means that only one of the three CAN buses is reachable. The other pins can be soldered on, although in my case, whenever I popped open the connector's casing, it seemed like one of the other wires had come loose from its pin. Such are the ins and outs of small-production hardware, though.

On the whole, the CANBus Triple is impressive because it fits right into a gap that no other product is addressing. The fact that it is open hardware and it is built on entirely open-source software makes it all the more likely that the car hackers in the community will pounce on it to do something interesting. And it's hard to beat the snappy orange color scheme, either.

Comments (25 posted)

The FREAK crypto downgrade attack

For the most part, cryptography is not what is letting us down security-wise. Most of the algorithms are solid and, if used correctly, essentially uncrackable in any sensible time frame. There are exceptions, of course, but by and large cryptography has served us well. Unfortunately, we commonly have programs that provide ways to downgrade the cryptographic algorithms or hamper them in other ways (e.g. shorter keys) so that they no longer serve to protect our data as they should.

When an attacker can cause systems to use a weaker encryption algorithm, perhaps one that has serious flaws, it is called a "downgrade attack". The latest entrant into the pantheon of downgrade attacks is one that has been dubbed FREAK (which stands for "Factoring attack on RSA-EXPORT keys"). It uses the ability of a man-in-the-middle attacker to downgrade the encryption used by many web sites when accessed by users with the Safari and Android browsers. As the name indicates, attackers could cause the affected systems to use the purposely weakened export version of the RSA cipher that limited key length to 512 bits.

In a delicious irony, the web site of the US National Security Administration (NSA)—which mandated the use of export-grade RSA keys during the crypto wars of the 1990s—was affected by FREAK. Many other sites were too, of course. But those sites are only vulnerable when they are accessed from a vulnerable client.

Even though the export version of the RSA suite is considered to be a "zombie" by many, it is evidently still present in multiple web servers. As Matthew Green noted in his analysis, it shouldn't matter if the web servers still have the export version of RSA available if the clients do not request the 512-bit export-grade keys. Also, clients need to reject those keys if they haven't requested them. It is the latter piece that fails here.

It turns out that a man in the middle can alter the client's initial message to the server (which is sent in the clear) to change its request for the standard RSA cipher to a request for the export version. If the server supports export RSA, it will happily send back a weak RSA key that is vouched for by the server's certificate. Apparently, roughly 35% of the internet, much of that being systems run by content delivery networks (CDNS), does or did support export-grade RSA. The client, though, knows (or should know) that it requested the standard RSA cipher so it should reject the weakened export key. But the vulnerable clients do not do that.

Even after the twin assumptions that few servers still supported export RSA and that few clients would request and accept those keys were invalidated, there was still one more line of defense: factoring 512-bit RSA keys is not trivial, so traffic could not be intercepted in realtime. It could only be read later, after the key was broken, which takes some seven hours and costs $104 on Amazon's cloud servers.

But, in reality, even that "barrier" fell. It turns out that, by default, Apache's mod_ssl will only generate a single export RSA key every time it starts up. From then on, it uses the same key over and over again. So, breaking that key once allows realtime decryption of all of the traffic from then on—until the web server gets restarted.

While man-in-the-middle attacks seemed somewhat difficult for attackers to engineer at one point in time, that is really no longer operative (if it ever really was). It is clear that governments, at least, are fully willing to perform such attacks. Any internet service provider (ISP) is perfectly positioned to attack its customers that way, as are the operators of the wireless networks in millions of locations worldwide. But, of course, encryption is meant to thwart man-in-the-middle attacks.

Part of the problem here is that TLS (or, really, its predecessor, SSL) was "designed to be broken", Green said. Because of US government mandates about exporting cryptography, SSL needed to handle both strong and weak encryption. That led to the cipher suite negotiation that would (hopefully) allow clients and servers to arrive at the strongest encryption that both could support. But it also led to attackers exploiting the ability of clients and servers to back down to simpler ciphers and protocols.

FREAK is not the first downgrade vulnerability that we have seen and it certainly won't be the last. POODLE is a famous one that exploited systems that would fall back to SSL 3, rather than require TLS 1.1 or higher, and there have been others. Even ISPs that remove STARTTLS from SMTP server responses are performing a kind of downgrade attack. When protocols are designed with the possibility to downgrade, attackers are going to try to find a way to exploit that. It is something for protocol designers to keep in mind down the road.

Comments (6 posted)

Security quotes of the week

Comments (16 posted)

apache2: denial of service

Comments (none posted)

arc: directory traversal

Comments (none posted)

cabextract: privilege escalation

Comments (none posted)

firefox: multiple vulnerabilities

Comments (none posted)

foreman-proxy: restriction bypass

Comments (none posted)

httpd: denial of service

Comments (none posted)

kfreebsd-9: denial of service

Comments (none posted)

librsvg2: multiple unspecified vulnerabilities

Comments (none posted)

libuv: privilege escalation

Comments (none posted)

mozilla: code execution

Comments (none posted)

python-rope: code execution

Comments (none posted)

request-tracker: multiple vulnerabilities

Comments (none posted)

qt: denial of service

Comments (none posted)

unace: code execution

Comments (none posted)

vorbis-tools: denial of service

Comments (none posted)

Kernel release status

Stable updates: 3.18.8, 3.14.34, and 3.10.70 were released on February 26. The 3.19.1, 3.18.9, 3.14.35, and 3.10.71 updates are in the review process as of this writing; they can be expected on or after March 6.

Comments (none posted)

Quotes of the week

Comments (none posted)

Memory management when failure is not an option

Some introductory concepts

The kernel's memory-management subsystem is charged with ensuring that memory is available when it is needed by either the kernel or a user-space process. That job is easy when a lot of memory is free, but it gets harder once memory fills up — as it inevitably does. When memory gets tight and somebody is requesting more, the kernel has a couple of options: (1) free some memory currently in use elsewhere, or (2) deny (fail) the allocation request.

The process of freeing (or "reclaiming") memory may involve writing the current contents of that memory to persistent storage. That, in turn, involves calling into the filesystem or block I/O code. But if any of those subsystems are, in fact, the source of the allocation request, calling back into them can lead to deadlocks and other unfortunate situations. For that reason (among others), allocation requests carry a set of flags describing the actions that can be performed in the handling of the request. The two flags of interest in this article are GFP_NOFS (calls back into filesystems are not allowed), and GFP_NOIO (no type of I/O can be started). The former inhibits attempts to write dirty pages back to files on disk; the latter can block activity like writing pages to swap.

Obviously, the more constrained the memory-management subsystem is, the higher the chances of it being unable to satisfy an allocation request at all. Kernel developers have long been told that (almost) any allocation request can fail; as a result, the kernel is full of error-handling paths meant to deal with that eventuality. But it became clear recently that the memory-management code does not actually allow smaller requests to fail; it will, instead, loop indefinitely trying to free some memory. That behavior has been seen to lead occasionally to locked-up systems, despite the fact that the code involved is prepared to deal with allocation failures. The "too small to fail" behavior is controversial, but would prove hard to change at this point.

There are, however, places in the kernel that are simply unprepared to deal with allocation failures, usually because the allocation happens deep within a complex series of operations that would be difficult to unwind. The __GFP_NOFAIL flag exists to explicitly state that failure is not an option for a given request, though its use is heavily discouraged.

The following discussion, in the end, focuses on two related questions: (1) should the kernel really be treating small allocations as if they all had __GFP_NOFAIL set, and (2) should failure-proof allocations be supported at all, and, if so, how can that support be made more robust?

No longer too small to fail

The discussion (re)started when Tetsuo Handa noted that memory allocation behavior had changed in the 3.19 kernel; in particular, small allocations with the GFP_NOIO or GFP_NOFS flags would fail under severe memory pressure. In previous kernels, such allocations would loop indefinitely if no memory was available. Among other things, this change can cause filesystem operations to fail on memory-stressed systems where they would have (eventually) succeeded before.

The behavior change is the result of this patch from Johannes Weiner which was aimed at avoiding the memory-allocation deadlocks that started the December discussion. The intent was to avoid looping forever in an allocation attempt if it appeared that no progress was being made toward freeing some memory for that allocation, but, by accident, it also prevented looping entirely in the GFP_NOIO and GFP_NOFS cases. So those allocations can now fail; that is a significant change from how previous kernels worked.

Johannes initially wanted to keep the new behavior, saying that it "makes more sense". But the filesystem developers disagreed strongly. It seems that there are numerous places in the filesystem code that depend on allocations succeeding reliably, and that many of them are not marked with __GFP_NOFAIL. Ted Ts'o threatened to add a lot of __GFP_NOFAIL flags to allocation calls in the ext4 filesystem if the change were not reverted. The memory-management developers were thus faced with the need to pick the option they disliked least.

In the end, the filesystem developers won out on this one; Johannes merged a change into 4.0-rc2 restoring the looping behavior for those allocation types. This change is likely to end up in the 3.19 stable series as well. The original patch is a good argument for the approach of refusing "cleanup" patches late in the development cycle. It was merged for the 3.19-rc7 prepatch, meaning that there was almost no time for problems to be noticed before the final 3.19 release came out.

The discussion was not limited to the unexpected effects of one late-arriving memory-management patch, though. The bigger problems of how to avoid deadlocks in low-memory situations and how to ensure that important tasks can proceed in those situations remain unsolved.

The OOM killer

The out-of-memory (OOM) killer is implicated in a number of stall scenarios. In the original problem reported last December, the OOM killer would choose a victim that was blocked on a lock, but that lock was held by the process waiting (forever) for a memory allocation to proceed. As a result, the victim could not exit and, thus, could not free its memory. Since the OOM killer only goes after a single process at a time, everything would stop at that point.

Johannes suggested a change to how the OOM killer works: if a targeted process failed to exit after five seconds, the OOM killer would give up and move on to another victim. The idea was not hugely popular, though. David Rientjes pointed out that there was no guarantee that the next victim would be any more appropriate than the one that came before. Dave Chinner claimed more broadly that efforts to tweak the OOM killer are misdirected:

The end result is that OOM-killer timeouts will probably not find their way into the memory-management subsystem anytime soon.

__GFP_NOFAIL and looping

From the point of view of the memory-management developers, many things would get easier if any allocation request could fail when the necessary resources are not available. That would mean getting rid of the implicit "small allocations never fail" rule, but, beyond that, it would also require getting rid of the explicit __GFP_NOFAIL call sites. Michal Hocko was perhaps the most outspoken in this regard, saying that __GFP_NOFAIL "is deprecated and shouldn't be used". He also suggested that existing __GFP_NOFAIL call sites should be reimplemented in a way that allows them to recover from allocation failures.

Dave took issue with that idea, saying that failure-proof allocations are a hard requirement for the XFS filesystem. To rework XFS to be able to roll back dirty transactions in the face of an allocation failure would increase its complexity significantly, he said; the project would take a couple of years to reach a point where it could be put into production use. He summarized by saying "I'm not about to spend a couple of years rewriting XFS just so the VM can get rid of a GFP_NOFAIL user". Strangely enough, there were no other developers volunteering to take on that job either.

Contemporary filesystems are complex beasts that have to meet a wide variety of demands. They incorporate complex transaction mechanisms that help them to maintain filesystem integrity in every situation possible. Implementing such a mechanism in a way that allows it to recover from a memory-allocation failure in the middle of a transaction, after resources have been committed, locks taken, etc., is not a simple task. Filesystem developers on Linux have not taken on that task because, in the end, there has not been a need to. Allocations that cannot be allowed to fail have proved sufficient in almost all situations.

Once one accepts that some sort of failure-proof allocation mechanism is needed, though, the next question is: how should it be done? The __GFP_NOFAIL flag is one solution, but it turns out that quite a bit of code in the kernel does not make use of it. Instead, there are a number of places in the kernel that implement their own retry loops on top of a kmalloc() call without __GFP_NOFAIL. That is something that the memory-management developers don't like; those developers would rather not see __GFP_NOFAIL used at all, but they still prefer its use to retry loops implemented outside of the memory-management subsystem. Consider, for example, this message from Johannes saying that the XFS developers should replace a retry loop with a single __GFP_NOFAIL call.

There are couple of reasons why such loops exist. One of those is that __GFP_NOFAIL was explicitly deprecated in 2009; the patch (from Andrew Morton) said:

After this change went in, it became harder to get code containing __GFP_NOFAIL past reviewers. Whether it is done lamely or not, a hand-coded infinite retry loop is easier to sneak into the kernel than an easily greppable __GFP_NOFAIL use. So that is what developers did.

The memory-management developers dislike must-succeed allocations because they complicate the code and, as has occasionally been seen, create the possibility of deadlocks. If such allocations must be made, they would rather see the looping done in the memory-management code, where behavior can be tweaked and appropriate action taken (starting the OOM killer, for example) if it becomes clear that no progress is being made. In the real world, though, according to both Ted and Dave, looping actually works pretty well. The XFS code has a "canary" that puts out a warning when the looping goes on for too long, but, Dave said:

One might take that as a statement that the XFS developers are currently uninterested in replacing their own loops with __GFP_NOFAIL invocations. But they actually have another reason to maintain a loop outside of the memory-management code: they want to retain control over how the filesystem should respond to low-memory conditions. It is, in their mind, a policy decision that the memory-management code lacks the information to handle. There are currently plans afoot to expose some of that policy to user space, allowing administrators to configure what the filesystem's low-memory response should be.

Reservations

Still, there is no real disagreement over this idea: looping over a failing memory allocation is undesirable and best avoided whenever possible. Thus it may well be that the most useful part of the discussion came when the developers got around to the topic of avoiding allocation failures altogether. There are a few ways of working toward that goal.

One of those is preallocation — allocating all of the needed memory resources before the code gets to a point where it can't back out of a transaction. Preallocation is used in many contexts in the kernel and works well, so it was natural for the memory-management developers to ask whether it can be used in this context. Dave shot that idea down fairly quickly:

Mempools were also raised as a possibility. They are a form of preallocation that might avoid some of the overhead described above. But they are, according to Dave, poorly suited to the problem at hand. Mempools deal with a single size of object, while a filesystem transaction needs a wide variety of objects; that implies that several mempools would be needed at various levels in the stack. There is also a mismatch between object lifetimes that make mempools difficult to use across multiple transactions. So mempools do not appear to be an option either.

Dave's suggestion, instead, is to add the concept of "reservations" to the memory-management subsystem. Prior to entering a transaction, the filesystem code would inform the memory-management code that it will need guaranteed access to a certain amount of memory; calculating an approximate memory requirement is, apparently, not that hard. The memory-management code would then ensure that the requisite amount of memory would be available; subsequent allocation requests would dip into the reserve if need be. As long as the estimate for the size of the reserve is sufficient, there should be no problem with failing allocations during the transaction.

Reservations may look a lot like preallocation, but there is a crucial difference. The memory-management code already maintains a "watermark," a level of free memory below which it is unwilling to go unless absolutely necessary. A reservation would simply raise that watermark, making a bit less memory available to the system as a whole. If a reservation would raise the watermark above the amount of memory that is currently free, the request would block until more memory could be reclaimed. In the simplest case, a reservation would be represented as an increased value in a single integer variable.

There seems to be some general support for the addition of a reservation mechanism, but things get less clear once one looks at the details. Andrew Morton suggested a scheme where a process making a reservation would get a number of "tokens"; subsequent allocations done by that process would come from the reserve first. Dave does not like that idea, saying that it fails to account for the fact that many objects allocated during a transaction will be freed (perhaps by others) shortly and, thus, should not come from the reservation. His view of the reservation, instead, is a range of memory that is not touched at all unless there is no alternative; even then, only allocations using the GFP_RESERVE flag would be able to get at that memory. The reservation, in his view, comes into play when the kernel would have otherwise put the OOM killer into action.

Johannes, instead, says that this approach will not work. The problem is that "we simply don't KNOW the exact point when we ran out of reclaimable memory", so the memory-management subsystem cannot easily guarantee the sort of loose reservation that Dave has described. Dave disagreed with that assessment, it almost goes without saying. And that is about where the conversation wound down.

Reservations are a promising idea for a solution to some of the kernel's memory-allocation challenges. But, at this point, it is just an idea; it has neither code nor a design consensus behind it. The discussion has slowed for the moment, but that is almost certain to be a temporary state of affairs. The annual Linux Storage, Filesystem, and Memory Management Summit is less than one week away as of this writing. This subject is on the agenda, and LWN will be there to report on the discussion.

Comments (16 posted)

Python bindings added for libseccomp 2.2.0

The secure computing (seccomp) facility was added to Linux some ten years ago as a way to restrict programs so that they can only make a small subset of system calls. It is a way to sandbox processes but, over the years, it was found to be too restrictive. Thus, after a few false starts, a new seccomp mode that used the kernel's Berkeley Packet Filter (BPF) implementation to provide a way to more flexibly sandbox processes came about in 2012. To help application writers use the facility, Paul Moore created libseccomp, and he has just released version 2.2.0 of the library.

This is the first release of libseccomp for more than a year, with a 2.1.1 minor release in October 2013 (after 2.1.0 in June 2013). One of the headline features for 2.2.0 is the addition of Python bindings, which have been around for a while but have not been part of a release until now. Other big changes for this release include a switch to Autotools, support for the ARM 64-bit architecture, as well as support for several flavors of the MIPS architecture (mips, mips64, and mips64n32).

The newer seccomp mode (known as seccomp filters or seccomp mode 2) allows developers to specify which system calls are allowed to be made and to restrict the arguments that can be passed to those system calls. In order to do that, the kernel requires a program written using the BPF language, which was originally targeted at network filtering, though it has grown well beyond that. Libseccomp is meant to provide a higher-level interface to that functionality—from C and, now, Python.

If you have a program that is handling untrusted user input—HTTP traffic or image formats, say—you might want to restrict the kinds of operations that program can perform. For example, there might just be a handful of operations that should be allowed to the program, so that if it were compromised by unexpected input, there is little an attacker can actually do. On the other hand, though, the program might require a bit more than the four system calls (read(), write(), exit(), and sigreturn()) allowed by the original seccomp mode.

For instance, open() might be allowed, but only to open files for reading. Or, the write() system call might be restricted to certain file descriptors (say, 1 and 2 for stdout and stderr). Meanwhile, all other system calls, including powerful calls that attackers might want to use, such as execve() or socket(), would be disabled. As with the C interface, the actions taken when a disallowed system call is made depend on how the library is initialized. Those calls could cause the program to be killed, to receive a signal, to generate a ptrace() event, or for the call to fail with a particular errno value.

Using the Python bindings is similar in many ways to calling the library directly from C:

    import sys, os
    from seccomp import *

    f = SyscallFilter(defaction=KILL)

    f.add_rule(ALLOW, "open")
    f.add_rule(ALLOW, "write", Arg(0, EQ, sys.stdout.fileno()))

    f.load()

    x = os.open('/tmp/x', os.O_WRONLY)
    os.write(x, 'Hello, world\n')

However there is a bit more to it than that. When testing the non-error path by commenting out the os.write(), Python requires brk(), rt_sigaction(), and exit_group() to exit gracefully. So the following would need to be added to the list of rules:

    f.add_rule(ALLOW, "exit_group")
    f.add_rule(ALLOW, "rt_sigaction")
    f.add_rule(ALLOW, "brk")

While that does add to the list of allowed system calls, it doesn't really enlarge what an attacker could do when subverting this (extremely simple) program. Using the Python version of open() and write(), instead of those from the os module, requires opening up several more system calls (mmap(), read(), close(), and fstat()), which could be a bigger problem. Having both open() and read() available might allow an attacker to access files, but the contents can only be written to stdout, which may well be an impediment. Further refinement of the rules could limit an attacker even further.

When debugging seccomp filters, there is often a need to track down which system call caused a failure. That can be done a number of different ways. When the KILL action is used, as above, the process is forced to exit (with SIGSYS as its status), so the shell simply prints "Bad system call". But it also leaves an audit trail that records the system call number that failed:

    type=SECCOMP msg=audit(1425421709.486:42015): auid=1000 uid=1000
      gid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
      pid=25113 comm="python" exe="/usr/bin/python2.7" sig=31 arch=c000003e
      syscall=5 compat=0 ip=0x7faf36e38824 code=0x0

Rules can be built using the Arg() function to specify the allowable values for up to six arguments. Those values can be tested using the usual comparison operators, but by name (e.g. EQ for =, GE for >=, LT for <, and so on). There is also a MASKED_EQ operator that can be used for flag values:

    f.add_rule(ALLOW, "open",
               Arg(1, MASKED_EQ, os.O_RDONLY,
                   os.O_RDONLY | os.O_RDWR | os.O_WRONLY))

The 2.2.0 release comes with a test suite that includes both C and Python versions of each test. It also has man pages for each of the C language seccomp_* calls. The only Python language documentation appears to be in the src/python/seccomp.pyx file, but perhaps that will change before long. Anyone looking to sandbox their programs should definitely take a peek at libseccomp.

Comments (3 posted)

Ftrace and histograms: a fork in the road

The idea behind hist triggers is simple enough: a user interested in a histogram of tracing data would write a command string to the appropriate ftrace control file specifying the parameters of the histogram. For example, one could look at kmalloc() calls with a command like:

    # echo 'hist:key=call_site:val=bytes_req' > \
           /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger

Here, the hist: prefix indicates that histogram output is desired. The key= and val= parameters describe the axes of the histogram; in this case, the user will get the total amount of memory requested from each location where kmalloc() is called. One obtains the results by reading the hist file that magically pops up in the tracing control directory:

    # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist
    trigger info: hist:keys=call_site:vals=bytes_req:sort=hitcount:size=2048 [active]

    call_site: 18446744071581750326 hitcount:          1  bytes_req:         24
    call_site: 18446744071583151255 hitcount:          1  bytes_req:         32
    call_site: 18446744071582443167 hitcount:          1  bytes_req:        264
    call_site: 18446744072104099935 hitcount:          2  bytes_req:        464
    call_site: 18446744071579323550 hitcount:          3  bytes_req:        168
    [...]

There are additional options that can, for example, turn the call-site address into a symbolic location. This example was taken from the documentation posted with Tom's patch set; many more examples and details can be found there.

Tom thinks that this sort of functionality will be useful to a wide variety of tracing users. Indeed, it may reduce the need for more sophisticated tools:

Nobody seems to disagree that this would be a nice feature to have, but, still, criticism of the patch set came from two directions. Ftrace maintainer Steve Rostedt complained that the tracepoint code generating the histograms performs memory allocations; those allocations are necessary to maintain the hash table used to hold the histogram data. Tracepoint callbacks can be called with all sorts of locks held; allocating memory in such a situation is not a safe thing to do. So, Steve said, that aspect of the patch set is a "showstopper." Future versions of the patch set will, thus, have to accumulate this data without allocating memory in the tracepoint callbacks.

A different type of criticism came from Alexei Starovoitov, the developer behind the eBPF work that has gone into the kernel over the last year. One of the use cases for a tool like eBPF is to allow users to gather data in kernel space and generate output in forms like histograms. Alexei thus duly suggested that eBPF should be used to implement Tom's histogram functionality. Rather than parse the commands in the kernel, though, Alexei would like to see the development of a tool that would parse the same commands in user space and load an eBPF program to do the actual work.

To Tom, the idea seemed "silly"; a lot of work would be required to implement the functionality that already exists. He saw the request as an attempt to force the use of eBPF on users who may not want to deal with it. Alexei responded by saying:

Tom remained unimpressed:

In the end, there is an important question to answer here. The eBPF subsystem provides a mechanism by which a great deal of interesting tracing functionality could be implemented without having to hardwire the logic in the kernel. Now that eBPF is here, adding new tracing modes as more C code in the kernel could lead to duplicated functionality that needs to be supported indefinitely, even if, someday, an alternative implemented in eBPF draws most of the users.

On the other hand, the current interface to ftrace, wherein users write simple control strings to a set of virtual files, appeals to a lot of users. It is relatively easy to work with, does not require any additional tools to use, and is straightforward to script. Some of those users would not be pleased if they felt pushed to move over to an interface requiring the compilation and loading of eBPF programs to get their work done.

This has the look of a debate that could go on for some time. In the absence of a decision by decree from a suitably placed subsystem maintainer, it seems unlikely that the developers involved will settle on a single approach to the problem of how to add new tracing features. The kernel's tracing subsystem is arguably at a fork in the road, but we may not know which branch will be taken for a while yet.

Comments (2 posted)

A look at EasyNAS

Since EasyNAS and Rockstor are both working toward the same broad goal, comparisons between the two come naturally. For a start: while Rockstor is based on CentOS, EasyNAS uses openSUSE as its base. That choice makes a certain amount of sense; openSUSE is putting a lot of effort into making Btrfs a viable option for production use. Neither EasyNAS nor Rockstor appear to have any developers actively working on Btrfs, so the only hope for getting problems fixed is with the upstream distribution. That would suggest that Btrfs-related problems that turn up in EasyNAS are more likely to receive developer attention.

Then there is the issue of development communities. Rockstor is developed by a small group trying to make a business out of it, with a tiny amount of community support. EasyNAS, instead, appears to be the work of a lone developer who has expressed no business ambitions beyond putting up a donation link. The end result is that EasyNAS lags behind Rockstor in a number of important ways.

There are two installation images available for EasyNAS. There is a standard ISO image that can be used to install from a USB drive or an optical disk. On top of that, the project offers a "raw" image that enables the system to run off a USB drive with no separate installation step at all.

As is the case with all of NAS distributions reviewed, EasyNAS runs a web application as the primary administrative interface. It is possible to use TLS once one gets past the inevitable self-signed certificate warning. This interface has its ups and downs, but mostly works as one would expect. It must be said, though, that the built-in "help" functionality is useless; it puts up a window with a terse sentence or two of text — often the same text that already appears in the screen of interest.

Storage configuration follows the usual patterns. One starts by creating filesystems from one or more disks, then creating "volumes" from the filesystems. The volumes (implemented as Btrfs subvolumes) are the actual filesystems seen by users, of course. The volume administrative screen is, among other things, the place where one controls whether each volume is exported via NFS, SMB, AFP, or TFTP.

The disk management screen (seen to the right) has some quirks. The pie chart is thoroughly useless, showing simply how many drives have been put to work. The magnifying-glass icon on each disk device will, if clicked, start an I/O performance test. The identical magnifying glass that appears in a small search bar on every screen does nothing useful at all, as far as your editor can tell. Nowhere in the web-based system is it possible to find out how much space is free on a given drive or filesystem (such information can be had via the command line, of course). It is also not possible to put size limits on an given volume.

There is a rudimentary snapshot facility built into the system. A separate screen allows snapshots to be scheduled at regular times. Your editor initially thought that there was no way to request a snapshot manually — until he clicked on the magnifying-glass icon found on the volume administration screen. Once created, snapshots just look like new volumes on that screen as well. It is true that, underneath it all, snapshots and volumes are essentially the same thing to Btrfs, but users are likely to see them as being different and might like to manage them separately. The names given to snapshots (e.g. v_1502240010 for a snapshot of volume v) encode the date the snapshot was taken (February 24, 2015 in this case), though the format could be a bit more readable.

There is a "power management" screen, but all it can do is to shut down or reboot the system; there is no way to control power-management features like drive spindown times. User accounts can only be administered locally; there is no provision for working with an LDAP or NIS server on the net. EasyNAS also lacks the sort of plugin facility that other NAS-oriented distributions provide. The "resource monitor" screen gives plots of a few system parameters, but lacks useful ones like disk or network I/O bandwidth. On the other hand, there is a basic firewall mechanism whereby one can configure which other systems can access the EasyNAS box.

Those searching for the EasyNAS source on its web site have a long and frustrating task ahead of them. It would appear that there is no publicly available source repository for EasyNAS. The source can be found, though, under /easynas on an installed system. It consists mainly of about 5,700 lines of Perl CGI code licensed under GPLv3+. That is enough for a would-be contributor to work with, but the project would be much more likely to get contributions if it put a bit more effort into making it easy to work on the code.

To summarize, your editor would be reluctant to base a production NAS box on EasyNAS. The fundamentals are there, but there are important gaps in functionality and the project does not appear to have the resources to close those gaps. It is not clear what help is available if things go wrong; the project's forums are a lonely and desolate place. Things can always change, but, as they stand now, it is hard not to conclude that there are better options available.

Comments (5 posted)

Distribution quote of the week

Comments (none posted)

Vivid Vervet Beta 1 Released

Full Story (comments: none)

Debian Project Leader Elections 2015: Call for nominations

Full Story (comments: none)

Red Hat Enterprise Linux 6.4 Extended Update Support Retirement Notice

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly, Issue 599 (March 2)
• Tails report (August to December 2014)
• Ubuntu Weekly Newsletter, Issue 406 (March 1)

Comments (none posted)

New operators for Python dicts?

The Python dictionary is a commonly used data structure that supports a rich set of operations. But there are some operations that it lacks—two operators in particular: "+" and "+=". That lack is the subject of a recent discussion on the python-ideas mailing list. There are questions about the precise semantics of the operators, but there is also something of an existential question about the need for operators whose semantics can already be handled using existing operations.

Some background

Dictionaries (or dicts) are also known as associative arrays or hashes in other languages. In essence, they map some key, which is usually—but not always—a string, to some other value. A simple example:

    >>> a_dict = { 'a' : 3, 9 : 7, 'foo' : 'bar' }
    >>> a_dict['a']
    3
    >>> a_dict[9]
    7
    >>> a_dict['bar']
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    KeyError: 'bar'

One of the other fundamental Python types is the list, which provides an ordered sequence of objects, in many ways like arrays in other languages.

    >>> b_list = [ 1, 2, 3 ]
    >>> b_list[2]
    3

    >>> b_list + b_list
    [1, 2, 3, 1, 2, 3]
    >>> b_list += [ 4, 5, 6 ]
    >>> b_list
    [1, 2, 3, 4, 5, 6]

    >>> a_dict + a_dict
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

Adding + and +=

But Ian Lee would like to see that change. He first raised the issue in a brief subthread on the python-dev mailing list in a discussion about code review for PEP 448. Lee subsequently moved the topic to python-ideas, where he suggested adding both the + and += operators for dicts to put them on an equal footing with lists.

The semantics of adding two dicts that have no keys in common seems clear: the result is a dict with all of the key/value pairs from both operands. For +, the result is a new dict, while += modifies the dict on the left. The only real question is what to do when there are duplicate keys. Dicts already have an update() method that takes the value for a duplicate key from the argument dict:

    >>> a_dict = { 'a' : 1, 'b' : 2 }
    >>> a_dict.update( { 'b' : 9 } )
    >>> a_dict
    {'a': 1, 'b': 9}

    >>> a_dict = { 'a' : 1, 'b' : 2 }
    >>> b_dict = { 'b' : 'bar', 'c' : 'baz' }
    >>> a_dict + b_dict
    { 'a' : 1, 'b' : 'bar', 'c' : 'baz' }
    >>> b_dict += a_dict
    >>> b_dict
    { 'a' : 1, 'b' : 2, 'c' : 'baz' }

Donald Stufft liked the idea behind the change, but didn't like using +. He would rather use "|" to try to make it clearer that it is really more of a set union operation, rather than a concatenation or addition. Ethan Furman, though, sees + as a generic operator for combining things. On the other hand: "I suppose I could come around to '|', though -- it does ease the tension around the behavior of duplicate keys", he said.

After a bit of a digression through a question of commutativity (which is not preserved by the operators, but that is hardly unique—string concatenation doesn't either, for example), Marc-André Lemburg explained that he didn't see the need for +, though += could be useful:

Having one of those operators without the other seems a bit strange to some, though. Operators in Python are implemented as special methods on objects, so a + b becomes a.__add__(b) (similarly, += uses the __iadd__() special method). Dicts could pick up an __iadd__() method (or the | equivalent: __ior__()), but most developers, especially those new to the language, would probably expect + to work if += did.

Other options

In the case of duplicated keys, there are (at least) two other options. An exception could be raised when combining two dicts that have keys in common, as Greg Ewing suggested, though that might be surprising. Another option would be to apply the addition operator to the two values, but that might cause its own set of surprises:

    >>> a_dict = { 'a' : 2, 'b' : 'foo' }
    >>> b_dict = { 'a' : 4, 'b' : 'foo' }
    >>> c_dict = { 'b' : 3 }
    >>> a_dict + b_dict
    { 'a' : 6, 'b' : 'foofoo' }
    >>> b_dict + c_dict
    ...
    TypeError: cannot concatenate 'str' and 'int' objects

Lee summarized the ideas and approaches from early on in the thread in a kind of a pre-PEP document.

Even though there is a lot of precedent for operators like + and +=, Steven D'Aprano argued that they are actually flawed ideas that should not be further propagated. The fact that lists have those operators is not for the better:

D'Aprano described one of those corner cases (which also appears in the Python FAQ) for the tuple immutable sequence type:

    >>> t = ([], None)
    >>> t[0] += [1]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'tuple' object does not support item assignment
    >>> t
    ([1], None)

    setitem(t, 0, getitem(t, 0).__iadd__([1]))

    >>> l = t[0]
    >>> l += [1]
    >>> t[0] = l

The subject of dict.__add__() comes up on python-ideas with some frequency, and it is clear there are strong feelings on all of the different sides. Stufft thinks it would make a nice "mini-addition" to the language that might make newer versions a little more attractive:

    new_dict = dict1.copy()
    new_dict.update(dict2)

But Stephen J. Turnbull is not convinced that the semantics are so clear that what the operators do would be obvious to most. He noted that four different ways to handle the duplicate-key problem had been proposed and added two more, possibly with tongue in cheek. In addition, since there are existing ways to perform those operations, adding another violates the Python "there's only one way to do it" (TOOWTDI) guideline.

Early on, Lee indicated that he would try to shepherd a PEP through the process to see if the operators could be added to dicts. Brett Cannon agreed with that idea:

That's where things stand now. No PEP has yet appeared, though it seems likely that one will. It is an interesting question in that both sides seem to see their choice as the "obvious" one. There is precedent in that lists have the two operators, but that precedent does lead to some corner cases and warts. Even if the PEP were to be accepted, it would only be a feature for some upcoming version of Python 3—features are no longer being added to Python 2. One suspects that in the end it will come down to what benevolent dictator for life (BDFL) Guido van Rossum thinks—so far he has been silent in the thread.

Comments (4 posted)

Quotes of the week

Comments (7 posted)

LLVM 3.6 Released

Full Story (comments: none)

VLC 2.2.0 released

Version 2.2.0 of the VLC media player has been released. According to the announcement, highlights in the new version include automatic, hardware-accelerated rotation of portrait-orientation videos such as those shot on smartphones, resuming playback at the last point watched in the previous session, in-application download and installation of extensions, support for interactive Blu-Ray menus, and "compatibility with a very large number of unusual codecs". The release is available for Linux, Android, and Android TV, plus various Windows and Apple platforms.

Comments (18 posted)

IPython 3.0 released

Comments (none posted)

GNU GNATS 4.2.0 available

Version 4.2.0 of the GNU GNATS bug-tracking tool has been released. This is the first major update of the program in ten years; among the many improvements are the use of Automake as the build system, a license update (GPLv3), and multiple portability fixes.

Full Story (comments: 1)

Buildroot 2015.02 released

Buildroot 2015.02 is available; the changes include a reworked handling of shared libraries, a new warning whenever unsafe paths are encountered, and support for new processor architectures.

Full Story (comments: none)

Calligra 2.9 Released

Version 2.9 of the Caligra office suite has been released. New in this edition is the "Calligra Gemini" edition, which allows the same document to be worked on in the desktop environment and on a touchscreen tablet, as well as support for Microsoft Word documents in the Okular document viewer, and numerous updates and fixes.

Full Story (comments: none)

Development newsletters from the past week

• What's cooking in git.git (February 26)
• What's cooking in git.git (March 2)
• LLVM Weekly (March 2)
• OCaml Weekly News (March 3)
• OpenStack Community Weekly Newsletter (February 27)
• Perl Weekly (March 2)
• PostgreSQL Weekly News (March 1)
• Python Weekly (February 26)
• Qt Weekly (March 3)
• Ruby Weekly (February 26)
• This Week in Rust (March 2)
• Tor Weekly News (March 4)
• Wikimedia Tech News (March 2)

Comments (none posted)

The state of Linux gaming in the SteamOS era (Ars Technica)

Comments (28 posted)

GitLab acquires Gitorious

Comments (39 posted)

Samba user survey

Full Story (comments: none)

Jonas Öberg joins FSFE as Executive Director

Full Story (comments: none)

2015 Linux Jobs Report

Full Story (comments: none)

Free Software Supporter - Issue 83, March 2015

Full Story (comments: none)

FSFE Newsletter - March 2015

Full Story (comments: none)

CFP Deadlines: March 5, 2015 to May 4, 2015

If the CFP deadline for your event does not appear here, please tell us about it.

LibrePlanet

Full Story (comments: none)

pgDay Paris

Full Story (comments: none)

EuroPython 2015

Full Story (comments: none)

DebConf15

Full Story (comments: none)

Events: March 5, 2015 to May 4, 2015

If your event does not appear here, please tell us about it.