|
|
Subscribe / Log in / New account

LWN.net Weekly Edition for June 11, 2015

Systemd and containers

By Jonathan Corbet
June 10, 2015

LinuxCon Japan
As Lennart Poettering noted at the beginning of his LinuxCon Japan talk, containers are "all the rage" these days. He was there to talk about how systemd works with containers — on both the inside and the outside. The systemd developers believe that containers should be a core part of an operating system, a feature that is provided out-of-the-box, rather than being an add-on. The container concept should be integrated all the way through the operating system. This idea, Lennart admitted, is not new; it also characterized the Solaris Zones implementation.

Systemd development is also driven by the idea that the operating system running inside a container should be as similar as possible to that running outside. Applications should not care whether they are running inside a container or not; the operating system should hide the differences between those two settings. Lennart called this concept "integrated isolation." There is isolation in that the container should appear to be its own host, but there should be full integration with the real host; a system-log query, for example, should be able to return results from the host, from one or more containers, or all of the above.

The Docker system, Lennart said, wants to have a single tenant running in each container. Systemd takes a different approach. Daemons that run on the host, for example, should generally run within the container as well.

There are a number of other overriding principles that drive systemd container development. The core system should be minimal, without a lot of options for users to wade through — but the door should be open for alternatives when the need arises. The focus should be on getting the low-level parts right. "No hacks"; the implementation should be clean. That is why Btrfs is used with native systemd containers rather than adding the logical volume [Lennart Poettering] manager to the mix. New standards should not be created when standards already exist; thus, for example, systemd doesn't include its own packaging system since Docker and rkt already exist. Most systemd testing is done inside containers; that simply makes life easier, since it's no longer necessary to reboot the system for each testing cycle. Finally, systemd is not focused on cluster-wide orchestration; instead, it provides the building blocks that allow others to create solutions at that level.

The core of the systemd approach to containers is a daemon called systemd-machined; it serves as a registry for containers running on the system. Other tools that have container awareness integrate with systemd-machined to get their job done. Thus, the systemctl command, when given the -M option, will run a command in an arbitrary container. The -r option causes services running in containers to be listed as well. There is a list-machines subcommand to create a list of containers along with some health status. Other command-line tools, including loginctl, systemd-run, and journalctl, also understand the -M option. The machinectl command-line tool can be used to log into or stop a container. And so on.

Work has been done to integrate with other system tools as well. There is an option to the ps command to see which container owns any given process. Gnome-system-monitor has been extended in similar ways. The name service switch (NSS) subsystem has been enhanced with a module to do hostname resolution for containers. The sd-bus IPC library is container-aware. Importantly, all of these mechanisms work with the various container-management systems available on Linux; they are not limited to systemd's containerization mechanism.

That mechanism is built around systemd-nspawn, a simple container manager. The rkt container system was built around systemd-nspawn at the lowest levels. It is a minimal system which, naturally, integrates with systemd-machined. All it needs is a directory full of operating-system files and it will be able to boot from there. It can also boot raw disk images, the same ones that work with KVM or on bare metal. So, for example, Fedora Cloud images work with systemd-nspawn.

It can also be set up to provide "containers as a service." Each container can be a normal systemd service, meaning that all of systemd's resource-management and process-management features are available.

One of the newer tools is systemd-networkd, added about one year ago. It adds a network-configuration mechanism that, Lennart said, is "minimal" but "really powerful." A systemd-networkd daemon running on the host will, when a virtual Ethernet tunnel appears for a container, pick an IP range for the container, run a DHCP server for it, set up IPv4LL, set up network address translation, etc. When running in the container, instead, it will launch the DHCP client and perform IPv4LL setup. The end result is that networking for containers just works with no additional configuration needed.

Another recent addition is systemd-resolved, which does central hostname resolution. It is a centralized daemon, avoiding many of the hassles that come with NSS resolution; in particular, it maintains a single, global cache. There is no longer a need to run a resolver in each application which, among other things, should reduce the attack surface of the system. Lookups with LLMNR are set up by default.

The systemd-import tool can import and export container images. It understands simple formats like tarballs and raw disk images; it can also import from (but not export to) the Docker format. This tool is typically used via the machinectl pull-raw command to get container images off the net. These images can then be booted directly with systemd-nspawn.

Lennart closed the session with a brief discussion of the concept of stateless systems. These systems have everything they need to boot in /usr; other directories, including /etc and /var, are created on the first boot. With such systems, a factory reset is easily achieved by removing everything except /usr. To achieve a truly stateless system, one just has to always boot in this manner. Stateless systems are useful for containers, since it is possible to instantiate a lot of them and have them set themselves up on the first boot. A single /usr image can be used as a golden master image, making for easy centralized updating. The stateless systems idea is relatively new, but the systemd developers are hoping to push distributors in that direction.

Lennart's slides [PDF] are available for the curious.

[Your editor would like to thank the Linux Foundation for funding his travel to LinuxCon Japan]

Comments (54 posted)

Remote vehicle interaction in automotive Linux

By Nathan Willis
June 10, 2015

ALS 2015

At the 2015 Automotive Linux Summit in Tokyo, a team from Jaguar Land Rover (JLR) gave attendees the first look at the company's plans for implementing remote vehicle interaction (RVI) in Linux-based automotive systems. RVI can encompass a wide array of features that use long-range networking, but JLR's project focuses on just a few: controlling simple mechanical systems with mobile apps, remote data logging, and updating car software over-the-air (SOTA).

[Matt Jones]

The session was co-presented by Matt Jones and Arthur Taylor. Jones noted that there is an RVI expert group within the Automotive Grade Linux (AGL) project and that several GENIVI members have been participating as well. The implementation presented in the session is thus something of a joint effort—JLR being an active member of GENIVI and AGL. The expert group's mission statement is to specify, plan, and build a reference implementation of the infrastructure required for RVI. But, he added, the group had no plans to stop development after it completes support for its initial three RVI use cases. "We'll keep adding things as new use cases are presented. Please bring us your ideas."

The group is also committed to building a working implementation of the entire framework, Jones said. "We don't believe it's enough for this to be a computer science project." A real-world implementation, he said, has to be secure, thoroughly tested, and reliable enough that system manufacturers are willing to integrate it. Thus, the group has developed its implementation in the open; the code is available on GitHub.

The first piece of the system is a server component designed to run in a data center (not in a car). It receives, processes, and forwards messages between vehicles and other software services. On the vehicle's side, there is a corresponding client daemon. But, Jones pointed out, the framework does not require setting up a direct client-to-server link. Rather, the network model uses a store-and-forward design, so that it can cope with poor or intermittent network connections. Vehicles can also relay messages in a peer-to-peer fashion, which is particularly useful for fleet vehicle deployments.

The server supports the creation of basic user and group accounts that can be used to authenticate incoming messages and check that any included commands are authorized. It also tracks vehicles by name, make, model, and vehicle identification number (VIN). How this factors into authentication is most easily seen in the group's "control" demo, which provides an app for Android and iOS that can be used to unlock the car door. Each user account on the server can have multiple cryptographic keys, each of which is bound to a specific mobile device running the app.

Using a web application, the user can grant each key the "lock/unlock" permission for a specific vehicle ahead of time. When the app sends the unlock command to the server, the server checks that the originating device has the correct permissions before it relays the command to the car. For testing, Jones said, the unlock mechanism was a Raspberry Pi wired into an electrical relay on the door lock. The team is currently extending this system to support the control of additional features, with heater/air-conditioning control coming first.

The second use case tackled by the group was remote data collection. This is a somewhat simpler application, as it involves only sending data packets from the vehicle to the RVI server, where the data is logged in a database. The example implementation uses an off-the-shelf on-board diagnostic (OBD-II) dongle to capture status messages, storing the collected data in a Cassandra database.

Unlike the door-locking test, which is a purely internal project, there has already been outside interest in the data-logging feature. The Portland public transportation system got in touch with the group to ask for help developing an open-source solution to collecting data for its bus system, Jones said. A solution for Portland is still in development, but the project recently completed its first data-logging field test, collecting one month's worth of data from a taxi fleet in San Francisco, and developing web applications to plot the data historically as well as provide a live overview of the fleet's current status. Taylor demonstrated both web applications.

The final use case targeted by the team is SOTA, the over-the-air software update functionality, which Jones said can apply to several different types of updates. Manufacturers may want to deliver an update to a specific subsystem, third-party developers will periodically want to update their applications, carmakers may need to issue a critical security patch, and so on. In addition, he said, consumers have gotten used to the idea of upgrading their PC and phone operating systems periodically, so he expects that car owners will want the same feature—few car owners will be content to run a ten-year-old OS when all of their other electronic devices can be updated in the field.

So far, the team has implemented per-package updating. The RVI server notifies the car computer that an update is available and sends a network location from which the car can request the update when it is ready. The car asks the user to authorize the update, after which it downloads the new package. Currently, both RPM and binary diffs are supported package formats. The package sent by the server is first encrypted using a per-vehicle key pair to guard against man-in-the-middle attacks. Jones noted that software key pairs and hardware security modules are both supported by the framework. If the car computer decrypts and validates the package, it can install it and report success back to the RVI server; if not it reports the failure.

From the client's perspective, this is fairly straightforward (not too different from the way packages are updated on other Linux systems). The RVI server has some more complicated matters to keep track of, however. It can queue updates for multiple vehicles, set time windows to open and close access to the update system, restrict access by geographic location, stagger releases out by VIN ranges, and so on.

For now, the system stores the package updates in the RVI server database itself, although plugins to support GitHub and other packages sources are in development. Jones noted that the server can also deliver full disk images, although this has not been rolled out.

All three frameworks are, for now, fairly simple in scope. But they will get more complicated as the project progresses. The system is already modular: a number of database backends are available for the RVI server, and additional services can be added. Jones said that although the project itself is open source, it has been designed so that proprietary plugins will be possible, since some carmakers or software developers are likely to expect them. In response to an audience question, he said that the server is written mostly in Erlang, while the demo applications are primarily in Python, and the web front-ends (such as the unlock-permission application) are based on Django.

Security, of course, is a significant concern. Jones pointed out that the RVI server and client both use standard Linux security modules, leaving the details up to the implementer. Thus, since Tizen IVI uses Smack for access control, its client-side lock/unlock support would use Smack to restrict access to the door-lock service. Another system might use an entirely different configuration.

Another audience member asked how the unlock system would cope with the user's phone being stolen. Jones replied that the user can go to the web site and de-authorize the phone—which makes it more secure than physical car keys, which can also be stolen. Someone in the crowd then asked what would happen if the phone was stolen "in the desert" where there was no Internet access—to which Jones replied it would be much like having one's keys stolen in the desert.

The notion of RVI has long been the source of some unease for a portion of the developer community, perhaps because of the periodic news stories highlighting how non-secure contemporary car-computing systems are. But it is worth noting that this is an area where Linux and open-source software have a chance to improve on the status quo, rather than sink to its level. The AGL expert group still has a lot of work ahead of it to implement a full-fledged RVI system, but it is certainly good to see that it is taking multiple use cases, inconsistent network access, and security in mind from the beginning.

[The author would like to thank the Linux Foundation for travel assistance to attend ALS 2015.]

Comments (4 posted)

Resurrecting the SuperH architecture

By Nathan Willis
June 10, 2015

LinuxCon Japan

Processor architectures are far from trivial; untold millions of dollars and many thousands of hours have likely gone into the creation and refinement of the x86 and ARM architectures that dominate the CPUs in Linux boxes today. But that does not mean that x86 and ARM are the only architectures of value, as Jeff Dionne, Rob Landley, and Shumpei Kawasaki illustrated in their LinuxCon Japan session "Turtles all the way down: running Linux on open hardware." The team has been working on breathing new life into a somewhat older architecture that offers comparable performance to many common system-on-chip (SoC) designs—and which can be produced as open hardware.

[Jeff Dionne]

The architecture in question is Hitachi's SuperH, whose instruction set was a precursor to one used in many ARM Thumb CPUs. But the patents on the most important SuperH designs have all expired—and more will be expiring in the months and years to come—which makes SuperH a candidate for revival. Dionne, Landley, and Kawasaki's session [PDF] outlined the status of their SuperH-based "J2" core design, which can be synthesized in low-cost FGPAs or manufactured in bulk.

Dionne started off the talk by making a case for the value of running open-source software on open hardware. That is a familiar enough position, of course, but he went on to point out that a modern laptop contains many more ARM and MIPS processors than it does x86 processors. These small processors serve as USB and hard-drive controllers, run ACPI and low-level system management services, and much more. Thus, the notion of "taking control of your hardware" has to include these chips as well.

He then asked what constitutes the minimal system that can run Linux. All that is really needed, he said, is a flat, 32-bit memory address space, a CPU with registers to hold instructions, some I/O and storage (from which the kernel and initramfs can be loaded), and a timer for interrupts. That plus GCC is sufficient to get Linux running—although it may not be fast, depending on the specifics. One does not even need a cache, floating-point unit, SMP, or a memory-management unit (MMU).

At this point, Landley chimed in to point out that Dionne had been the maintainer of uClinux, which was an active project maintaining Linux on non-MMU systems up through 2003, when Dionne handed off maintainership to others where, unfortunately, development slowed down considerably. The requirements for running Linux are quite low, though; many of the open-hardware boards popular today (such as the Raspberry Pi) throw in all sorts of unnecessary extras.

That brings us to SuperH, which Dionne said was developed with a "massive research and development outlay." The SuperH SH2 was a highly optimized design, employing a five-stage Harvard RISC architecture with an instruction-set density considerably ahead of its contemporaries. That density is a common way to measure CPU efficiency, he explained; a dense architecture requires fewer instructions and thus fewer clock cycles to perform a given task. Most of a CPU's clock cycles are spent waiting for something, he said; waiting for instructions is such a bottleneck that if you can get them fast enough, "it almost doesn't matter what your clock speed is."

The SuperH architecture is so dense that a 2009 research paper [PDF] plotted it ahead of every architecture other than x86, x86_64, and CRIS v32. ARM even licensed the SuperH patent portfolio to create its Thumb instruction set in the mid-1990s.

Fortunately, the patents are now expiring. The last of the SH2 patents expired in 2014, with more to come. The SH2 processor was, he said, used in the Sega Saturn game console; the SH4 (found in the Sony Sega Dreamcast) will have the last of its patents expire in 2016. Though they are older chips, they were used in relatively powerful devices.

[Shumpei Kawasaki and Rob Landley]

In preparation for this milestone, Dionne, Landley, and others have been working on J2, a clean-room re-implementation of the SH2 that is implemented as a "core design kit." The source for the core is written in VHDL, and it can be synthesized on a Xilinx Spartan6 FPGA. The Spartan6 is a low-cost platform (boards can be purchased for around $50), but it also contains enough room to add additional synthesized components—like a serial controller, memory controller, digital signal processor, and Ethernet controller. In other words, a basic SoC.

The other main advantage of the J2 project is that the work for implementing SuperH support is already done in the kernel, GCC, GDB, strace, and most other system components. By comparison, there are a few other open CPU core projects like OpenRISC and RISC-V, but those developers must write all of their code from scratch—if the CPU core designs ever become stable enough to use. As Landley then added, "we didn't have to write new code; we just had to dig some of it up and dust it off."

The project has thus "inherited" an excellent ISA, and has even been in contact with many of the former Hitachi employees that worked on SuperH. But that is of little consequence if a $50 FPGA is the only hardware target. The Spartan6 is cheap as FPGAs go, but still more than most customers would pay for an SoC. So the J2 build chain not only generates a Xilinx bitstream (the output which is then synthesized onto the FPGA); it also generates an RTL circuit design that can be manufactured by an application-specific integrated circuit (ASIC) fabrication service.

Chip fabrication is not cheap if one shops around for the newest and smallest process, Dionne said—but, in reality, there are many ASIC vendors who are happy to produce low-cost chips on their older equipment because the cost of retooling a plant is exorbitant. A 180nm implementation of the J2 design, he said, costs around three cents per chip, with no royalties required. "That's disposable computing at the 'free toy inside' level."

As of today, the J2 is sufficient to build low-end devices, but the roadmap is heading toward more complex designs as more SuperH patents expire. In 2016, the next iteration, called J2+, will add SMP support and an array of DSPs that will make it usable for signal-processing applications like medical devices and Internet-of-Things (IoT) products like the oft-cited home electricity monitor. A year or so further out, the J4 (based on the SH4 architecture) will add single instruction, multiple data (SIMD) arrays and will be suitable for set-top boxes and automotive computing.

Landley and Dionne then did a live demonstration, booting Linux on a J2 core that they had synthesized onto an off-the-shelf Spartan6 board purchased the day before in Tokyo's Akihabara district. The demo board booted a 3.4 kernel—though it took several seconds—and started a bash prompt. A small victory, but it was enough to warrant a round of applause from the crowd. Dionne noted that they do have support in the works for newer kernels, too. Landley said that he was still in the process of setting up the two public web sites that will document the project. The nommu.org site will document no-MMU Linux development, he said (hopefully replacing the now-defunct uClinux site), while 0pf.org will document the team's hardware work.

In an effort to reduce the hardware cost and bootstrap community interest, the team is also planning a Kickstarter campaign that will produce a development board—hopefully with a more powerful FPGA than the model found on existing starter kits—in a Raspberry-Pi–compatible form factor. By including a larger FPGA, these boards should be compatible with the J4 SMP design; the Lx9 version of Spartan6 (which was used for the J2 development systems) simply does not have enough logic gates for SMP usage.

At the end of the talk, an audience member voiced concern that SuperH was old enough that support for it is unmaintained in a lot of projects. He suggested that the J2 team might need to act quickly to stop its removal. Landley noted that, indeed, the latest buildroot release did remove SuperH support, "but we're trying to get them to put it back now." Luckily, Dionne said, there are other projects keeping general no-MMU support in the kernel up-to-date, such as Blackfin and microblaze. The team has been working on getting no-MMU support into musl and getting some of the relevant GCC build tools "cleaned up" from some minor bit rot.

Another audience member asked whether or not the SuperH ISA was getting too old to be relevant. In response, Dionne handed the microphone over to Kawasaki, who had remained off to the side for the entire presentation up to that point. Kawasaki was one of the original SH2 architects and is now a member of the J2 project. There have been some minor additions, he said: the J2 adds four new instructions. One for atomic operations, one to work around the barrel shifter, "which did not work the way the compiler wanted it to," and a couple that are primarily of interest to assembly programmers. There are always questions about architecture changes, he said, but mostly the question is whether to make the changes mandatory or simply provide them as VHDL overlays. For the most part, though, the architecture already had everything Linux needs and works well, despite its age.

As of today, the nommu.org site is online and has an active mailing list, although the Git repository Landley promised is not yet up and running. The 0pf.org site is also up and running, and contains much more in the way of documentation. While the project is still in its early stages, it seems to be generating considerable interest, and with several more iterations of open CPU designs still to come.

[The author would like to thank the Linux Foundation for travel assistance to attend LCJ 2015.]

Comments (41 posted)

Page editor: Jonathan Corbet

Security

Entropy with NeuG

By Nathan Willis
June 10, 2015

LinuxCon Japan

At LinuxCon Japan 2015, Yutaka Niibe presented [PDF] his work on NeuG, a true random-number generator (TRNG) built entirely with free software and which can run on a variety of cheap hardware devices. There are several competing free-software TRNG options on the market these days; NeuG may not generate the fastest stream of random numbers, but it is notable for its flexible hardware support and its simplicity.

Niibe began the session with a brief autobiographical sketch. He has been a long-time GnuPG developer, with much of his work focusing on libgcrypt. Several years ago, he began working on a GnuPG smartcard implementation called Gnuk. The Gnuk project involved designing and building a secure hardware token that he dubbed the FST-01. Along the way, he said, he became convinced that users were in need of a hardware TRNG.

[Yutaka Niibe]

The issue, of course, is that cryptography requires good keys and generating good keys require random numbers. Interest in the subject is on the increase, he said, with projects like the Free Software Foundation's (FSF's) Email Self Defense and the Electronic Frontier Foundation's (EFF's) HTTPS Everywhere but, he reminded the audience, GnuPG is a fundamental part of free-software distribution, too, providing signatures for releases, commits, and packages.

Given the importance of the topic, hardware TRNGs are vital because users have to trust the hardware in order to trust their encryption keys. There may be backdoors in algorithms, he noted, but there may also be backdoors in hardware—as has been alleged about Intel's RDRAND instruction or in cryptographic chips. So he decided to buck the recent trend of building more powerful embedded devices, and set out to develop a TRNG that uses a cheap, general-purpose microcontroller rather than any specialty hardware.

NeuG is the result. The primary focus has been on implementing NeuG as an alternate firmware for the FST-01, he said, but it will run on several other devices. At the moment, there are five, all of which use the STM32F103 microcontroller found on the FST-01. The microcontroller features an ARM Cortex-M3 core running at 72MHz and has full-speed USB 2.0 connectivity. The various boards include between 20 and 64KiB of RAM and between 64 and 512KiB of flash ROM.

The cheapest option is the ST-Link debugger board that ships with ST's STM8S-DISCOVERY kit; it can be had online for less than ten dollars. The FST-01 remains the reference design, though. It includes 20KiB of RAM, 128KiB of ROM, a 12MHz crystal, a status LED, and some additional flash memory. The schematics and circuit-board designs are free as well.

When connected to a Linux PC, the FST-01 appears as /dev/ttyACM0 (or whatever /dev/ttyACM device is available) and also presents a small USB mass-storage device holding a copy of the GPLv3 license and a README file for using the software. The random-number generation works by sampling the microcontroller's two built-in analog-to-digital converters (ADCs). One ADC samples the microcontroller's reference voltage pin and built-in temperature sensor, while the other is tied to two of the microcontroller's analog input pins (both of which are pulled up to the power supply pin). These four signal sources are sampled in various combinations, passed through four rounds of CRC32, then passed through a SHA-256 conditioning function.

The output, Niibe said, is about 256 bits of entropy from every 280 bits sampled. The FST-01 produces around 80KB of entropy per second in this fashion. The NIST SP 800-90B standard defines the conditioning requirements needed to remove sampling bias, he said, and it defines three health checks that one can use to test the quality of the output. The NeuG software includes a Python program to test the device's output, he said, "but if you don't trust me, you can test it with external tools, too." He recommended TestU01 and PractRand, though he cautioned that they can require extremely large data sets for proper analysis.

Normally, he continued, the device is used with the kernel's rng-tools; the NeuG software bundle includes a udev rule and systemd service to feed output into the kernel's entropy pool. But it can be sampled directly, too, with raw samples, CRC32-filtered bits, and fully SHA-256–conditioned output all available. Altogether, the NeuG library is quite compact: the version that ships on the FST-01 is just 1,500 lines of code, including the USB stack, the USB communications device class (CDC) and mass-storage drivers, and a small thread library called Chopstx.

Regarding the performance of the TRNG design, Niibe admitted that many electrical engineers prefer to talk about other, more esoteric entropy sources like avalanche diodes, photon decay, and radio waves. They sound better, he said, but such exotic sources are not usually needed in practice. People also ask him whether NeuG offers any advantage over haveged, to which he replies that haveged may be good enough for many people on its own, but that it is always better to have multiple, independent entropy sources.

An audience member asked how much entropy a system needs. Niibe replied that some people go overboard, but in his opinion it is best to think of entropy like spice. "Don't drink your soy sauce," he said to a collective chuckle from the crowd, "just use a little wasabi if you want to enjoy your sushi."

Others in attendance asked about the use of the microcontroller at the core of the project. Could the code be ported to other microcontrollers, perhaps? And is there any protection against hardware trojans in the microcontroller itself? Niibe replied that the code is quite portable, and could probably be ported to a variety of other microcontrollers if someone is interested.

As for the possibility of hardware trojans, he said that the risk of compromised hardware was the reason he chose a modest, general-purpose part, adding "although now that I've given this talk, I guess the bad guys will start attacking it." But the STM32F103 has been around for eight years or so, he said, so if users are really paranoid, they should just go buy one now (or even try to find an old one) before the spies get to them. He is also occasionally asked why the NeuG uses a TTY with no encryption to deliver its data, he said. His response is that he could implement USB-channel encryption, but he does not think it makes sense to encrypt your USB bus. If the NSA is snooping your internal USB traffic, he said, "I think you're already finished."

Niibe ended the session with a few "further reading" recommendations for those interested in random-number generation. In summary, he said that free software is about controlling one's own computing. Where encryption is concerned, we need something at the center to serve as the anchor point. No one can control the random number sequence, he said, so he would like to see more people use hardware TRNGs, whether they buy one of his or another.

The most direct comparison to NeuG might be Moonbase Otago's OneRNG, which we looked at in January. Both are open hardware in addition to running free software. The OneRNG uses an avalanche diode as its entropy source and its bit rate is estimated at 350Kbps. So users have some decisions to make if they are interested in choosing one or the other—the NeuG may not produce as much entropy, but it may produce enough for daily usage. On the other hand, the OneRNG is a recent Kickstarter product: although shipments have begun, there is only one source for the hardware, as opposed to the five different boards supported by NeuG. Regardless of how those factors weigh out, however, it is good to have multiple options—and options that take substantially different approaches to the problem of random-number generation.

[The author would like to thank the Linux Foundation for travel assistance to attend LCJ 2015.]

Comments (8 posted)

Brief items

Security quotes of the week

If we don't vigorously fight back against government efforts to weaken encryption, we're all going to be badly burned.
Lauren Weinstein

Yes. No. Yes. Maybe. Yes. Okay, it's complicated.
Bruce Schneier "answers" the question of "Should Companies Do Most of Their Computing in the Cloud?" (in three parts: part 2 and part 3)

Comments (none posted)

Let's Encrypt Root and Intermediate Certificates

The Let's Encrypt project has announced that it has created the root and intermediate keys and certificates it will use to sign certificates. Let's Encrypt is the no-cost certificate authority announced by the Electronic Frontier Foundation (EFF) back in November. In April, the Linux Foundation announced that it would be hosting the project. "The keys and certificates that will underlie Let’s Encrypt have been generated. This was done during a key ceremony at a secure facility today." The intermediate certificates will be cross-signed by IdenTrust so that they will be accepted by browsers before the Let's Encrypt root certificate has been propagated. A bit more news from the blog post: "In the next few weeks, we’ll be saying some more about our plans for going live."

Comments (42 posted)

New vulnerabilities

abrt: multiple vulnerabilities

Package(s):abrt CVE #(s):CVE-2015-1869 CVE-2015-1870 CVE-2015-3142 CVE-2015-3147 CVE-2015-3150 CVE-2015-3151 CVE-2015-3159 CVE-2015-3315
Created:June 10, 2015 Updated:July 8, 2015
Description: From the Red Hat advisory:

It was found that ABRT was vulnerable to multiple race condition and symbolic link flaws. A local attacker could use these flaws to potentially escalate their privileges on the system. (CVE-2015-3315)

It was discovered that the kernel-invoked coredump processor provided by ABRT wrote core dumps to files owned by other system users. This could result in information disclosure if an application crashed while its current directory was a directory writable to by other users (such as /tmp). (CVE-2015-3142)

It was discovered that the default event handling scripts installed by ABRT did not handle symbolic links correctly. A local attacker with write access to an ABRT problem directory could use this flaw to escalate their privileges. (CVE-2015-1869)

It was found that the ABRT event scripts created a user-readable copy of an sosreport file in ABRT problem directories, and included excerpts of /var/log/messages selected by the user-controlled process name, leading to an information disclosure. (CVE-2015-1870)

It was discovered that, when moving problem reports between certain directories, abrt-handle-upload did not verify that the new problem directory had appropriate permissions and did not contain symbolic links. An attacker able to create a crafted problem report could use this flaw to expose other parts of ABRT to attack, or to overwrite arbitrary files on the system. (CVE-2015-3147)

Multiple directory traversal flaws were found in the abrt-dbus D-Bus service. A local attacker could use these flaws to read and write arbitrary files as the root user. (CVE-2015-3151)

It was discovered that the abrt-dbus D-Bus service did not properly check the validity of the problem directory argument in the ChownProblemDir, DeleteElement, and DeleteProblem methods. A local attacker could use this flaw to take ownership of arbitrary files and directories, or to delete files and directories as the root user. (CVE-2015-3150)

It was discovered that the abrt-action-install-debuginfo-to-abrt-cache helper program did not properly filter the process environment before invoking abrt-action-install-debuginfo. A local attacker could use this flaw to escalate their privileges on the system. (CVE-2015-3159)

Alerts:
Scientific Linux SLSA-2015:1210-1 abrt 2015-07-07
Oracle ELSA-2015-1210 abrt 2015-07-07
Red Hat RHSA-2015:1210-01 abrt 2015-07-07
Fedora FEDORA-2015-10193 libreport 2015-06-30
Fedora FEDORA-2015-10193 gnome-abrt 2015-06-30
Fedora FEDORA-2015-10193 abrt 2015-06-30
Fedora FEDORA-2015-9886 satyr 2015-06-21
Fedora FEDORA-2015-9886 libreport 2015-06-21
Fedora FEDORA-2015-9886 gnome-abrt 2015-06-21
Fedora FEDORA-2015-9886 abrt 2015-06-21
Oracle ELSA-2015-1083 abrt 2015-06-09
Red Hat RHSA-2015:1083-01 abrt 2015-06-09
CentOS CESA-2015:1083 abrt 2015-06-15
Scientific Linux SLSA-2015:1083-1 abrt 2015-06-10

Comments (none posted)

cups: two vulnerabilities

Package(s):cups CVE #(s):CVE-2015-1158 CVE-2015-1159
Created:June 9, 2015 Updated:November 2, 2015
Description: From the Debian LTS advisory:

CVE-2015-1158 - Improper Update of Reference Count
Cupsd uses reference-counted strings with global scope. When parsing a print job request, cupsd over-decrements the reference count for a string from the request. As a result, an attacker can prematurely free an arbitrary string of global scope. They can use this to dismantle ACL’s protecting privileged operations, and upload a replacement configuration file, and subsequently run arbitrary code on a target machine.

This bug is exploitable in default configurations, and does not require any special permissions other than the basic ability to print.

CVE-2015-1159 - Cross-Site Scripting
A cross-site scripting bug in the CUPS templating engine allows the above bug to be exploited when a user browses the web. This XSS is reachable in the default configuration for Linux instances of CUPS, and allows an attacker to bypass default configuration settings that bind the CUPS scheduler to the ‘localhost’ or loopback interface.

Alerts:
Gentoo 201510-07 cups 2015-10-31
Slackware SSA:2015-188-01 cups 2015-07-07
Fedora FEDORA-2015-9801 cups 2015-06-21
Fedora FEDORA-2015-9726 cups 2015-06-21
Mageia MGASA-2015-0247 cups 2015-06-19
SUSE SUSE-SU-2015:1041-1 cups 2015-06-11
SUSE SUSE-SU-2015:1044-1 cups154 2015-06-11
Ubuntu USN-2629-1 cups 2015-06-10
Debian-LTS DLA-239-1 cups 2015-06-09
Oracle ELSA-2015-1123 cups 2015-06-17
SUSE SUSE-SU-2015:1044-2 cups154 2015-06-11
SUSE SUSE-SU-2015:1011-1 cups 2015-06-08
Scientific Linux SLSA-2015:1123-1 cups 2015-06-17
Oracle ELSA-2015-1123 cups 2015-06-17
CentOS CESA-2015:1123 cups 2015-06-18
Debian DSA-3283-1 cups 2015-06-09
CentOS CESA-2015:1123 cups 2015-06-18
Red Hat RHSA-2015:1123-01 cups 2015-06-17
openSUSE openSUSE-SU-2015:1056-1 cups 2015-06-12
Arch Linux ASA-201506-2 cups 2015-06-10

Comments (none posted)

jackrabbit: information leak

Package(s):jackrabbit CVE #(s):CVE-2015-1833
Created:June 9, 2015 Updated:July 1, 2015
Description: From the CVE entry:

XML external entity (XXE) vulnerability in Apache Jackrabbit before 2.0.6, 2.2.x before 2.2.14, 2.4.x before 2.4.6, 2.6.x before 2.6.6, 2.8.x before 2.8.1, and 2.10.x before 2.10.1 allows remote attackers to read arbitrary files and send requests to intranet servers via a crafted WebDAV request.

Alerts:
Debian DSA-3298-1 jackrabbit 2015-07-01
Mageia MGASA-2015-0242 jackrabbit 2015-06-08

Comments (none posted)

libapache-mod-jk: information disclosure

Package(s):libapache-mod-jk CVE #(s):CVE-2014-8111
Created:June 4, 2015 Updated:July 1, 2015
Description: From the Debian advisory:

An information disclosure flaw due to incorrect JkMount/JkUnmount directives processing was found in the Apache 2 module mod_jk to forward requests from the Apache web server to Tomcat. A JkUnmount rule for a subtree of a previous JkMount rule could be ignored. This could allow a remote attacker to potentially access a private artifact in a tree that would otherwise not be accessible to them.

Alerts:
Mageia MGASA-2015-0254 apache-mod_jk 2015-07-01
Debian-LTS DLA-240-1 libapache-mod-jk 2015-06-09
Debian DSA-3278-1 libapache-mod-jk 2015-06-03

Comments (none posted)

mbedtls: code execution

Package(s):mbedtls CVE #(s):
Created:June 9, 2015 Updated:June 19, 2015
Description: From the mbed TLS 1.3.11 release announcement:

Handling of the SSL_VERIFY_OPTIONAL authmode was changed to make sure that information about keyUsage and extendedKeyUsage was properly propagated and accessible to the calling function. Just as a reminder, SSL_VERIFY_REQUIRED is the recommended security setting for authmode.

A team of security researchers dove back into the Lucky 13 issue and found potential weaknesses in SSL implementations when the attacker can execute code on the same physical machine. mbed TLS was modified to always generate a cache miss and make sure that attackers gain no information from that event.

Alerts:
Fedora FEDORA-2015-9625 mbedtls 2015-06-18
Fedora FEDORA-2015-9614 mbedtls 2015-06-18
Fedora FEDORA-2015-9607 mbedtls 2015-06-09

Comments (none posted)

pcre: code execution

Package(s):pcre CVE #(s):CVE-2015-3210
Created:June 5, 2015 Updated:November 25, 2015
Description: From the Arch Linux advisory:

Several buffer overflows have been found in pcre <= 8.37. By compiling a crafted regular expression, it is possible to write more than the expected size into various buffers, allowing arbitrary code execution.

Alerts:
openSUSE openSUSE-SU-2016:2805-1 pcre 2016-11-15
Red Hat RHSA-2016:2750-01 rh-php56 2016-11-15
Red Hat RHSA-2016:1132-01 rh-mariadb100-mariadb 2016-05-26
openSUSE openSUSE-SU-2016:3099-1 pcre 2016-12-12
Ubuntu USN-2943-1 pcre3 2016-03-29
Fedora FEDORA-2016-f59a8ff5d0 mingw-pcre 2016-02-17
Fedora FEDORA-2016-fd1199dbe2 mingw-pcre 2016-02-17
Slackware SSA:2015-328-01 pcre 2015-11-24
Ubuntu USN-2694-1 pcre3 2015-07-29
Fedora FEDORA-2015-11019 pcre 2015-07-18
Fedora FEDORA-2015-11027 pcre 2015-07-13
Arch Linux ASA-201506-1 pcre 2015-06-05

Comments (none posted)

pcs: access restriction bypass

Package(s):pcs CVE #(s):CVE-2015-3983
Created:June 5, 2015 Updated:June 10, 2015
Description: From the Red Hat bugzilla entry:

It was found that the pcs daemon did not sign cookies containing session data that were sent to clients connecting via the pcsd web UI. A remote attacker could use this flaw to forge cookies and bypass authorization checks, possibly gaining elevated privileges in the pcsd web UI.

Alerts:
Fedora FEDORA-2015-8765 pcs 2015-06-04
Fedora FEDORA-2015-8788 pcs 2015-06-04
Fedora FEDORA-2015-8761 pcs 2015-06-04

Comments (none posted)

python-tornado: side-channel attack

Package(s):python-tornado CVE #(s):CVE-2014-9720
Created:June 9, 2015 Updated:May 16, 2016
Description: From the Tornado 3.2.2 release announcement:

The XSRF token is now encoded with a random mask on each request. This makes it safe to include in compressed pages without being vulnerable to the BREACH attack. This applies to most applications that use both the xsrf_cookies and gzip options (or have gzip applied by a proxy).

Alerts:
Debian-LTS DLA-475-1 python-tornado 2016-05-15
openSUSE openSUSE-SU-2015:1998-1 python-tornado 2015-11-16
Debian-LTS DLA-279-1 python-tornado 2015-07-22
Mageia MGASA-2015-0251 python-tornado 2015-07-01
Fedora FEDORA-2015-9143 python-tornado 2015-06-09
Fedora FEDORA-2015-8606 python-tornado 2015-06-10

Comments (none posted)

qemu: multiple vulnerabilities

Package(s):qemu, qemu-kvm CVE #(s):CVE-2015-3209 CVE-2015-4037 CVE-2015-4103 CVE-2015-4104 CVE-2015-4105 CVE-2015-4106
Created:June 10, 2015 Updated:September 10, 2015
Description: From the Ubuntu advisory:

Matt Tait discovered that QEMU incorrectly handled the virtual PCNET driver. A malicious guest could use this issue to cause a denial of service, or possibly execute arbitrary code on the host as the user running the QEMU process. In the default installation, when QEMU is used with libvirt, attackers would be isolated by the libvirt AppArmor profile. (CVE-2015-3209)

Kurt Seifried discovered that QEMU incorrectly handled certain temporary files. A local attacker could use this issue to cause a denial of service. (CVE-2015-4037)

Jan Beulich discovered that the QEMU Xen code incorrectly restricted write access to the host MSI message data field. A malicious guest could use this issue to cause a denial of service. This issue only applied to Ubuntu 14.04 LTS, Ubuntu 14.10 and Ubuntu 15.04. (CVE-2015-4103)

Jan Beulich discovered that the QEMU Xen code incorrectly restricted access to the PCI MSI mask bits. A malicious guest could use this issue to cause a denial of service. This issue only applied to Ubuntu 14.04 LTS, Ubuntu 14.10 and Ubuntu 15.04. (CVE-2015-4104)

Jan Beulich discovered that the QEMU Xen code incorrectly handled MSI-X error messages. A malicious guest could use this issue to cause a denial of service. This issue only applied to Ubuntu 14.04 LTS, Ubuntu 14.10 and Ubuntu 15.04. (CVE-2015-4105)

Jan Beulich discovered that the QEMU Xen code incorrectly restricted write access to the PCI config space. A malicious guest could use this issue to cause a denial of service, obtain sensitive information, or possibly execute arbitrary code. This issue only applied to Ubuntu 14.04 LTS, Ubuntu 14.10 and Ubuntu 15.04. (CVE-2015-4106)

Alerts:
Gentoo 201604-03 xen 2016-04-05
SUSE SUSE-SU-2016:0658-1 Xen 2016-03-04
Mageia MGASA-2016-0098 xen 2016-03-07
openSUSE openSUSE-SU-2015:2249-1 xen 2015-12-10
openSUSE openSUSE-SU-2015:2003-1 xen 2015-11-17
openSUSE openSUSE-SU-2015:1965-1 xen 2015-11-12
openSUSE openSUSE-SU-2015:1964-1 xen 2015-11-12
SUSE SUSE-SU-2015:1952-1 xen 2015-11-10
SUSE SUSE-SU-2015:1908-1 xen 2015-11-04
SUSE SUSE-SU-2015:1894-1 xen 2015-11-03
SUSE SUSE-SU-2015:1853-1 xen 2015-10-30
Gentoo 201510-02 qemu 2015-10-31
SUSE SUSE-SU-2015:1643-1 Xen 2015-09-25
SUSE SUSE-SU-2015:1426-1 kvm 2015-08-21
Fedora FEDORA-2015-13402 qemu 2015-08-18
Mageia MGASA-2015-0310 qemu 2015-08-11
SUSE SUSE-SU-2015:1519-1 qemu 2015-09-09
Fedora FEDORA-2015-13404 qemu 2015-09-01
SUSE SUSE-SU-2015:1206-1 Xen 2015-07-08
SUSE SUSE-SU-2015:1156-1 Xen 2015-06-29
SUSE SUSE-SU-2015:1157-1 Xen 2015-06-29
Scientific Linux SLSA-2015:1189-1 kvm 2015-06-26
SUSE SUSE-SU-2015:1152-1 KVM 2015-06-26
Oracle ELSA-2015-1189 kvm 2015-06-25
CentOS CESA-2015:1189 kvm 2015-06-26
Red Hat RHSA-2015:1189-01 kvm 2015-06-25
Fedora FEDORA-2015-9965 xen 2015-06-24
Fedora FEDORA-2015-9978 xen 2015-06-24
Fedora FEDORA-2015-10001 xen 2015-06-24
openSUSE openSUSE-SU-2015:1094-1 xen 2015-06-22
openSUSE openSUSE-SU-2015:1092-1 xen 2015-06-22
Fedora FEDORA-2015-9599 qemu 2015-06-21
Fedora FEDORA-2015-9466 xen 2015-06-14
Debian DSA-3284-1 qemu 2015-06-12
Red Hat RHSA-2015:1087-01 qemu-kvm 2015-06-10
SUSE SUSE-SU-2015:1045-1 Xen 2015-06-11
Fedora FEDORA-2015-9601 qemu 2015-06-11
Fedora FEDORA-2015-9456 xen 2015-06-14
Scientific Linux SLSA-2015:1087-1 qemu-kvm 2015-06-10
Debian DSA-3286-1 xen 2015-06-13
SUSE SUSE-SU-2015:1042-1 xen 2015-06-11
Oracle ELSA-2015-1087 qemu-kvm 2015-06-10
CentOS CESA-2015:1087 qemu-kvm 2015-06-10
Debian DSA-3285-1 qemu-kvm 2015-06-12
Red Hat RHSA-2015:1089-01 qemu-kvm-rhev 2015-06-10
Ubuntu USN-2630-1 qemu, qemu-kvm 2015-06-10

Comments (none posted)

rabbitmq-server: multiple vulnerabilities

Package(s):rabbitmq-server CVE #(s):CVE-2014-9649 CVE-2014-9650 CVE-2015-0862
Created:June 9, 2015 Updated:March 9, 2016
Description: From the CVE entries:

Cross-site scripting (XSS) vulnerability in the management plugin in RabbitMQ 2.1.0 through 3.4.x before 3.4.1 allows remote attackers to inject arbitrary web script or HTML via the path info to api/, which is not properly handled in an error message. (CVE-2014-9649)

CRLF injection vulnerability in the management plugin in RabbitMQ 2.1.0 through 3.4.x before 3.4.1 allows remote attackers to inject arbitrary HTTP headers and conduct HTTP response splitting attacks via the download parameter to api/definitions. (CVE-2014-9650)

Multiple cross-site scripting (XSS) vulnerabilities in the management web UI in the RabbitMQ management plugin before 3.4.3 allow remote authenticated users to inject arbitrary web script or HTML via (1) message details when a message is unqueued, such as headers or arguments; (2) policy names, which are not properly handled when viewing policies; (3) details for AMQP network clients, such as the version; allow remote authenticated administrators to inject arbitrary web script or HTML via (4) user names, (5) the cluster name; or allow RabbitMQ cluster administrators to (6) modify unspecified content. (CVE-2015-0862)

Alerts:
Red Hat RHSA-2016:0368-01 rabbitmq-server 2016-03-08
Red Hat RHSA-2016:0369-01 rabbitmq-server 2016-03-08
Red Hat RHSA-2016:0367-01 rabbitmq-server 2016-03-08
Red Hat RHSA-2016:0308-01 rabbitmq-server 2016-02-29
Mageia MGASA-2015-0240 rabbitmq-server 2015-06-08

Comments (none posted)

redis: code execution

Package(s):redis CVE #(s):CVE-2015-4335
Created:June 8, 2015 Updated:October 6, 2015
Description: From the Debian advisory:

It was discovered that redis, a persistent key-value database, could execute insecure Lua bytecode by way of the EVAL command. This could allow remote attackers to break out of the Lua sandbox and execute arbitrary code.

Alerts:
Gentoo 201702-16 redis 2017-02-21
openSUSE openSUSE-SU-2015:1687-1 redis 2015-10-06
Red Hat RHSA-2015:1676-01 redis 2015-08-24
Fedora FEDORA-2015-9498 redis 2015-07-18
Fedora FEDORA-2015-9488 redis 2015-07-18
Mageia MGASA-2015-0244 redis 2015-06-19
Debian DSA-3279-1 redis 2015-06-06

Comments (none posted)

strongswan: information disclosure

Package(s):strongswan CVE #(s):CVE-2015-4171
Created:June 8, 2015 Updated:June 18, 2015
Description: From the Debian advisory:

When an IKEv2 client authenticates the server with certificates and the client authenticates itself to the server using pre-shared key or EAP, the constraints on the server certificate are only enforced by the client after all authentication steps are completed successfully. A rogue server which can authenticate using a valid certificate issued by any CA trusted by the client could trick the user into continuing the authentication, revealing the username and password digest (for EAP) or even the cleartext password (if EAP-GTC is accepted).

Alerts:
Ubuntu USN-2628-1 strongswan 2015-06-08
openSUSE openSUSE-SU-2015:1082-1 strongswan 2015-06-18
Debian-LTS DLA-244-1 strongswan 2015-06-12
Debian DSA-3282-1 strongswan 2015-06-08

Comments (none posted)

thermostat1: code execution

Package(s):thermostat1 CVE #(s):CVE-2015-3201
Created:June 4, 2015 Updated:June 11, 2015
Description: From the Red Hat advisory:

It was discovered that the Thermostat web application stored database authentication credentials in a world-readable configuration file. A local user on a system running the Thermostat web application could use this flaw to access and modify monitored JVM data, or perform actions on connected JVMs. (CVE-2015-3201)

Alerts:
Fedora FEDORA-2015-8867 thermostat 2015-06-09
Red Hat RHSA-2015:1052-01 thermostat1 2015-06-04
Fedora FEDORA-2015-8919 thermostat 2015-06-10

Comments (none posted)

zarafa: file overwrites

Package(s):zarafa CVE #(s):CVE-2015-3436
Created:June 8, 2015 Updated:June 10, 2015
Description: From the Red Hat bugzilla:

Guido Günther detected and reported that replacing "/tmp/zarafa-upgrade-lock" by a symlink makes the zarafa-server process following that symlink and thus allows to overwrite arbitrary files in the filesystem (assuming zarafa-server runs as root which is not case by default at Fedora, but upstream default). One just needs write permissions in /tmp and wait until the zarafa-server is restarted.

Alerts:
Fedora FEDORA-2015-8479 zarafa 2015-06-05
Fedora FEDORA-2015-8487 zarafa 2015-06-05

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 4.1-rc7, released on June 7. "Normally rc7 tends to be the last rc release, and there's not a lot going on to really merit anything else this time around. However, we do still have some pending regressions, and as mentioned last week I also have my yearly family vacation coming up, so we'll have an rc8 and an extra week before 4.1 actually gets released."

Stable updates: 4.0.5, 3.14.44, and 3.10.80 were released on June 6.

Comments (none posted)

Quotes of the week

And naming matters: a good name is both descriptive and short, as we know it from the excellent examples of 'Linux' and 'Git'. Oh wait ...
Ingo Molnar

Why not go the full monty and call it strtrtsstrrrtst(), with strtrtssstrrtst() doing almost, but not quite the same thing?
Al Viro

Comments (6 posted)

Huston: Multipath TCP

Geoff Huston has written a lengthy column on multipath TCP. "For many scenarios there is little value in being able to use multiple addresses. The conventional behavior is where each new session is directed to a particular interface, and the session is given an outbound address as determined by local policies. However, when we start to consider applications where the binding of location and identity is more fluid, and where network connections are transient, and the cost and capacity of connections differ, as is often the case in todays mobile cellular radio services and in WiFi roaming services, then having a session that has a certain amount of agility to switch across networks can be a significant factor." (See also: LWN's look at the Linux multipath TCP implementation from 2013).

Comments (2 posted)

Kernel development news

The difficult task of doing nothing

By Jonathan Corbet
June 9, 2015

LinuxCon Japan
Kristen Accardi started her LinuxCon Japan session with the claim that idle is the most important workload on most client systems. Computers placed in offices are busy less than 25% of the time; consumer systems have even less to do. So idle performance, especially with regard to power consumption, is important. The good news is that hardware engineers have been putting a lot of work into reducing the power consumption of idle systems; the bad news is that operating systems are often failing to take full advantage of that work.

In the "good old days," Kristen said, power management was relatively easy — and relatively ineffective. The "Advanced Power Management" (APM) mechanism was entirely controlled by the BIOS, so operating systems didn't have to pay much attention to it. Intel's "SpeedStep" offered exactly one step of CPU frequency scaling. The operating system could concern itself with panel dimming on laptop systems. That was about the extent of the power-management capabilities provided by the hardware at that time.

With the rise of the mobile market, though, power management started to get more complicated. ACPI was introduced, putting more power-management work into the operating system's domain. With ACPI came the notion of "S-states" (for system-wide power-management states), "C-states" (for CPU idle states), and "P-states" (for performance-level states — frequency and voltage scaling). There can be up to 25 of these states.

But things do not stop there; in recent years there has been an explosion of power-management features. They have names like SOix (a new low-power state) and PSR ("panel self refresh"). All of these features must be understood by the operating system, and all must work together for them to be effective.

Degrees of idleness

There are, Kristen said, three fundamental degrees of idleness in a system, differing in the amount of power they use and the time it takes to get back to an active state. The level with the lowest power consumption is "off." That is an increasingly uninteresting state, though; many consumer devices no longer have an "off" switch at all. Operating system support for the "off" state tends to be easy, so there wasn't much to talk about there.

The other two states are "suspend" and "runtime idle". A suspended system is in an intermediate state between running and off; runtime idle is closer [Kristen Accardi] to a running system with nearly instant response when needed. The support for the two states in the kernel is entirely different in a number of ways. Suspend is a system-wide state initiated by the user, while runtime idle is a device-specific state that happens opportunistically. In a suspended system, all tasks are frozen and all devices are forced into the idle state; with runtime idle, instead, tasks are still scheduled and devices may be active. Suspend can happen at any time, while runtime idle only comes into play when a device is idle.

Device drivers must support these two states separately; it is more work, but it's important to do. But platform-level support is also important. In current times, everything is a system-on-chip (SoC) with a great many interactions between components. If one of those components is active, it can keep the entire system out of the lower-power states.

To see how that can come to pass, consider the "latency tolerance reporting" (LTR) mechanism built into modern buses. Any device on the bus can indicate that it may need the CPU's attention within a given maximum time (the latency tolerance). The CPU, in turn, maintains a table describing the amount of time required to return to active operation from each of its idle states. When the CPU is ready to go into a low-power state, the latency requirements declared by active devices will be compared against that table to determine the lowest state that the CPU can go into. So, if a device is running and declaring a tight latency tolerance, it can prevent the CPU from entering a deep-idle state.

Where the trouble lies

Kristen then gave a tour of the parts of the system that are, in her experience, particularly likely to trip things up. At the top of the list was graphics processors (GPUs); these are complex devices and it tends to take quite a while to get power management working properly on them. The "RC6" mechanism describes a set of power states for GPUs; naturally, one wants the GPU to be in a low-power state when it doesn't have much to do. Beyond that, framebuffer compression can reduce memory bandwidth use depending on what's in the framebuffer; sending less video data results in lower power usage. Kristen suggested that users choose a simple (highly compressible) background image on their devices for the best power behavior. "Panel self refresh" allows the CPU to stop sending video data to the screen entirely if things are not changing; it can be inhibited by things like animated images on the screen.

Another "problem child" is audio. On many systems, audio data can be routed through the GPU, preventing it from going into an idle state. Audio devices tend to be complex, consisting of, at a minimum, a controller and a codec; drivers must manage power-management states for both of those devices together.

On the USB side, the USB 3.0 specification added a number of useful power-management features. USB 2.0 had a "selective suspend" feature, but it adds a lot of latency, reducing its usefulness. In 3.0, the controller can suspend itself, but only if all connected devices are suspended. The USB "link power management" mechanism can detect low levels of activity and reduce power usage.

There are three power-management technologies available for SATA devices. The link power management mechanism can put devices into a sleep state and, if warranted, turn the bus off entirely. "ZPODD" is power management for optical devices, but Kristen has never seen anybody actually use it; optical devices are, in general, not as prevalent as they once were. The SATA controller itself offers some power-management features, but they tend to be problematic, she said, so they are not much used in Linux.

The PCI Express bus has a number of power-management options, including ASPM for link-level management, RTPM as a runtime power-management feature, and latency tolerance reporting. The I2C bus has fewer features, being a simpler bus, but it is usually possible to power-down I2C controllers. Human-input devices, which are often connected via I2C, tend to stay powered up while they are open, which can be a problem for system-wide power management.

And, of course, software activity can keep a system from going into deep idle states. If processes insist on running, the CPU will stay active, leaving suspend as the only viable option for power savings. Even brief periods running in the CPU can, if they cause it to wake from idle often, significantly reduce battery life.

Idle together

The conclusion from all of this is that power management requires a coordinated effort. For a system to go into a low-power state, a number of things must happen. User space must be quiet, the platform must support low-power states across all devices, and the kernel must properly support each device's power-management features. The system must also be configured properly; Kristen expressed frustration at mainstream distributions that fail to set up proper power management at installation time, wasting the effort that has been put into power-management support at the lower levels. Getting all of the pieces to work together properly can be a difficult task, but the result — systems that efficiently run our most important workload — is worth the trouble.

[Your editor would like to thank the Linux Foundation for funding his travel to LinuxCon Japan]

Comments (3 posted)

Enforcing mount options for sysfs and proc

By Jake Edge
June 10, 2015

The sysfs and proc filesystems do not contain executables, setuid programs, or device nodes, but they are typically mounted with flags (i.e. noexec, nosuid, and nodev) that disallow those types of files anyway. Currently, those flags are not enforced when those filesystems are mounted inside of a namespace (i.e. the mount will succeed without those flags being specified). Furthermore, a sysfs or proc filesystem that has been mounted read-only in the host can be mounted read-write inside a namespace, which is a bigger problem. The others are subtle security holes, or they could be, so Eric Biederman is trying to close them, though it turns out that the fixes break some container code.

In mid-May, Biederman posted a patch set meant to address the problems, which boil down to differences in the behavior of mounting these filesystems inside a container versus bind-mounting them. In the latter case, any restrictions the administrator has placed on the original proc and sysfs mounts will be enforced on the bind mounts. If, instead, those filesystems are directly mounted inside of a user namespace, those restrictions won't be enforced. The problem is largely moot, at least for now, for executables, setuid programs, and device nodes, but that is not so for the read-only case. If the administrator of the host has mounted /proc as read-only, a process running in a user namespace could mount it read-write to evade that restriction, which is clearly a problem.

But Biederman was well aware that he might be breaking user-space applications by making this change. In particular, he was concerned about Sandstorm, LXC, and libvirt LXC, all of which employ user namespaces. So he put out the patches for testing (and comment).

That led to two reports of breakage, the first from Serge Hallyn about a problem he found using the patches with LXC. The LXC user space was not passing the three flags that restrict file types allowed for sysfs, which caused the mount() call to fail with EPERM due to Biederman's changes. The fix for LXC is straightforward but, as Andy Lutomirski pointed out, Biederman's change is an ABI break for the kernel. Given that there aren't executables or device nodes on sysfs or proc, dropping enforcement of those flags from the patch would not have any practical effect, Lutomirski argued.

Sandstorm lead Kenton Varda suggested that instead of returning EPERM, mount() should instead ignore the lack of those flags when the caller has no choice in the matter:

That is, in cases where mount() currently fails with EPERM when not given, say, MS_NOSUID, it should instead just pretend the caller actually set MS_NOSUID and go ahead with a nosuid mount. Or put another way, the absence of MS_NOSUID should not be interpreted as "remove the nosuid bit" but rather "don't set the nosuid bit if not required".

As Varda noted, that would fix the problem without LXC needing to change its code. He also thought it would be less confusing than getting an EPERM in that situation. Neither Biederman nor Lutomirski liked the implicit behavior that Varda suggested, however.

It turns out that libvirt LXC has a similar problem, as reported by Richard Weinberger. It is mounting /proc/sys, but not preserving the mount flags from /proc in the host, thus the mount() was failing. Once again, there is a simple fix.

Lutomirski suggested removing the noexec/nosuid/nodev part, but keeping the read-only enforcement, to avoid the ABI break. Biederman disagreed with that approach. It may not matter now that proc and sysfs are mounted that way, but it has mattered in the past and could again in the future:

So I am leaning towards enforcing all of the mount flags including nosuid, noexec, and nodev. Then when the next subtle bug in proc or sysfs with respect to chmod shows up I will be able to sleep soundly at night because the mount flags of those filesystems allow a mitigation, and I did not [sabotage] the mitigation.

Plus contemplating code that just enforces a couple of mount flags but not all of [them] feels wrong.

He did want to avoid breaking LXC and libvirt LXC, though, at least until those programs could be fixed and make their way out to users over the next few years. So Biederman added a patch that relaxed the requirement for noexec and nosuid (nodev turns out to be a non-issue due to other kernel changes), but printed a warning in the kernel log. Since it is a security fix (though not currently exploitable), he targeted the stable kernels with the fix too. However, Greg Kroah-Hartman pointed out that adding warnings for things that have been working just fine is not acceptable in stable kernels.

Though others disagree, Biederman does not see his changes as breaking the ABI. They do cause a behavior change and break two user-space programs (at least that are known so far), however. He would prefer not to break those programs, so the warning is kind of a stop-gap measure, he argued. The changes are fixing security holes, though, even if it appears they are not exploitable right now:

Given that I have not audited sysfs and proc closely in recent years I may actually be wrong. Those bugs may actually be exploitable. All it takes is chmod to be supported on one file that can be made executable. That bug has existed in the past and I don't doubt someone will overlook something and we will see the bug again in the future.

As it stands, the changes will still allow current LXC and libvirt LXC executables to function (though the version targeting the mainline will warn about that kind of use). Biederman plans to get it into linux-next, presumably targeting 4.2. After that, he plans to remove the warning and enforce the mount options in a subsequent kernel release. It is a bit hard to argue that either of the two broken programs were actually doing what their authors intended in the mount() calls, even though it worked. Assuming no other breakage appears, that might be enough to get this patch added without triggering Linus Torvalds's "no regression" filter.

Comments (none posted)

Obstacles to contribution in embedded Linux

By Jonathan Corbet
June 9, 2015

LinuxCon Japan
Tim Bird has worked with embedded Linux for many years; during this time he has noticed an unhappy pattern: many of the companies that use and modify open-source software are not involved with the communities that develop that software. That is, he said, "a shame." In an attempt to determine what is keeping companies from contributing to the kernel in particular, the Consumer Electronics Linux Forum (a Linux Foundation workgroup) has run a survey of embedded kernel developers. The resulting picture highlights some of the forces keeping these developers from engaging with the development community and offers some ideas for improving the situation.

The problem, Tim said, is not small. A typical system-on-chip (SoC) requires 1-2 million lines of out-of-tree code to function. Keeping that code separate obviously hurts the kernel, but it is also painful for the companies involved. There is a real cost to carrying that much out-of-tree code. Sony (where Tim works), for example, was managing 1,800 patches for each phone release — and that was just kernel code.

Fixing this problem requires engagement between embedded developers and the development community. "Engagement" means more just using the code and complying with its licensing; it means explaining requirements to the community, interacting in the community's forums, and merging changes upstream. A lot of companies still find that idea scary — they don't want to be producing code that will be used by their competitors. That is how our community works, though.

Obstacles

The idea behind the survey (conducted in September 2014) was to identify the obstacles to engagement with the community. So the group set out to locate developers who are not contributing code upstream and figure out why that isn't happening. The top reasons turned out to be:

Reason% agreed
Developing against an older kernel54%
Work depends on out-of-tree code50%
It's too hard45%
Unable to test41%
Employer does not allow time40%
Patch not good enough35%
Afraid of rejection33%

One thing that is not lacking is the desire to contribute: 92% of developers said that they think upstreaming code is important. But 21% of them said that management explicitly disapproves of such work, and 40% said that their management did not allow the necessary time. Issues that didn't matter included the need to use English (only 9% agreed) or developers feeling that upstreaming isn't their responsibility (6%). A bit more significant was the 26% of developers who said that their company's internal processes made contribution too hard.

At the top of the list of obstacles was "version gap" — developing code against an older kernel. Companies developing kernel code tend to start with whatever kernel was provided by the SoC vendor rather than the mainline; this is especially true for Android products. A typical Android kernel starts at Google, passes to the SoC vendor, and finally ends up at the equipment manufacturer; by that time, it is far behind the mainline.

As an example, Tim mentioned the Sony mobile kernel, which started as the 3.4 release when Google settled on it. By now, this kernel is 16 releases and three years behind the mainline. Some 26,000 commits, adding up to 1.8 million lines, have been added to Sony's kernel. The mainline kernel, of course, is in a state of constant change. Distance from the mainline makes [Tim Bird] contribution harder, since the kernel community needs patches against current kernels. As a result, it is unlikely that those 26,000 changes to a 16-release-old kernel can be easily upstreamed.

Another problem is the perceived difficulty of contributing to the kernel; to many, the process seems cumbersome and fraught with pitfalls. There are some documents on how to contribute, but those documents do not fully cover the community's social issues, timing constraints, procedures, and more. That said, Tim noted that some things are getting better; the availability of tools to find trivial issues with patches is helpful.

Kernel subsystem maintainers tend to be strict and terse with contributors, mostly as a result of overload; they simply don't have the time to fix up problematic patches. If a contributor gets a reputation for submitting bad patches and wasting maintainer time, their life will get worse. Silly mistakes can cause a patch to be quickly rejected or ignored. The problem here is that embedded developers are not full-time contributors, so they have a hard time maintaining a proficiency in dealing with the process. It is also hard for them to respond to requests for changes after their project has moved on.

Adding to the problem is the fact that much embedded code simply is not at the required level of quality for upstreaming. This code is often low-quality and highly specialized; it features the sort of workarounds and quick hacks that tend to get a poor reception on the kernel mailing lists.

Dependencies on other out-of-tree code also make things worse. Tim raised the example of the out-of-tree Synaptics touchscreen driver; Sony had developed a number of fixes and enhancements for this driver, but Synaptics had no interest in taking them. So where should these patches go? It is, he noted, not at all fun to be in the business of mainlining a supplier's code.

Developers complained that management does not give them the time they need to work with upstream communities. Product development teams work on tight schedules with the object of producing a "good enough to ship" solution. This is true throughout the industry; it is not unique to open-source software. These developers have no time to respond to change requests from maintainers. The kernel community, meanwhile, is not interested in any particular company's product timelines.

Things get worse, of course, when companies wait until late in the development process to release their code — something everybody does, according to Tim. When it comes time to mainline the code, the developers discover that major changes will be needed, which is discouraging at best. It would be far better to get input from the community or an internal expert early in the process.

Overcoming the obstacles

If a company wants to overcome version-gap problems, Tim said, the best place to start is to get current mainline kernels running on the hardware it is working with. One development team can then work on the mainline, while product engineers work with whatever ancient kernel is intended for the shipping product. The two-team approach can also help with the product-treadmill problem; if a small team is dedicated to mainlining the company's code, it can operate independently of the deadlines that drive the product engineers.

Companies should employ internal mentors and train developers to work with the wider community. Tim also stressed that it is important to use the community's processes internally. When working with kernel code, keep patches in Git; using a tool like Perforce to manage patches will not ease the task of engaging with the community.

With regard to low-quality code, Tim admitted that he had no silver bullet to offer. We all have to do hacks sometimes. The best that can be done is to examine the code to determine whether it should be maintained going forward; it should be kept in Git and reviewed (and possibly improved) at forward-porting time.

To avoid the highly specialized code problem entirely, don't use specialized hardware if at all possible. Manufacturers should, Tim said, require mainline drivers from their suppliers. The cost of software development should be figured into the bill of materials when the hardware is selected. Figuring in the cost of out-of-tree drivers is "an important next step" for the industry.

Unfortunately, companies following Tim's advice are likely to run into a new issue, which he called the "proxy problem." Having a special team dedicated to mainlining code can ease interactions with the community, but it also creates a situation where the community-facing developers are not the subject-matter experts. When they try to upstream code that they did not write, they cannot quickly answer questions or test changes. There is no avoiding the need for the original developers to help the proxies in situations like this.

Why bother?

Tim closed out the session by asking: why should companies bother with upstreaming their code in the first place? He pointed out that Sony has 1,100 developers who have made patches to its kernels in the last five years; many of them are applying the same patches over and over again. Sony would like to decrease the amount of time going into that sort of activity; mainlining its changes is the obvious way to do that.

Getting code upstream has a significant financial benefit for companies: it reduces the maintenance cost of that code and allows others to share the work. Even more importantly, having code upstream can reduce the time to market for future projects. Going through the community process improves the quality of the code. It can also fend off the need to migrate over to a competing company's implementation in the future. Finally, upstreaming is a reward for developers; it is something they want to do, and it will turn them into better developers.

These are all completely selfish reasons, Tim said; they are an entirely sufficient justification for working upstream even without getting into the ethical issues.

To further this effort, the "device mainlining project" is working within the CE Linux Forum. This project will continue to analyze the obstacles to contribution and attempt to address them by promoting best practices, collecting training materials, and publishing code analysis tools. There is also work ongoing to justify community engagement to management and quantify the costs of using out-of-tree code. This group will have its next meeting at the Embedded Linux Conference Europe in October.

[Your editor would like to thank the Linux Foundation for funding his travel to LinuxCon Japan]

Comments (70 posted)

Patches and updates

Kernel trees

Linus Torvalds Linux 4.1-rc7 ?
Greg KH Linux 4.0.5 ?
Greg KH Linux 3.14.44 ?
Greg KH Linux 3.10.80 ?

Architecture-specific

Core kernel code

Device drivers

Device driver infrastructure

Filesystems and block I/O

Memory management

Networking

Security-related

Virtualization and containers

Miscellaneous

Page editor: Jonathan Corbet

Distributions

OpenStack and stable point releases

By Jake Edge
June 10, 2015

OpenStack is a large project, with lots of moving pieces. Every six months, a new release of those pieces is made, with periodic stable point releases made thereafter. But those point releases are a lot of work—possibly for little gain—so there is a discussion about whether to continue to make those releases going forward. Obviously, security and other bug fixes will still need to be made, but there may be better ways to package them for consumption by distributions and other users.

OpenStack Technical Committee chair Thierry Carrez raised the issue on the openstack-dev mailing list at the end of May. In some discussions at the Vancouver OpenStack Summit, there were questions about continuing to make the point releases. There were a number of reasons behind that discussion, which Carrez outlined:

  • Some sub-projects follow their own version and release schemes, so the illusion of all of the OpenStack sub-projects having stable point releases at the same time is not really going to work.
  • Making the stable point releases is a fair amount of work.
  • Since the stable branches are intended to be always usable, and the point releases did not receive much additional testing, just grabbing any stable commit from those trees should be more or less the same.

He also recognized that not all of the stakeholders were present at the meeting in Vancouver, so his post was not meant as an announcement, but was, instead, a request for comment:

If you were a consumer of those and will miss them, please explain why. In particular, please let us know how consuming that version (which was only made available every n months) is significantly better than picking your preferred time and get all the current stable branch HEADs at that time.

While many in the thread were supportive of eliminating the point releases, there were also some concerns. Ihar Hrachyshka asked about release notes, and about "changes that require operator consideration or action" in particular. Several offered up solutions to format the Git commit messages into release notes, but Carrez pointed out that those kinds of changes are generally "a failure in the stable branch process -- this branch is supposedly safe to draw updates from". For security fixes that may require operator intervention, separate "security notes" would accompany the advisories and would detail the steps needed.

The proposal from Carrez made a distinction between libraries and other components (there are quite a few of those, as shown by the 2015.1.0 "Kilo" release notes). Libraries would still receive point releases "for critical bugs and security issues", but Dave Walker wanted clarification on what is considered a library in OpenStack, since "everything in python is a library". Matt Riedemann suggested that the libraries are the dependencies for the major server components such as Nova, Keystone, Cinder, and so on.

Haïkel Guémar, who is one of the packagers of OpenStack for Fedora, CentOS, and RHEL, would like to see the point releases continue, at least in some fashion. He is concerned that a lack of point releases will lead to fragmentation, with each distribution having its own set of versions, potentially each with its own set of backported security fixes. It will be difficult for operators (and others) to determine which fixes have been made to a given version, as well. He suggested switching to a time-based system that could be automated; it would periodically tag the repository, then generate and upload tarballs. Thomas Goirand, who packages OpenStack for Debian, largely concurred with Guémar.

At that point, Carrez summarized the objections raised in the discussion, which was, as he noted, quite constructive in tone. There were four main complaints that he listed: a lack of focused attention on the stable branch that comes from having a point release, the lack of proper release notes, no common reference points for distributions and deployments to be able to compare and discuss, and the loss of a sense of what pieces work well together.

Carrez also replied to his summary, addressing each of the points in turn. Only focusing attention on the stable branch at the time of point releases is actually a disadvantage, he said; it would be better to keep the branch in good shape at all times. He thinks the release notes problem is serious, but solvable. He is concerned about losing the common reference points for deployments to describe what they are running. Finally, the "works well together" piece is largely illusory anyway, since the stable branch rules should make all of the versions on those branches work together.

In his message, Carrez did come up with some alternate plans. Plan B would be similar to the way things are currently done, though it would essentially be a time-based tag of all of the projects that were (briefly) tested together on that date. Plan C would simply allow the projects themselves to tag stable point releases in an unsynchronized fashion. Alan Pevec added a Plan D that would turn each commit into its own release number (e.g. 2015.1.234 for the 234th commit on the stable branch). Each of the plans would address one or more of the objections. Carrez neatly summarized the plans and their downsides.

No consensus emerged from the subsequent discussion. There seem to be two schools of thought, however. There wasn't a huge difference between plans A and D in some people's minds, though Carrez said he now preferred D. Others saw either A or D as a big change for users and thought C would be a better incremental step. There seem to be no real proponents of plan B, at least so far.

Conceptually, at least, eliminating the point releases is a big step. The argument, though, seems to be that they don't add much for their cost. The discussion is still ongoing, however, and no changes will be made until after the Liberty (2015.2) release due in October.

Comments (none posted)

Brief items

Distribution quotes of the week

There are 2 hard problems in computing:
  • Caching problems
  • Naming things
  • Off-by-One Errors
-- Stephan Kulow

Booting from the live media brings up the Plasma 5 desktop environment. The background is decorated with deep blue wallpaper. The wallpaper looks slightly wrinkled and reminds me of the cave walls from classic Star Trek episodes.
-- Jesse Smith (DistroWatch review)

There are many bad things to be said about TempleOS, many aspects of it that seem poorly constructed or wouldn’t work in the “real world”. I’m going to ignore them here. It’s very easy to be negative, but you will never learn anything new by doing so.
-- Richard Mitton

Comments (4 posted)

CentOS releases

The CentOS Project has three new releases. First, there is a 1505 rolling build iso for CentOS 7. Next is a beta release of CentOS Linux 7 (1503) for i686 compatible hardware. There are also CentOS 7 x86_64 images for Vagrant.

Comments (none posted)

Debian 8.1 released

The Debian project has announced the first update of Debian 8 "jessie". "This update mainly adds corrections for security problems to the stable release, along with a few adjustments for serious problems. Security advisories were already published separately and are referenced where available."

Full Story (comments: none)

Distribution News

Fedora

FESCo, Council and Env and Stacks elections

Nominations are open for FESCo, Fedora Council, and Env and Stack elections. There are 4 seats open on FESCo, 1 seat on Council, and 5 seats on Env and Stacks. "If you have questions you'd like to ask candidates, please add them to the Elections Questionnaire wiki page. Nominees will answer these questions and the answers will be published simultaneously on June 22nd as part of campaign period. Questions may be moderated to fit Fedora Magazine interview format."

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

Comments (none posted)

Peppermint: Desktop Linux for the cloud generation (ZDNet)

ZDNet has a review of Peppermint. "You can, of course, use the usual Ubuntu software installations tools, such as the Ubuntu Software Manager, to install the standard Linux desktop apps. That would be missing Peppermint's point. Peppermint delivers a fast Linux desktop, based on Lightweight X11 Desktop Environment (LXDE), that's meant to run cloud apps well on minimal hardware with a good Internet connection."

Comments (none posted)

Page editor: Rebecca Sobol

Development

Removing the PostgreSQL contrib tree

By Jake Edge
June 10, 2015

The PostgreSQL contrib tree contains a number of useful tools and other features that are not part of the database core for various reasons. In some ways, the contrib tree can act something like the kernel's staging tree, providing a way to try out and improve features before they get "promoted" into the PostgreSQL core. But the tree is shipped in parallel with the core, adding to the maintenance burden for the core developers, so a reorganization of the contents of contrib has been proposed—with the idea of eliminating the contrib tree itself entirely.

Joshua D. Drake raised the idea of eliminating contrib in a post to the pgsql-hackers mailing list. In his post, he prefaced his proposal by noting the explanation from the documentation of why the contents of the contrib directory are not part of the core: "mainly because they address a limited audience or are too experimental to be part of the main source tree". That, however, "does not preclude their usefulness", it continues. In addition, Drake said, the contrib tree has long been considered to be the place to keep features that are eventually destined for the core.

The discussion of an auditing extension (pg_audit) that was committed to contrib by Stephen Frost in mid-May (which has since been reverted) was the proximate cause of Drake's proposal. The discussion started in the pgsql-committers mailing list but soon moved to pgsql-hackers. Some developers were not pleased with the code and documentation quality of pg_audit, were concerned about potential security holes it introduced, and were unhappy with how it got committed, which they thought circumvented the normal PostgreSQL community development process.

If pg_audit (and, by extension, other similar efforts) were not committed into the contrib tree in parallel with the core, some of these problems would go away, Drake argued. The code and documentation quality questions, along with any circumvention of the normal development process, would be non-issues if these new features were put into some other repository. That would also reduce the amount of code that the core developers need to maintain and test. He suggested going through the list of 45 modules to determine which should move into the core and which should move to either a new project (perhaps called "contrib") elsewhere or, if they are extensions, to the PostgreSQL Extension Network (PGXN). He suggested the criteria for where the contrib modules ended up come down to having a visible community (so, narrowly focused features would not make the cut) as well as having been included into contrib for at least two releases.

Frost agreed that some clarification of the mission of contrib is probably in order, but he noted that there is no location in the existing core tree where extensions could be placed. An "extensions" directory could be created, of course. He also wondered how the new contrib project would differ from PGXN. In addition, Peter Eisentraut concurred with the general idea, but suspected it would be a big undertaking.

But Drake didn't think the problem was all that large and claimed that it should be "obvious" to simply include many of the contrib modules into core. Another option might be to freeze contrib: "What is in there now, is all there will ever be in there and the goal is to slowly reduce it to the point that it doesn't matter."

Frost was not in favor of the freezing approach. But he did go through the list making recommendations on which modules belonged in the core. For his part, Drake largely agreed with Frost's suggestions. But others are not so sure. Fabian Coelho wondered if there was enough benefit to go through the exercise: "Reaching a consensus about what to move here or there will consume valuable time that could be spent on more important tasks... Is it worth it?" Jeff Janes was also concerned about how users would decide which modules and extensions to trust.

But Drake would be willing to see all of contrib move into the core and be installed by default as part of the standard installation. He simply doesn't think that the contrib distinction in the main tree is useful:

I care about this idea that contrib exists. It isn't needed and leads to a discussion like this one (or the pg_audit), almost every release.

Contrib made sense years ago. It does not any longer. Let's put the old horse down and raise a new herd of ponies on a new pasture.

On the other hand, though, Robert Haas thinks that it doesn't make sense to talk about getting rid of contrib when each new PostgreSQL release adds new modules to it. Those features are generally "pretty good stuff". There needs to be a place for that code: "We wouldn't have been better off rejecting it, and we wouldn't have been better off putting it into the main tree." Contrib has already been cleaned up along the way, he said, and simply renaming contrib is not particularly productive either. Even just categorizing the modules may not be as straightforward as it seems:

One thing that may be worth doing yet is separating the code that is just intended as a POC [proof of concept] (like worker_spi, auth_delay and test_decoding) from the stuff that you are really intended to run in production (like tcn and hstore). But that distinction is fuzzier than you might think, because while auth_delay was intended as a POC, I've subsequently heard rumors of it being used in production with satisfactory results. It's very easy to get the idea that you know "what PostgreSQL users use" but usually that tends to mean "what I use" and the community is broad enough that those things are Not The Same.

Haas sees contrib as a place for "things we want to include in the core distribution without baking them irrevocably into the server". Trying again, Drake restated his position, trying to clarify some of the points that Haas and others had disagreed with. But Andres Freund didn't agree with Drake's reasoning, nor did Haas.

One of the main reasons Drake cites for the change is the disagreement about pg_audit, which is, he said, the same argument that has come up frequently over the last fifteen years. But both Freund and Haas see that issue differently. Moving things into core will just move the argument from what goes into contrib to what goes into core, Freund said. Haas was more specific:

The argument about pg_audit has little to do with contrib. It is primarily about code quality, and secondarily about whether one committer can go do something [unilaterally] when a long list of other committers and contributors have expressed doubts about it.

As might be guessed, Frost did not agree with that characterization. By then, however, he had already reverted the change.

In another sub-thread, Jim Nasby elaborated on the concern raised by Janes: users (and distribution packagers) tend to trust the contrib tree because it comes with PostgreSQL. But that could all be made more explicit, Nasby said:

I think the real problem here that we're skirting around is this idea of 'blessed extensions', because that's really the only user benefit contrib brings: the idea that this stuff is formally blessed by the community. If that's really what we're after then we should just be explicit about that. Then we can decide if the best way to approach that is keeping it in the main repo (as opposed to say, publishing a list of [explicit] PGXN package versions and their checksums).

Personally, I'd rather we publish a list of formally vetted and approved versions of PGXN modules. There are many benefits to that, and the downside of not having that stuff as part of make check would be overcome by the explicit testing we would need to have for approved modules.

There was general agreement with Nasby's idea; a vetted list of extensions and other modules would be highly useful. In fact, Neil Tiffin said that he would never install anything from PGXN because it is hard to tell "what is actively maintained and tested, and what is an abandoned proof-of-concept or idea". Furthermore, it is not clear what version of PostgreSQL a PGXN module runs on or has been tested on, he said. That is a problem for open-source software in general, David E. Wheeler pointed out. Beyond that, the PGXN Tester does provide some of the information Tiffin is seeking.

Overall, it would seem that there aren't too many other core developers who see the problems the way that Drake does. The changes he suggested may not even address the main problem he sees. But there clearly are some changes that could be made—Nasby's suggestion chief among them. Whether a list of vetted and "blessed" extensions becomes a reality is unclear; no one seemed to volunteer for that particular mission. On the flipside, though, it is clear that not everyone is on the same page about the purpose of the contrib tree; that is probably worth clarifying one way or another.

Comments (none posted)

Brief items

Quotes of the week

Demos, the great God of demonstrations and examples. He preferred to teach other Gods, as opposed to humans. Usually he could be found skulking nearby or with API, the God of documentation.

Conf, the lesser known God of configurations and setups. He made sure that everything was well defined in a single place in a concise format, such as acceleration due to gravity, etc.

"hnyc" (Thanks to Paul Wise)

I guess that's why the GNU autoconf/configure system has always advised testing for particular wanted features, instead of looking at versions and then relying on carnal knowledge to know what those versions imply.
Neil Jerram, on reinventing build-system wheels (Thanks to James Bottomley)

Comments (none posted)

GNU Octave 4.0.0 Released

GNU Octave, which is a high-level programming language for numerical computations that is largely compatible with MATLAB, has made its 4.0 release. There are lots of new features in this major release, which are described in the release notes. Some of those features include defaulting to the graphical user interface instead of the command-line interface, OpenGL graphics and Qt widgets by default, a new syntax for object-oriented programming using classdef, audio functions, better MATLAB compatibility, and more.

Full Story (comments: 14)

Git v2.4.3 available

Git 2.4.3 has been released. Although primarily a bugfix release, this update does roll in several new minor features and a host of documentation updates. The fixes also include some user-visible changes, such as correcting the broken "verbosity" behavior found in git clone 2.4 and fixing the SSH transport-initiation code.

Full Story (comments: none)

Discourse 1.3 released

Version 1.3 of the Discourse web-commenting framework is now available. The new release supports desktop-notification integration, anonymous commenting, and a "civilized" mute function for ignoring specific commenters. Whether the latter two features are related is left up to the reader to decide.

Comments (none posted)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

As open source code, Apple's Swift language could take flight (ITWorld)

ITWorld reports that Apple will release its Swift programming language under an open source license. "When Swift becomes open source later this year, programmers will be able to compile Swift programs to run on Linux as well as on OS X and iOS, said Craig Federighi, Apple’s head of software engineering, during the opening keynote of Apple’s Worldwide Developers Conference Monday in San Francisco. The source code will include the Swift compiler and standard library, and community contributions will be “accepted—and encouraged,” Apple said."

Comments (28 posted)

Inside NGINX: How We Designed for Performance & Scale

The folks behind the NGINX web server have put up a highly self-congratulatory article on how the system was designed. "NGINX scales very well to support hundreds of thousands of connections per worker process. Each new connection creates another file descriptor and consumes a small amount of additional memory in the worker process. There is very little additional overhead per connection. NGINX processes can remain pinned to CPUs. Context switches are relatively infrequent and occur when there is no work to be done."

Comments (14 posted)

Page editor: Nathan Willis

Announcements

Articles of interest

FSFE Newsletter – June 2015

The June edition of the Free Software Foundation Europe newsletter covers International Day Against DRM, FSFE is looking for a systems architect, and several other topics.

Full Story (comments: none)

One year of encryption with the Email Self-Defense guide

The Free Software Foundation celebrates the first anniversary of the launch of "Email Self-Defense", an infographic and guide to encrypting your email with free software. "Even if you have nothing to hide, using encryption helps protect the privacy of people you communicate with and makes it harder for surveillance systems to isolate messages sent by those that do need protection, like whistleblowers and activists. By encrypting more of the traffic on the net, we give the surveillance systems more opaque information to sift through."

Full Story (comments: none)

Calls for Presentations

Ohio LinuxFest 2015 Call for Presentations

Ohio LinuxFest will take place October 2-3 in Columbus, Ohio. The call for papers deadline is July 17. "Our audience consists of people at all skill levels. Prior speaking experience is a plus, although we do try to provide opportunities for first-time speakers."

Full Story (comments: none)

2nd Call For Papers - 22nd Annual Tcl/Tk Conference (Tcl'2015)

Tcl'2015 will take place October 19-23 in Manassas, Virginia. Proposals are due by August 24. "The program committee is asking for papers and presentation proposals from anyone using or developing with Tcl/Tk (and extensions)."

Full Story (comments: none)

CFP Deadlines: June 11, 2015 to August 10, 2015

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

DeadlineEvent Dates EventLocation
June 11 June 25
June 28
Linux Vacation Eastern Europe 2015 Grodno, Belarus
June 15 August 15
August 22
DebConf15 Heidelberg, Germany
June 15 September 24 PostgreSQL Session 7 Paris, France
June 15 November 17
November 18
PGConf Silicon Valley San Francisco, CA, USA
June 17 October 5
October 7
LinuxCon Europe Dublin, Ireland
June 17 October 5
October 7
Embedded Linux Conference Europe Dublin, Ireland
June 19 August 20 Tracing Summit Seattle, WA, USA
June 28 August 28
September 3
ownCloud Contributor Conference Berlin, Germany
June 28 November 11
November 13
LDAP Conference 2015 Edinburgh, UK
June 30 November 16
November 19
Open Source Monitoring Conference 2015 Nuremberg, Germany
July 5 October 27
October 29
Open Source Developers' Conference Hobart, Tasmania
July 15 October 8
October 9
CloudStack Collaboration Conference Europe Dublin, Ireland
July 15 October 27
October 30
OpenStack Summit Tokyo, Japan
July 15 November 6
November 8
Jesień Linuksowa 2015 Hucisko, Poland
July 17 October 2
October 3
Ohio LinuxFest 2015 Columbus, OH, USA
July 19 September 25
September 27
PyTexas 2015 College Station, TX, USA
July 24 September 22
September 23
Lustre Administrator and Developer Workshop 2015 Paris, France
July 31 October 26
October 28
Kernel Summit Seoul, South Korea
July 31 October 24
October 25
PyCon Ireland 2015 Dublin, Ireland
July 31 November 3
November 5
EclipseCon Europe 2015 Ludwigsburg, Germany
August 2 February 1
February 5
linux.conf.au Geelong, Australia
August 2 October 21
October 22
Real Time Linux Workshop Graz, Austria
August 2 August 22 FOSSCON 2015 Philadelphia, PA, USA
August 7 October 27
October 30
PostgreSQL Conference Europe 2015 Vienna, Austria
August 9 October 8
October 9
GStreamer Conference 2015 Dublin, Ireland

If the CFP deadline for your event does not appear here, please tell us about it.

Upcoming Events

EuroPython 2015: Educational Summit

There will be an educational summit at EuroPython (July 20-26 in Bilbao, Spain). "We will have Educational Summit focused talks, trainings, birds of a feather sessions to debate and also Educational Sprints for the building of education focused projects during the weekend."

Full Story (comments: none)

Events: June 11, 2015 to August 10, 2015

The following event listing is taken from the LWN.net Calendar.

Date(s)EventLocation
June 10
June 13
BSDCan Ottawa, Canada
June 11
June 12
infoShare 2015 Gdańsk, Poland
June 12
June 14
Southeast Linux Fest Charlotte, NC, USA
June 16
June 20
PGCon Ottawa, Canada
June 22
June 23
DockerCon15 San Francisco, CA, USA
June 23
June 26
Open Source Bridge Portland, Oregon, USA
June 23
June 25
Solid Conference San Francisco, CA, USA
June 23
June 26
Red Hat Summit Boston, MA, USA
June 25
June 26
Swiss PostgreSQL Conference Rapperswil, Switzerland
June 25
June 28
Linux Vacation Eastern Europe 2015 Grodno, Belarus
June 26
June 28
FUDCon Pune 2015 Pune, India
June 26
June 27
Hong Kong Open Source Conference 2015 Hong Kong, Hong Kong
July 3
July 5
SteelCon Sheffield, UK
July 4
July 10
Rencontres Mondiales du Logiciel Libre Beauvais, France
July 6
July 12
SciPy 2015 Austin, TX, USA
July 7
July 10
Gophercon Denver, CO, USA
July 15
July 19
Wikimania Conference Mexico City, Mexico
July 18
July 19
NetSurf Developer Weekend Manchester, UK
July 20
July 24
O'Reilly Open Source Convention Portland, OR, USA
July 20
July 26
EuroPython 2015 Bilbao, Spain
July 25
July 31
Akademy 2015 A Coruña, Spain
July 27
July 31
OpenDaylight Summit Santa Clara, CA, USA
July 30
July 31
Tizen Developer Summit Bengaluru, India
July 31
August 4
PyCon Australia 2015 Brisbane, Australia
August 7
August 9
GUADEC Gothenburg, Sweden
August 7
August 9
GNU Tools Cauldron 2015 Prague, Czech Republic
August 8
August 14
DebCamp15 Heidelberg, Germany

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol


Copyright © 2015, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds