LWN.net Weekly Edition for May 21, 2015
What's new in SPDX 2.0
Version 2.0 of the Software Package Data Exchange (SPDX) specification was unveiled on May 13. SPDX is designed to facilitate license-compliance efforts for large projects (or projects that simply include a large number of upstream components), as well as those undertaken by component vendors and product manufacturers. The new revision adds some important flexibility to the format, enabling cross-references between packages and support for packages that are delivered via version-control systems.
The SPDX format is the product of the SPDX workgroup at the Linux Foundation (LF). The version 2.0 page includes the formal specification [PDF] as well as some background material created by the workgroup, such as the requirements document [PDF] that set out goals for the new revision.
Chief among those goals was adapting SPDX to work well as a metadata format that propagates easily through a "supply chain" from one vendor to another. In SPDX 1.x, a single file was used to capture the licensing information for every package. As such packages get incorporated into a larger combined work, however, the information from those files had to be copied into a single replacement file describing the derivative work, because the SPDX 1.x format could not reference external files. The most significant changes in version 2.0 are those that overcome this limitation.
SPDX uses RDF/XML to record several types of metadata about a software package. The focus is generally placed on license information, given the importance of license compliance in open-source software, but the SPDX format incorporates several other sections containing other types of metadata fields, such as a general-purpose package-information section (which includes the version number, the original source of the package, the provider of this particular copy of the program, and so on).
If it is not clear how this information would be of use to a development team, a presentation [PPT] from the 2015 Collaboration Summit includes a real world example for ActiveMQ. The official ActiveMQ packages released by the Apache Software Foundation bundle in Jetty (which itself is copyrighted by the Eclipse Foundation), while Jetty bundles in javax.servlet from Glassfish (which is copyrighted by Oracle). So the SPDX document for ActiveMQ denotes which of the many files in the package are under which license (in this case, Apache 2.0 for ActiveMQ, either Apache 2.0 or Eclipse 1.0 for Jetty, and either CDDL or GPL for javax.servlet) as well as the respective copyright holders. It also concludes that the combined ActiveMQ release is under the Apache 2.0 license. However, noting those per-component licenses is still important because some downstream developers might be interested in using only part of the whole.
Packages, references, and relationships
The biggest change in SPDX 2.0 is that all of this information no longer has to be fused together into a single, massive document. Assuming that the individual components (i.e., javax.servlet and Jetty) ship with their own SPDX 2.0 documents, the SPDX 2.0 document for ActiveMQ can reference the licensing, copyright, and other metadata information from those other SPDX files. This is done with two additions to the format: a globally unique SPDX identifier for each document and an internal identifier for each XML element within the document. SPDX documents can thus reference individual elements inside of other SPDX documents unambiguously. In supply-chain terms, this means that the ActiveMQ project could preserve the SPDX documents that come with Jetty and javax.servlet, then create a shorter SPDX document for its combined release by simply including references to those existing files.
Perhaps a more subtle result of this change is that it is no longer necessary to create a separate SPDX file for each individual software package that a company or vendor releases. Thanks to the ability to unambiguously reference individual elements within a SPDX document, each package can contain more than one top-level "package" element. Of course, since each SPDX 2.0 document can contain multiple packages or can reference packages in other documents, parsing a SPDX file is not as simple as it was in the 1.x days. But such is the price of flexibility.
The format also now allows users to explicitly designate the relationships between various files, packages, and other elements. The relationships supported include simple dependencies, plus "generates" relationships (to, for example, designate that a particular binary file is generated from a specific source file), designations that a file is a test case or data file, notes that a file has been removed or altered from its upstream version, and so on.
It is also worth noting that some of these relationships capture what one might call time-sequential information—like the addition or removal of a file. This is a departure from the SPDX 1.x era, where only the current hierarchical state of the package was represented. The ability to record such time-based information is an aspect of the supply-chain model that the SPDX workgroup hopes to support with 2.0.
The files referenced in a SPDX document can also now include a variety of data formats, such as audio, video, images, documentation files, and generic plain text. Documentation is, of course, often text itself, but the semantic meaning may be valuable to downstream users of a package. A number of new hash algorithms are also supported, so that checksums for different file types can be recorded in the SPDX document, too. The upshot of all of these changes is that SPDX 2.0 can more completely capture the semantic meaning of the files and packages provided by a particular project.
Licenses
SPDX documents contain a section named "Other Licensing Information Detected" for referencing licenses that are not on the SPDX official list. As the Collaboration Summit presentation expresses it, the SPDX license list aims to cover 90% of the world's FOSS code, which it says can be done with subset of the total license ecosystem—about 20 out of the 2000-plus licenses used in the wild. Nevertheless, the format somehow needs to account for that other 10%.
Off-list licenses are described with a set of XML attributes and RDF tag:value pairings. They include the license name, URL, and a text snippet (potentially the entire license). Despite this flexibility, in previous SPDX revisions there was no clearly defined way to express certain complex licensing situations. The most important is when a package is released under a choice of licenses (i.e., dual-licensed or tri-licensed). The 2.0 specification attempts to standardize how such licensing information is recorded by defining an augmented Backus-Naur Form (ABNF) syntax for expressing compound licensing relationships.
Like other elements in a SPDX 2.0 document, the off-list licenses described can be referenced elsewhere in the file by their internal ID, and cross-referenced by other SPDX documents.
And more
Among the other noteworthy changes to SPDX 2.0 is an extension of the Package Download Location field within the package-information section. Starting with this version, the field can contain a reference to a version-control system (VCS), whereas earlier versions expected an HTTP URL. The VCS references supported include Git, Mercurial, Subversion, and Bazaar, using HTTPS or SSH transport URLs. The syntax is taken from that used by the Python Package Index, and includes support for denoting specific branch, sub-path, commit hash, and tag names.
Finally, one feature from SPDX 1.x has been deprecated in SPDX 2.0. A new "Annotation" section replaces the "Review Information" section of earlier versions. This now-deprecated section was used to record human review information: the reviewer, review date, and an optional comment. All of that has been replaced by a more general-purpose annotation system. The format is essentially the same; an annotation has a "annotator" field, plus a date and room for comments. The new wrinkle is that Annotations can reference any SPDX element by its identifier. Thus, they can note per-element changes, rather than being attached only to the whole package (as was the case in SPDX 1.x).
What's next
It will be interesting to see how SPDX 2.0 catches on with users, specifically within corporate software environments (who tend to want far more detailed provenance about the components they use than do volunteer projects). There is a lot of additional flexibility in SPDX 2.0, but that flexibility does mean that SPDX users will have to restructure the metadata files for their packages. For large and complex software projects, the complexity of the new format compounds the amount of work required.
It is also interesting to note what did not make it into SPDX 2.0. In particular, the requirements document noted earlier mentions a desire to increase the granularity with which metadata can be recorded. Per-file licensing is too coarse for some projects, so there was interest in accommodating more finer-grained components (e.g., functions or classes). This does not appear to have made it into the 2.0 release. Nor did one other proposed requirement: a mechanism to verify the creator and reviewer information listed in the SPDX metadata for a package. Presumably, such a mechanism would resemble the one already used to provide a means for verifying file integrity—by, say, including GPG signatures—but the feature was tabled. The SPDX workgroup indicates that the standard will not be sitting still, however, so perhaps these and other features will soon be implemented in a new update.
New projects from day two of CoreOS Fest
While day one of CoreOS Fest 2015 introduced CoreOS architecture, plans, and specifications, day two introduced multiple open-source projects and tools. Presentations showed systemd-nspawn, Project Calico, Sysdig, and others. Most of these projects have been in development for a year or more, but the talks at the conference were the first look for most attendees.
While the talks themselves were interesting, the most remarkable thing was the sheer number of new tools that have been developed in the last year or so. Building up the software scaffolding for Linux containers seems to have happened faster than many other major changes introduced in Linux. One of the most fundamental pieces of this new infrastructure is systemd — the new init system for Linux — with its support for containers "out of the box".
Systemd and CoreOS
Lennart Poettering of Red Hat gave a presentation on systemd and CoreOS, describing the systemd tools that integrate with container management. Building containers using systemd really displays its benefits compared with other init systems, according to Poettering; systemd supplies all of the tools required to manage diverse containers on a single machine.
He noted that his talk was not an official Red Hat presentation, but then spent a fair amount of time speaking for the systemd team at Red Hat. The team isn't a product team, he explained, "we consider ourselves more of a research department than people who work on products."
This attitude explains some of the design decisions, such as choosing Btrfs as the primary filesystem for systemd-nspawn templates and containers. "Btrfs has a reputation for instability, but [the Btrfs project] is trying to solve fundamental filesystem issues," he said. Also, he explained that it is acceptable for containers to run on an unstable filesystem because "they're not where the data is." Important user data should be stored in external volumes, not in the container.
Systemd has multiple daemons that support containers, including systemd-machined, systemd-networkd, and systemd-resolved. In general, all of systemd is container-compatible because, according to Poettering, "systemd is tested on containers more often than on bare metal". Using containers allowed him to test init without rebooting his laptop frequently. He sees this deep integration with containers as a vital feature for Linux; "containers should be part of the OS itself, like Solaris Zones are."
It is also the goal of his team to be container-agnostic, supporting not just rkt, but also Docker, libvirt-lxc, OpenVZ, and others. The idea is that while systemd supplies a lot of container utility, it should be a low-level building block and not provide a sophisticated user interface. Projects like CoreOS and Kubernetes can then use systemd's functionality for basic operations.
Systemd-machined and its command-line tool, machinectl, are the most obvious piece of container management in systemd. With machinectl, users can list, start, stop, and even login to containers interactively. Systemd-machined is "really just a registry of containers" with which any container can register. Further, it can be used together with systemd to run any command inside a container using "systemd-run -M". Systemd-machined also allows running containers to appear in ps command listings and in GNOME's system monitor.
Systemd-nspawn is a lightweight container executor that provides a Docker-like tool that can start and run containers. It can be used to start a container using any filesystem or block device containing an MBR or GUID partition table. For users who want a limited-feature container manager that requires no configuration, systemd-nspawn will be an attractive option. Rkt uses systemd-nspawn under the hood to run container instances.
Systemd-networkd and systemd-resolved, the network and host-name-resolution daemons of systemd, also support containers. Systemd-networkd will automatically start a container's networking and do internal DHCP address assignment. Systemd-resolved provides host names for containers, using "link-local multicast name resolution", or LLMNR, an automatic name-discovery system invented by Microsoft. While LLMNR was designed for client applications and mobile devices, it can be used by containers to find each other on the network.
Based on Poettering's presentation, it seems like systemd will offer a strong alternative to Docker's libcontainer and other container initialization and management tools. Since the systemd tools will be built into most versions of Linux, they will eventually be widely available by default in many user environments. Perhaps that's why so many of the companies in the container business are focused on orchestration, which is one area where systemd doesn't concern itself.
Go and containers
One of the things that CoreOS, Inc. CEO Alex Polvi announced during his keynote was the company's sponsorship of the second Gophercon, the conference for Go language programmers. In fact, if you look at the list of sponsors for Gophercon, you'll see six of the major container-promoting companies listed there, which is around a quarter of the overall sponsors. This is not a coincidence; both CoreOS, Inc. and Docker, Inc. use Go almost exclusively. "Etcd could only have been built with Go," said Polvi.
In almost every talk and meeting room at the container conferences I've been to, people are talking about, and coding in, Go. Docker is written mostly in Go. Etcd, fleet, Swarm, Kubernetes, Kurma, and many other utilities and daemons for containers were built with it. The rise of Linux containers as a platform is likely to also be the rise of Go as a language.
Go started at Google in 2007 as an internal project with three developers, and today has over 500 contributors both inside and outside Google. The project is open source under the BSD license, but it is still run by Google staff and contributing requires signing a Contributor License Agreement (CLA) to Google. Increasingly, Go is used as an "automation language" for scalable server infrastructure; prior to Linux containers, it was popular for implementing network proxies, cloud server management tools, distributed search engines, and redundant data stores. So it's perhaps unsurprising that container utility programmers should also have chosen the language.
Because of CoreOS's close ties with Go, Brad Fitzpatrick gave a general session on Go's continuous build infrastructure. Fitzpatrick, known for LiveJournal, memcached, and OpenID, is now on the Google Go team. He presented at the conference on the automated build infrastructure that is used to test the language on many possible platforms. It started out as a Google App Engine application, plus a chain of mobile devices on Fitzpatrick's desk, and grew. His talk covered some of the history and mechanics of how it works.
Since Go is a compiled, rather than interpreted, language, it's critically important for users to know that binaries will execute on different platforms. Every check-in of Go gets built on hundreds of platform variations in a large machine lab at Google. Containers play a minor part in this because so many of the platforms to be tested don't support, or work with, containers. Linux variants are tested using Docker, but operating systems like Mac OS X and Android need special-purpose hardware to test them. You can see the current build test status and which builds are broken for various platforms on the Go Dashboard.
Project Calico
While Project Calico has been open source for almost a year, it was new to most of the audience when core developer Spike Curtis presented it. Calico is multi-host network routing software that includes a distributed, per-service firewall. It is designed for containers and virtual machines, especially Docker and OpenStack environments. The project is written in Python and developed by Metaswitch Networks, which is currently Calico's only commercial support vendor. Calico looks like a potential solution for users who want to deploy containers in production, but have stringent security requirements.
"Remember three-tier architectures?" complained Curtis. "That's still how admins secure networks. You have your external network, your DMZ with web resources, and your data layer, which needs to be the most secure."
"Microservices" running in containers on an orchestration network break down this three-tier model. First, microservices are defined by what service they provide, rather than their security characteristics. Second, orchestration frameworks expect an undifferentiated data center network and aren't designed with the concept of security tiers. Most of all, microservices require defining security policies and zones for literally hundreds of entities, instead of the few dozen network administrators expect. As he described, "it's a zoo and you've torn down the walls."
However, microservices offer a security opportunity as well. Because each one only does one thing, you can characterize its security requirements in simpler terms. This means that services can be compartmentalized in a more sophisticated way without added complexity, and that's what Project Calico is designed to do.
With each microservice or container mapped to a single IP address, Calico implements a simple iptables-based firewall running on each physical host for each of those IP addresses. Each service is defined by tags stored in etcd, and a JSON-formatted configuration file defines which other services are allowed to connect to it — or if it's available to the Internet.
Project Calico is designed to integrate with any orchestration framework that supplies an IP address for each service. Curtis demonstrated using Calico with Kubernetes, including using an extended Kubernetes pod definition to define security settings for each container. Apache Mesos is currently working on the IP-per-service feature, so it doesn't work with Calico yet.
Sysdig
The final "new" project described at CoreOS Fest was Sysdig. Like Project Calico, it was released about a year ago but most attendees saw it there for the first time. Also, like Project Calico, Sysdig is backed by a single company, Sysdig Cloud, which offers commercial support for the tool. Loris Degioanni, CEO of Sysdig Cloud, presented the tool at CoreOS Fest.
Sysdig is a traffic-monitoring system that is partially implemented as a Linux kernel module. The module captures all network traffic on the system, especially traffic between containers. The Sysdig tool supports writing filters in Lua (called "chisels") for this information, which allows users to aggregate it for statistical analysis. It can be thought of as a more advanced version of wireshark and tcpdump combined with container-awareness.
Degioanni said that Sysdig is an improvement on the Google cAdvisor project — frequently used with Docker containers — because cAdvisor only tells you about overall CPU, memory, and network usage of containers. Sysdig also gives you the ability to distinguish the endpoints and content of traffic. This means that you can, for example, filter for certain database queries, or troubleshoot unusual lag between two specific IP addresses.
One of the things Degioanni demonstrated was the soon-to-be-released open-source curses-based user interface for Sysdig, which is intended to allow system administrators to do interactive monitoring over SSH. He showed how to dig into traffic between containers and summarize it, as well as how to look into network delays. At the Sysdig Cloud booth, its staff showed off a much fancier, proprietary graphical user interface that supports clicking through to nested layers of servers, pods, and containers.
Day two wrap up
The new projects, tools, draft standards, and architectures I learned about at CoreOS Fest showed the rapid pace of development in the Linux container world. A year ago, when I reported on the first DockerCon, most of the techniques and tools covered at CoreOS Fest had just been launched or didn't even exist. Next year, we will see if development is still so high-velocity.
Of course, there's one major topic we haven't yet covered: the ongoing issue of storing persistent data in containers. As mentioned above, there is currently an expectation that containers are stateless and do not keep data. Removing that expectation raises a number of problems for container management and orchestration that are only beginning to be addressed, such as management of external volumes, container migration, and load-balancing of stateful services. Join us next week for coverage of multiple topics related to persistent data and containers from both CoreOS Fest and Container Camp.
Font editing with Glyphr Studio 1.0
Glyphr Studio is a free-software font-editing application that runs inside a browser window. Version 1.0 was released on May 7. Although it is not yet a full-featured application, it does provide a good interface for the core tasks of font development.
![[Editing in Glyphr Studio 1.0]](https://static.lwn.net/images/2015/05-glyphr-edit-sm.png)
The Glyphr Studio code is hosted at GitHub. The application is licensed under GPLv3, and is based on a number of open-source JavaScript libraries. At the moment, it can only be run from a web server; the latest release is accessible online at glyphrstudio.com/online. That means one must have an active Internet connection to work, of course. There is a feature request open to package the application for offline usage with Node.js, but it is not clear how far away such a change might be.
The project has been in development since 2010 (although it was a private, personal project in its early days), and has regularly made public beta releases since 2013. The most recent of those releases, Beta 5.2, arrived in January 2015. Subsequently, developer Matt LaGrandeur announced a roadmap in which the 1.0 milestone would designate that import/export functionality had landed, thus making the application useful for basic work. To implement that functionality, LaGrandeur chose to use the existing OpenType.js library. As it turns out, integrating OpenType.js proved to be such an easy transition that the 1.0 release also incorporated several other useful features.
Production
Glyphr Studio 1.0 offers a vector-based drawing canvas on which users can create the glyphs in a font, tools for constructing ligatures and setting up basic kerning, an interface for configuring basic font metadata, and a playground for testing out the in-development characters on some sample text. Users can also import generic SVG shapes, so they can design glyphs in another application first and bring them into Glyphr Studio for further refinement.
![[Settings in Glyphr Studio 1.0]](https://static.lwn.net/images/2015/05-glyphr-fontsettings-sm.png)
As far as the import/export functionality is concerned, at the moment it is basic. Users can import SVG fonts (as distinct from the generic SVG files that can be pulled in as drawing elements), OpenType .OTFs, and TrueType .TTFs. Export is a tad more limited. The program can export SVG fonts or OpenType .OTFs, but those .OTFs will not store any of the kerning or ligature tables. This is a limitation inherited from OpenType.js; when that project adds the requisite features, Glyphr Studio will pick them up. SVG fonts can be exported with both kerning and ligature tables.
Glyphr Studio can also save projects in its own, JSON-based format—files that will be saved to the user's local machine, not kept on the server, so there is little risk of data loss while one waits for OpenType.js to mature. In practice, the bigger limitation is that Glyphr Studio cannot import or export the Unified Font Object (UFO) file format used by many other tools, but there are feature requests open for that, too.
That said, there are a variety of font-engineering tools that can operate directly on OTF files, including scripts to insert kerning and ligature tables. These tables store relatively simple information (e.g., a kerning pair consists of two references to glyphs and an integer saying how much the space between those glyphs should be adjusted when they appear in order). It might not be trivial, but one could probably use Glyphr Studio in conjunction with some command-line utilities to develop and release a fully functioning text font.
Usage
The ability to technically produce a valid font is not what most people will be interested in, however. Designing and testing all of the glyphs that go into a usable finished product is a labor-intensive task that can be quite repetitive. Where such an interactive application often succeeds or fails is on the ease-of-use of its editing interface and how well its tool set stands up to the user's requirements and expectations.
The Glyphr Studio interface does a good job of being easy to navigate and unobtrusive. It can be run in a single browser window or the editing canvas can be split out into a window of its own to provide maximum editing space while still providing access to the on-screen settings and tools. Within the editing canvas, the tools and navigation buttons (e.g., zoom and pan) are straightforward enough. The navigation and tool panels (i.e., the parts of the interface outside of the editing canvas) are clear and well-organized, too, providing one-click access to each of the major tasks.
![[UI glitch in Glyphr Studio
1.0]](https://static.lwn.net/images/2015/05-glyphr-missingpoint.png "A disappearing control-point handle in Glyphr Studio")
As with all modern font editors, the emphasis is on creating and adjusting Bezier contours. In my tests—most of which were conducted with Glyphr Studio 1.00 running in Chromium—I encountered some oddities. The curve-adjustment tools occasionally threw a surprise at me, such as the handles for a control point disappearing when I selected a different point. I am also not quite convinced that I like the interface's habit of making options disappear entirely when they are not needed (rather than simply graying them out). This behavior makes the other options and buttons jump around on the screen, which seems like it would impede one's ability to commit the UI to muscle memory.
There are also isolated parts of the interface that are more difficult to navigate than others merely because they reuse icons and symbols from other places in the UI. Presumably that can be fixed. In addition, there are a few places where features seem to be only partially implemented—such as layer support, where the buttons to move an object up or down in the layer stack are present, but there is no way to add layers other than the default layer itself.
On the other hand, Glyphr Studio deserves high marks for laying out options and controls in a manner that relates directly to the workflow that a user will employ. For example, as noted earlier, the navigation panel includes buttons for each of the major tasks in designing a font (at least, those currently supported in Glyphr Studio): glyph editing, ligatures, kerning, and test-driving. That more or less reflects how most type designers seem to operate. Those four tasks are interrelated; users tend to jump back and forth between them, particularly when designing something from scratch.
![[Kerning in Glyphr Studio 1.0]](https://static.lwn.net/images/2015/05-glyphr-kerning-sm.png)
In contrast, FontForge—which, for many years, has been the dominant free-software font editor— also offers kerning and ligature construction, but the tools used to work on those tasks are accessible only from within a menu or buried several levels deep in modal dialog boxes. FontForge's functionality is more complete, but it can be quite a bit harder to use. On a daily basis, the inconveniences can wear some users out.
The one genuinely original feature that Glyphr Studio 1.0 offers is its "components" support. Components were a surprise addition to the 1.0 release when the OpenType.js integration went faster than expected. In essence, a component in Glyphr Studio is a reusable shape that the user can draw once and then reference in as many glyphs as necessary. The simplest example might be an accent mark, which readers expect to look the same on a variety of letters, but it can be anything: the "feet" on certain Cyrillic letters, the lower-case "n" reused as the left half of "m", and so on.
Most type designers seem to employ some form this technique already (though informally; perhaps by putting the reusable component in a slot assigned to no Unicode point), so it is interesting to see it elevated to the level of first-class feature. Whether it attracts and retains new users will be interesting to watch.
The last word
Two weeks ago, we reported on the Libre Graphics Meeting session in which Dave Crossland opined that Glyphr Studio and some other web-based font-development tools would soon surpass FontForge in functionality. Glyphr Studio 1.0 does not do that, although it does raise the bar in terms of usability, and it provides a serious challenge to FontForge going forward.
That is not to say Glyphr Studio has a clear road ahead of it; as the conventional wisdom says, implementing the initial, basic functionality is often easy for a development team. Filling in that "last 10%" is what adds unexpected complexity and ends up taking 90% of developers' time. In the meantime, though, Glyphr Studio is a worthy project for those in graphic-design fields to pay attention to. Regardless of what the future holds, another quality free-software option for font developers is a win for the community.
Security
Another crypto downgrade attack: Logjam
If more evidence was needed that deliberately weakening cryptographic algorithms based on politics is likely to go astray, the new Logjam attack against TLS seems to provide it. All of that evidence probably won't stop politicians (and others) from further attempts to cripple crypto, sadly, but it may at least provide some guidance to those who must implement those decisions down the road. In the meantime, though, there is yet another attack against the encryption used by the web and other internet protocols.
Downgrade attack
There are two related pieces to Logjam. The first is a crypto downgrade attack (somewhat akin to the FREAK attack) that, once again, abuses the export-strength ciphers that were mandated by the US government and are often still supported for backward compatibility. In this case, though, the attack is against Diffie-Hellman (DH) key exchange, rather than the RSA keys targeted by FREAK. Also unlike FREAK, the Logjam crypto downgrade attack uses a flaw in the TLS protocol instead of exploiting a bug in some TLS implementations.
The basic problem that Logjam exploits is that the message a server sends for key exchange can be undetectably replaced with a weaker variant. The message that uses a DH key that has been restricted to using prime numbers of at most 512 bits for political reasons (i.e. DHE_EXPORT ciphers) and one that uses the non-export variety can't be distinguished by clients. So a properly functioning server that supports, but does not prefer, DHE_EXPORT ciphers will send a key exchange message using a 1024-bit (or larger) DH group (i.e. one that has been derived from a 1024-bit prime). A man in the middle can intercept that message and rewrite it using the server's 512-bit group without alerting the client that anything has changed.
It turns out that the vast majority of servers that support DHE_EXPORT also use the same 512-bit prime number. The Logjam paper [PDF] shows that 82% of DHE_EXPORT-supporting web servers used a prime that had been distributed with the Apache web server for a number of years. It also reports that nearly 5% of the 14.3 million HTTPS web sites supported DHE_EXPORT, so roughly 4% of the net is using that one prime number. This practice was not seen as a problem, as the paper notes:
In order to exploit the downgrade to DHE_EXPORT, an attacker needs to be able to calculate discrete logarithms in near real-time. It is a nearly impossible math problem, but one that is made vastly easier with access to the prime number of interest. Using that, the man in the middle can precompute values that allow it to extract the session key from the key exchange. Once that is obtained, the man in the middle can pretend to be the server with impunity; the client will be none the wiser.
On a system with four six-core Xeon processors, the researchers' proof of concept was able to crack the session key in an average of 90 seconds. That may seem like a long time for users to wait for the browser to connect to the server—and it is, at least for interactive sessions. There are a number of techniques described in the paper to work around that delay.
To
start with, non-interactive uses, such as curl or Git, are often
unattended so they have long timeouts and "we could hijack their
connections without much difficulty
". There is a TLS warning
alert mechanism that can be used to reset the browsers' handshake timers.
The delay still might be noticed by the user, however, so the attacker
might choose to compromise a request for some background resource that
won't delay the page display.
There is also the fact that many TLS implementations cache their ephemeral
key and reuse it multiple times; that allows the calculation to be done once
and to be used again and again.
Beyond that, more optimization or more powerful systems could potentially
reduce the average time to crack the key substantially.
Diffie-Hellman and state-level attackers
The second piece of the Logjam report looks at how vulnerable the DH key
exchange is to attack when there is no downgrade ability. Once again,
various protocols
often use (or even specify) the same prime numbers everywhere. The paper
specifically mentions SSH, TLS, and Internet Key
Exchange (IKE) which is used by IPsec.
Currently, 768-bit DH groups are
fairly widespread and the paper estimates that precomputing the values
needed would require around 36,000 core-years, which is "within
reach by computing power available to academics
". Once that
precomputation is done, doing a single 768-bit discrete log computation is
on the order of two core-days.
Perhaps even more eye-opening, though, is that 1024-bit DH groups are estimated to be crackable by state-level attackers (i.e. governments and their ilk). Precomputation is on the order of 45 million core-years, which is an enormous effort, but could be reduced with specialized hardware. Once that was done, the discrete log calculations could be done in roughly 30 core-days. That estimate leads to a possible answer to a question that has been asked frequently in the crypto community:
Documents leaked by Snowden indicate that the NSA has been passively decrypting virtual private network (VPN) traffic. The paper concludes that the documented information is consistent with an attack against the 1024-bit DH group used by IKE. That doesn't mean it is the mechanism used by the NSA, but it could be.
That leads the authors of the paper (David Adrian, Karthikeyan Bhargavan, Zakir Durumeric, Pierrick Gaudry, Matthew Green, J. Alex Halderman, Nadia Heninger, Drew Springall, Emmanuel Thomé, Luke Valenta, Benjamin VanderSloot, Eric Wustrow, Santiago Zanella-Béguelin, and Paul Zimmermann) to a number of recommendations. Disabling DHE_EXPORT ciphers, and configuring the other DHE ciphers to use groups of 2048 bits or larger, head that list. Browsers and other clients should require groups of at least 1024 bits to avoid the downgrade attack but still interoperate with servers that have not been upgraded.
Avoiding fixed-prime groups is another recommendation, as is transitioning to elliptic curve DH (ECDH), which does not provide as much of an advantage for precomputation. The parameters for ECDH still need to be worked out—those recommended by the US government are viewed with suspicion due to NSA influence on their choice.
The last two recommendations are admonishments to politicians and the
crypto community respectively. First: "In combination with FREAK,
our attacks warn of the long-term
debilitating effects of deliberately weakening cryptography.
" But
there is also lots of room for better communication by the crypto
community:
For those wanting to check their status, the Logjam web site will warn if their browser is susceptible to the attack (i.e. if it will accept DH groups with less than 1024 bits). Another page provides tools to test servers and information on how to upgrade those that are found to have an inadequate DH group size (i.e. less than 2048). For example, the LWN.net server needs an upgrade to its DH group, which we will be taking care of soon.
This research is quite timely, interesting, and important, though it may not require a "drop everything" response—at least for sites that don't support DHE_EXPORT. For those that believe their traffic may be targeted by state-level attackers, however, prudence would seem to indicate a faster response. At this point, after the information leaked by Snowden and others, these kinds of attacks seem a little less surprising—we have known that secret services were decrypting lots of this data for some time. Now, at least, we may know some of the "how" as well.
Brief items
Security quotes of the week
Hardening Hypervisors Against VENOM-Style Attacks (Xen Project Blog)
The Xen Project looks at a mechanism to mitigate vulnerabilities like VENOM that attack emulation layers in QEMU. "The good news is it’s easy to mitigate all present and future QEMU bugs, which the recent Xen Security Advisory emphasized as well. Stubdomains can nip the whole class of vulnerabilities exposed by QEMU in the bud by moving QEMU into a de-privileged domain of its own. Instead of having QEMU run as root in dom0, a stubdomain has access only to the VM it is providing emulation for. Thus, an escape through QEMU will only land an attacker in a stubdomain, without access to critical resources. Furthermore, QEMU in a stubdomain runs on MiniOS, so an attacker would only have a very limited environment to run code in (as in return-to-libc/ROP-style), having exactly the same level of privilege as in the domain where the attack started. Nothing is to be gained for a lot of work, effectively making the system as secure as it would be if only PV drivers were used." The Red Hat Security Blog also noted this kind of mitigation for VENOM-style attacks.
New vulnerabilities
avidemux: denial of service
| Package(s): | avidemux | CVE #(s): | CVE-2015-1872 | ||||||||||||||||
| Created: | May 19, 2015 | Updated: | May 20, 2015 | ||||||||||||||||
| Description: | From the Mageia advisory:
An attacker can force a read at an invalid address in mjpegdec.c of FFmpeg, in order to trigger a denial of service. | ||||||||||||||||||
| Alerts: |
| ||||||||||||||||||
avidemux: multiple vulnerabilities
| Package(s): | avidemux | CVE #(s): | CVE-2014-9316 CVE-2014-9317 CVE-2014-9603 CVE-2015-3417 | ||||||||||||||||
| Created: | May 19, 2015 | Updated: | June 15, 2015 | ||||||||||||||||
| Description: | From the CVE entries:
The mjpeg_decode_app function in libavcodec/mjpegdec.c in FFMpeg before 2.1.6, 2.2.x through 2.3.x, and 2.4.x before 2.4.4 allows remote attackers to cause a denial of service (out-of-bounds heap access) and possibly have other unspecified impact via vectors related to LJIF tags in an MJPEG file. (CVE-2014-9316) The decode_ihdr_chunk function in libavcodec/pngdec.c in FFMpeg before 2.1.6, 2.2.x through 2.3.x, and 2.4.x before 2.4.4 allows remote attackers to cause a denial of service (out-of-bounds heap access) and possibly have other unspecified impact via an IDAT before an IHDR in a PNG file. (CVE-2014-9317) The vmd_decode function in libavcodec/vmdvideo.c in FFmpeg before 2.5.2 does not validate the relationship between a certain length value and the frame width, which allows remote attackers to cause a denial of service (out-of-bounds array access) or possibly have unspecified other impact via crafted Sierra VMD video data. (CVE-2014-9603) Use-after-free vulnerability in the ff_h264_free_tables function in libavcodec/h264.c in FFmpeg before 2.3.6 allows remote attackers to cause a denial of service or possibly have unspecified other impact via crafted H.264 data in an MP4 file, as demonstrated by an HTML VIDEO element that references H.264 data. (CVE-2015-3417)
| ||||||||||||||||||
| Alerts: |
| ||||||||||||||||||
dcraw: denial of service
| Package(s): | dcraw | CVE #(s): | CVE-2015-3885 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | May 14, 2015 | Updated: | January 23, 2017 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Mageia advisory:
The dcraw tool bundled in darktable's libraw copy suffers from an integer overflow condition which leads to a buffer overflow. A maliciously crafted raw image file can be used to trigger the vulnerability, causing a Denial of Service condition. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
gnutls: signature algorithm verification botch
| Package(s): | gnutls | CVE #(s): | |||||
| Created: | May 14, 2015 | Updated: | May 20, 2015 | ||||
| Description: | From the Red Hat bugzilla entry:
It was reported by the GnuTLS project that a ServerKeyExchange signature sent by the server is not verified to be in the acceptable by the client set of algorithms. That has the effect of allowing MD5 signatures (which are disabled by default) in the ServerKeyExchange message. It is not believed that this bug can be exploited because a fraudulent signature has to be generated in real-time which is not known to be possible. However, since attacks can only get better it is recommended to update to a GnuTLS version which addresses the issue. | ||||||
| Alerts: |
| ||||||
kernel: privilege escalation
| Package(s): | kernel | CVE #(s): | CVE-2014-9710 | ||||||||||||||||||||||||
| Created: | May 20, 2015 | Updated: | May 20, 2015 | ||||||||||||||||||||||||
| Description: | From the Ubuntu advisory:
Alexandre Oliva reported a race condition flaw in the btrfs file system's handling of extended attributes (xattrs). A local attacker could exploit this flaw to bypass ACLs and potentially escalate privileges. | ||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||
moodle: multiple vulnerabilities
| Package(s): | moodle | CVE #(s): | CVE-2015-3174 CVE-2015-3175 CVE-2015-3176 CVE-2015-3178 CVE-2015-3179 CVE-2015-3180 CVE-2015-3181 | ||||||||||||
| Created: | May 19, 2015 | Updated: | May 20, 2015 | ||||||||||||
| Description: | From the Mageia advisory:
In Moodle before 2.6.11, leaving gradebook feedback is a trusted action and such capabilities in other modules already have an XSS mask, 'mod/quiz:grade' was missing this flag (CVE-2015-3174). In Moodle before 2.6.11, some error messages display a button to return to the previous page. Redirecting to non-local referer should not be allowed as it can potentially be used for phising (CVE-2015-3175). In Moodle before 2.6.11, on sites with enabled self-registration, not registered users can retrieve fullname of registered users if they know their usernames (CVE-2015-3176). In Moodle before 2.6.11, if a user who is not XSS-trusted attempts to insert a script as part of the input text, it will be cleaned when displayed on the Moodle website but may be displayed uncleaned in the external application because external_format_text() cleans and formats text incorrectly when returning it from Web Services (CVE-2015-3178). In Moodle before 2.6.11, when self-registration is enabled and a user's account was suspended after creating the account but before actually confirming it, the user is still able to login when confirming their email, but only once (CVE-2015-3179). In Moodle before 2.6.11, if a user is enrolled in the course but his enrollment is suspended, they can not access the course but still were able to see the course structure in the navigation block (CVE-2015-3180). In Moodle before 2.6.11, users with the revoked capability 'moodle/user:manageownfiles' are still able to upload private files using a deprecated function in Web Services (CVE-2015-3181). | ||||||||||||||
| Alerts: |
| ||||||||||||||
nbd: denial of service
| Package(s): | nbd | CVE #(s): | CVE-2015-0847 | ||||||||||||||||||||||||||||
| Created: | May 18, 2015 | Updated: | August 14, 2015 | ||||||||||||||||||||||||||||
| Description: | From the Debian LTS advisory:
Tuomas Räsänen discovered that unsafe signal handling is present in nbd-server. This vulnerability could be exploited by a remote client to cause a denial of service. | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
NetworkManager: denial of service
| Package(s): | NetworkManager | CVE #(s): | CVE-2015-2924 | ||||||||||||||||||||
| Created: | May 18, 2015 | Updated: | November 26, 2015 | ||||||||||||||||||||
| Description: | From the Red Hat bugzilla:
NetworkManager: denial of service (DoS) attack against IPv6 network stacks due to improper handling of Router Advertisements. | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
php: multiple vulnerabilities
| Package(s): | php | CVE #(s): | CVE-2015-4021 CVE-2015-4022 CVE-2015-4024 CVE-2015-4025 CVE-2015-4026 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | May 19, 2015 | Updated: | June 25, 2015 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From Mageia's updated advisory:
Memory Corruption in phar_parse_tarfile when entry filename starts with null (CVE-2015-4021). Integer overflow in ftp_genlist() resulting in heap overflow, potentially exploitable by a hostile FTP server (CVE-2015-4022). PHP Multipart/form-data parsing remote DoS Vulnerability (CVE-2015-4024). Various functions allow \0 in paths where they shouldn't. In theory, that could lead to security failure for path-based access controls if the user injects a string with \0 in it. These functions include set_include_path(), tempnam(), rmdir(), and readlink() (CVE-2015-4025), as well as pcntl_exec() (CVE-2015-4026). PHP 5.5.25 and 5.6.9 have been released fixing multiple bugs and potential security issues. See the PHP 5.5.25 changelog and the PHP 5.6.9 changelog for details. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
php-ZendFramework2: CRLF injection
| Package(s): | php-ZendFramework2 | CVE #(s): | CVE-2015-3154 | ||||||||||||||||||||||||||||||||||||||||||||
| Created: | May 20, 2015 | Updated: | June 9, 2015 | ||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Zend advisory:
CRLF (Carriage Return "\r" and Line Feed "\n") is a significant sequence of characters, representing the the End Of Line (EOL) marker for many Internet protocols, including, but not limited to MIME (e-mail), NNTP (newsgroups), and, more importantly, HTTP. When programmers write code for web applications, they split headers based on where the CRLF is found. If a malicious user is able to inject his own CRLF sequence into an HTTP stream, he is able to maliciously control the way a web application functions. | ||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||
phpMyAdmin: two vulnerabilities
| Package(s): | phpMyAdmin | CVE #(s): | CVE-2015-3902 CVE-2015-3903 | ||||||||||||||||||||||||||||
| Created: | May 18, 2015 | Updated: | May 26, 2015 | ||||||||||||||||||||||||||||
| Description: | From the Red Hat bugzilla:
CVE-2015-3902: By deceiving a user to click on a crafted URL, it is possible to alter the configuration file being generated with phpMyAdmin setup. Versions 4.0.x (prior to 4.0.10.10), 4.2.x (prior to 4.2.13.3), 4.3.x (prior to 4.3.13.1) and 4.4.x (prior to 4.4.6.1) are affected. CVE-2015-3903: A vulnerability in the API call to GitHub can be exploited to perform a man-in-the-middle attack. Versions 4.0.x (prior to 4.0.10.10), 4.2.x (prior to 4.2.13.3), 4.3.x (prior to 4.3.13.1) and 4.4.x (prior to 4.4.6.1) are affected. | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
qemu: denial of service
| Package(s): | qemu | CVE #(s): | CVE-2014-9718 | ||||||||||||||||||||||||||||||||||||||||||||
| Created: | May 14, 2015 | Updated: | May 20, 2015 | ||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Debian advisory:
CVE-2014-9718: It was discovered that the IDE controller emulation is susceptible to denial of service. | ||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||
ruby-rest-client: session fixation attack
| Package(s): | ruby-rest-client | CVE #(s): | CVE-2015-1820 | ||||
| Created: | May 18, 2015 | Updated: | May 20, 2015 | ||||
| Description: | From the Mageia advisory:
When Ruby rest-client processes an HTTP redirection response, it blindly passes along the values from any Set-Cookie headers to the redirection target, regardless of domain, path, or expiration. This can be used in a session fixation attack or in stealing cookies. | ||||||
| Alerts: |
| ||||||
wireshark: multiple vulnerabilities
| Package(s): | wireshark-cli | CVE #(s): | CVE-2015-3808 CVE-2015-3809 CVE-2015-3810 CVE-2015-3813 CVE-2015-3815 | ||||||||||||||||||||||||||||||||||||
| Created: | May 15, 2015 | Updated: | May 20, 2015 | ||||||||||||||||||||||||||||||||||||
| Description: | From the Arch Linux advisory: CVE-2015-3808 - There is an infinite loop condition in dissect_lbmr_pser() in epan/dissectors/packet-lbmr.c. It's possible for an attacker to set the the variable 'option_len' to 0, causing the loop to never terminate. This issue is leading to excessive CPU resources consumption by injecting a malformed packet onto the wire or by convincing someone to read a malformed packet trace file. CVE-2015-3809 - There is an infinite loop condition in dissect_lbmr_pser() in epan/dissectors/packet-lbmr.c. It's possible for an attacker to set the the variable 'option_len' to 0, causing the loop to never terminate. This issue is leading to excessive CPU resources consumption by injecting a malformed packet onto the wire or by convincing someone to read a malformed packet trace file. CVE-2015-3810 - The Websocket dissector recurses to dissect more data. Since the minimum valid websocket frame length is two bytes, this allows for quite some recursion. With a minimal IP and TCP header (20 bytes each), you can get 32747 recursions which is leading to excessive CPU resources consumption by injecting a malformed packet onto the wire or by convincing someone to read a malformed packet trace file. CVE-2015-3813 - It has been discovered that the packet reassembly code could leak memory under certain circumstances. This may lead to crash by excessive memory consumption via injecting malformed packets onto the wire or by convincing someone to read a malformed packet trace file. CVE-2015-3815 - It has been discovered that the Android Logcat file parser could crash under certain circumstances. This issue is leading to denial of service by injecting a malformed packet onto the wire or by convincing someone to read a malformed packet trace file. | ||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||
wireshark: three dissector vulnerabilities
| Package(s): | wireshark | CVE #(s): | CVE-2015-3811 CVE-2015-3812 CVE-2015-3814 | ||||||||||||||||||||||||||||||||||||||||||||
| Created: | May 14, 2015 | Updated: | June 11, 2015 | ||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Mageia advisory:
The WCP dissector could crash while decompressing data (CVE-2015-3811). The X11 dissector could leak memory (CVE-2015-3812). The IEEE 802.11 dissector could go into an infinite loop (CVE-2015-3814). | ||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||
Page editor: Jake Edge
Kernel development
Brief items
Kernel release status
The current development kernel is 4.1-rc4, released on May 18. "So here it is, last-minute fix and all. The -rc4 patch is a bit bigger than the previous ones, but that seems to be mainly due to normal random timing - just the fluctuation of when submaintainer trees get pushed."
Stable updates: 4.0.4, 3.14.43, and 3.10.79 were released on May 17.
Quote of the week
Kernel development news
Delay-gradient congestion control
Network congestion-control algorithms have a difficult task to perform. They must moderate each endpoint's outgoing traffic to keep the Internet from being overwhelmed by packet congestion (as happened in 1986 before these algorithms were introduced). But, at the same time, the algorithm is expected to allow a machine to make full use of the bandwidth available to it, sharing that bandwidth with other systems without any sort of central control mechanism. Err on one side, and the network as a whole suffers; err on the other, and performance will suffer. So it is not surprising that, long after workable solutions to the congestion-control problem exist, research continues in this area. One relatively new algorithm is called "CAIA delay gradient" (or CDG); it is named after the Centre for Advanced Internet Architectures where it was first developed for FreeBSD. A patch adding CDG to the kernel was recently posted for review.Most congestion-control algorithms are based on the use of packet loss as a signal that indicates congestion somewhere along the path used by the connection. When all goes well, a connection should not experience packet loss. If a router's buffers fill, though, that router will start to drop packets; a congestion-control algorithm will respond to that packet loss by reducing the number of packets that can be outstanding (the "congestion window") at any given time. Typically, the congestion window will then be allowed to creep back up until the next packet loss occurs, indicating that the limit has once again been reached.
Loss-based congestion control has served the net well for the better part of thirty years, but it is not without its drawbacks. By its nature, it will cause packets to be dropped occasionally; that will necessarily introduce latency into the connection which, perhaps, can ill afford it. Loss-based algorithms can only find the limit for a given connection by pushing the slowest link to its limit, meaning that it forces a router buffer somewhere to overflow; this behavior can also worsen bufferbloat-related problems. There are also problems when packets are lost for reasons other than congestion, as can happen with wireless links, for example. The congestion-control code will interpret that loss as a congestion signal, slowing transmission unnecessarily.
The alternative, as implemented by CDG, is to try to infer the state of a connection by looking at the variation in the round-trip time (RTT) — the time it takes for a packet to transit the connection in both directions. Using RTT to estimate congestion is easy if one knows the actual characteristics of the connection in use; one need only look at how much "extra" time is currently needed to get a packet through the link. That is why, for example, smartphone driving-time estimates for a well-known commute can be a useful indication of road congestion. But that knowledge is not available on the Internet as a whole, so some other approach will be required.
The CDG approach is to look at the minimum and maximum RTTs observed for a given connection over a period of time. The minimum is called Τmin, while the maximum is Τmax. From subsequent observations of Τmin and Τmax, the algorithm calculates the rate of change of each. The rate at which the minimum RTT is changing is δmin, while the rate at which the maximum RTT is changing is δmax. These two parameters are then used in a couple of interesting ways.
The key use, of course, is to try to come up with an estimate of how congested the link is. To simplify a bit: if Τmin is growing (δmin is positive), chances are that the link is getting more congested. Every RTT interval, CDG calculates a "probability of backoff" based on δmin; as δmin grows, that probability approaches one. That probability is then compared against a random number to determine whether the congestion window should be decreased; should a decrease be decided upon, the number of packets in the congestion window will be reduced by a configurable factor (0.3 by default).
In cycles where the algorithm decides not to decrease the congestion window, it will, instead, increase it by one packet. That allows the window to creep upward and continually test the limits of the connection. In theory, the delay-gradient should detect that limit without pushing the connection to the point of packet loss.
There is an interesting question that comes up at about this point, though: imagine a situation where a system using CDG is sharing a slow link with another system using traditional loss-based congestion control. As RTT increases, the CDG system will back off, but the loss-based system will continue to pump out packets until the router starts dropping things. Indeed, it may increase its transmission rate to soak up the bandwidth that the CDG system is no longer using. If CDG allows itself to be muscled out of a contended link in that manner, one can predict with high confidence that it will not find many users.
To deal with this problem, the CDG authors developed a heuristic to detect situations where competition with a loss-based system is occurring. If CDG is working properly, a decision to slow down the transmission rate should result in the RTT values getting smaller — δmin and δmax should go negative. If that fails to happen after a few backoff operations, the algorithm will conclude that somebody else is playing by different rules and stop backing off for a while. CDG also remembers the previous maximum value of the congestion window (as the "shadow window"); this value can be used to quickly restore the congestion window in the event of a packet drop.
CDG's handling of packet drops is interesting. The motivations for using delay gradients are to avoid depending on packet loss as a signal and to avoid slowing transmission in response to packet losses that do not result from congestion. But congestion-related packet loss will still happen when CDG is in use, and the algorithm should respond accordingly. Backing off in response to packet loss is easy; the tricky part is determining whether that loss is a congestion signal or not.
As a general rule, congestion manifests itself as an overflow of the packet queue for the slowest link used by a connection. If that queue is known to be full, then a packet loss is likely to be a result of the congestion there; if, instead, it is known to be relatively empty, congestion is probably not the problem. The heuristic used to determine the state of the queue is this: when a queue fills, Τmax will reach its largest possible value and stop increasing (because the queue cannot get any longer), but Τmin will continue to increase. CDG will only treat a packet loss as a congestion signal when a full queue has been detected in this manner.
At least, that is how the algorithm was designed; the current Linux patch
does not quite work that way. In the patch posting, Kenneth Klette
Jonassen notes that: "We decided to disable the loss
tolerance heuristic by default due to concerns about its safety outside
closed environments.
" So that aspect of CDG will have to wait until
the algorithm's behavior on the full Internet is better understood.
Even so, networking developer Eric Dumazet said that the new congestion-control module
"looks awesome
". Its true level of awesomeness can only be
determined via years of real-world experience. But getting CDG into the
Linux kernel is a good first step toward the acquisition of that
experience. Should things go well, loss-based congestion control might
end up backing off in its role on the net.
(See this paper [PDF] for the full details on how the CDG algorithm works.)
Kernel support for SYN packet fingerprinting
The initial packet of a TCP connection (i.e. the SYN packet) contains information that can be used to detect attributes of the remote system through TCP/IP fingerprinting. But that data is contained in the headers of the packets, which means it is only accessible to the kernel. A patch set that was recently merged into the net-next tree would change that to allow user-space servers to request the header information on connections they have accepted.
Eric Munson started the conversation when he posted a patch that would allow a program to request that the SYN packets be saved by using setsockopt() on a listening socket. The SYN headers could then be retrieved, once, via a getsockopt() call on the socket returned by accept(). That would allow user space to examine the TCP and IP headers to identify (or at least narrow down) the operating system of the remote host that made the connection.
Munson's patch simply stored the SKB (i.e. struct sk_buff) that contained the SYN packet, which could be rather large (up to 4KB), as Eric Dumazet pointed out. For millions of client connections, that memory can add up, he said.
Instead, Dumazet suggested, a 2012 patch from Tom Herbert (or one based on that) should be used. That code has been used internally at Google for around two years, he said, without any problems handling large numbers of simultaneous connections. Instead of storing the SKB, it allocates space just for the headers with kmalloc()—usually less than 128 bytes per connection.
When Herbert posted his patch, there were concerns about adding eight
bytes to each SKB for a "very fringe feature
" (in the
words
of network maintainer David Miller). Herbert's original patch also stored
the SKB like Munson's does. The patch was never merged, but
Dumazet modified it to kmalloc() space for the headers and it was
put into production at Google.
Munson was not particularly tied to his implementation; he said that he was happy to back Dumazet's patch if it met his needs. That patch was posted on May 3. It adds two new socket options that are used to request and retrieve the SYN headers. Servers request that the kernel save the headers by calling setsockopt() with TCP_SAVE_SYN either before or after the listen() call; the kernel will save the headers for subsequent connection requests. IP and TCP headers can be retrieved, once, by calling getsockopt() with TCP_SAVED_SYN on the socket returned from accept().
Michael Kerrisk complained that the option names were too similar, while also asking about how the interface would be used. Dumazet disagreed about the names, but provided a test program used by Google to demonstrate how the new options work for user space.
Andy Lutomirski wondered if too much
information was being returned to user space with Munson's patch. It turned
out that Ethernet headers were also being returned, which Munson agreed was probably not needed. John Heffner
asked a related question: "Are there conditions where, for security purposes, you don't want an
application to have access to the raw SYNs?
" Dumazet indicated that it was believed to be safe to
provide the IP and TCP headers.
The patch was applied by Miller on May 5, though he noted that the behavior when a too-small buffer was passed to getsockopt() should be rethought. The original patch simply copied as much data as it could into the user-space buffer, but that gave no indication that the SYN headers were not complete. Miller suggested that it should return an error and indicate the proper length so that the program could allocate more space if needed. Munson subsequently posted a patch to do just that.
The feature seems like it will be useful; it appears that it already has been for Google. It is interesting to note that the company has been collecting these fingerprints on (at least) some portion of its vast server farm, though it is not clear what it is doing with all of that information. Soon, though, others will be able to do so too—once 4.2 is released.
An introduction to Clear Containers
Containers are hot. Everyone loves them. Developers love the ease of creating a "bundle" of something that users can consume; DevOps and information-technology departments love the ease of management and deployment. To a large degree, containers entered the spotlight when Docker changed the application-development industry on the server side in a way that resembles how the iPhone changed the client application landscape.The word "container" is not just used for applications, though; it is also used to describe a technology that can run a piece of software in an isolated way. Such containers are about using control groups to manage resources and kernel namespaces to limit the visibility and reach of your container app. For the typical LWN reader, this is likely what one thinks about when encountering the word "container."
Many people who advocate for containers start by saying that virtual machines are expensive and slow to start, and that containers provide a more efficient alternative. The usual counterpoint is about how secure kernel containers really are against adversarial users with an arsenal of exploits in their pockets. Reasonable people can argue for hours on this topic, but the reality is that quite a few potential users of containers see this as a showstopper. There are many efforts underway to improve the security of containers and namespaces in both open-source projects and startup companies.
We (the Intel Clear Containers group) are taking a little bit of a different tack on the security of containers by going back to the basic question: how expensive is virtual-machine technology, really? Performance in this regard is primarily measured using two metrics: startup time and memory overhead. The first is about how quickly your data center can respond to an incoming request (say a user logs into your email system); the second is about how many containers you can pack on a single server.
We set out to build a system (which we call "Clear Containers") where one can use the isolation of virtual-machine technology along with the deployment benefits of containers. As part of this, we let go of the "machine" notion traditionally associated with virtual machines; we're not going to pretend to be a standard PC that is compatible with just about any OS on the planet.
To provide a preview of the results: we can launch such a secured container that uses virtualization technology in under 150 milliseconds, and the per-container memory overhead is roughly 18 to 20MB (this means you can run over 3500 of these on a server with 128GB of RAM). While this is not quite as fast as the fastest Docker startup using kernel namespaces, for many applications this is likely going to be good enough. And we aren't finished optimizing yet.
So how did we do this?
Hypervisor
With KVM as the hypervisor of choice, we looked at the QEMU layer. QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual).
However, we have the kvmtool mini-hypervisor at our disposal (LWN has covered kvmtool in the past). With kvmtool, we no longer need a BIOS or UEFI; instead we can jump directly into the Linux kernel. Kvmtool is not cost-free, of course; starting kvmtool and creating the CPU contexts takes approximately 30 milliseconds. We have enhanced kvmtool to support execute-in-place on the kernel to avoid having to decompress the kernel image; we just mmap() the vmlinux file and jump into it, saving both memory and time.
Kernel
A Linux kernel boots pretty fast. On a real machine, most of the boot time in the kernel is spent initializing some piece of hardware. However, in a virtual machine, none of these hardware delays are there—it's all fake, after all—and, in practice, one uses only the virtio class of devices that are pretty much free to set up. We had to optimize away a few early-boot CPU initialization delays; but otherwise, booting a kernel in a virtual-machine context takes about 32 milliseconds, with a lot of room left for optimization.
We also had to fix several bugs in the kernel. Some fixes are upstream already and others will go upstream in the coming weeks.
User space
In 2008 we talked about the 5-second boot at the Plumbers Conference, and, since then, many things have changed—with systemd being at the top of the list. Systemd makes it trivial to create a user space environment that boots quickly. I would love to write a long essay here about how we had to optimize user space, but the reality is—with some minor tweaks and just putting the OS together properly—user space boots pretty quickly (less than 75 milliseconds) already. (When recording bootcharts at high resolution sampling, it's a little more, but that's all measurement overhead.)
Memory consumption
A key feature to help with memory consumption is DAX, which the 4.0 kernel now supports in the ext4 filesystem. If your storage is visible as regular memory to the host CPU, DAX enables the system to do execute-in-place of files stored there. In other words, when using DAX, you bypass the page cache and virtual-memory subsystem completely. For applications that use mmap(), this means a true zero-copy approach, and for code that uses the read() system call (or equivalent) you will have only one copy of the data. DAX was originally designed for fast flash-like storage that shows up as memory to the CPU; but in a virtual-machine environment, this type of storage is easy to emulate. All we need to do on the host is map the disk image file into the guest's physical memory, and use a small device driver in the guest kernel that exposes this memory region to the kernel as a DAX-ready block device.
What this DAX solution provides is a zero-copy, no-memory-cost solution for getting all operating-system code and data into the guest's user space. In addition, when the MAP_PRIVATE flag is used in the hypervisor, the storage becomes copy-on-write for free; writes in the guest to the filesystem are not persistent, so they will go away when the guest container terminates. This MAP_PRIVATE solution makes it trivial to share the same disk image between all the containers, and also means that even if one container is compromised and mucks with the operating-system image, these changes do not persist in future containers.
A second key feature to reduce memory cost is kernel same-page merging (KSM) on the host. KSM is a way to deduplicate memory within and between processes and KVM guests.
Finally, we optimized our core user space for minimal memory consumption. This mostly consists of calling the glibc malloc_trim() function at the end of the initialization of resident daemons, causing them to give back to the kernel any malloc() buffers that glibc held onto. Glibc by default implements a type of hysteresis where it holds on to some amount of freed memory as an optimization in case memory is needed again soon.
Next steps
We have this working as a proof of concept with rkt (implementing the appc spec that LWN wrote about recently). Once this work is a bit more mature, we will investigate adding support into Docker as well. More information on how to get started and get code can be found at clearlinux.org, which we will update as we make progress with our integration and optimization efforts.
Patches and updates
Kernel trees
Architecture-specific
Core kernel code
Device drivers
Device driver infrastructure
Documentation
Filesystems and block I/O
Memory management
Networking
Security-related
Virtualization and containers
Miscellaneous
Page editor: Jonathan Corbet
Distributions
A preview of Fedora 22
Fedora 22 is currently scheduled to be released at the end of May. While certain changes—such as the switch from the Yum package manager to DNF—have received a fair share of attention already, there are quite a few other differences that users can expect to see in the final release.
Naturally, a great many packages will be updated. Run-of-the-mill applications can be bumped up to a newer release without fanfare, but updates to certain critical components (such as core libraries, servers, and development frameworks) require approval by the Fedora Engineering Steering Committee (FESCo) before they make it onto the official list of accepted changes.
The GNOME 3.16 release, for example, will be included in Fedora 22. That will provide users with access to a number of improvements, as well as giving them a taste of several new applications—like the e-book manager GNOME Books and the Unicode character browser GNOME Characters.
Arguably a bigger change, though, is the inclusion of KDE's Plasma 5 environment. Plasma 5 represents a major overhaul, moving to the Qt5 and KDE Frameworks 5 libraries (both of which were available in Fedora 21). Plasma 5 includes only the core KDE desktop components: the KWin window manager, workspaces, and settings. But it does offer a fully hardware-accelerated graphics stack that uses OpenGL ES. Updating individual KDE applications may take more time, since the move from Plasma 4 to Plasma 5 includes API breaks (Fedora's KDE special-interest group is patching some Plasma 4 applications and retiring others), but including Plasma 5 is an important milestone for KDE users.
At a much lower level, the default compiler in Fedora 22 will change to GCC 5. Only a handful of libraries and other packages in Fedora had to be recompiled with GCC 5, since most do not rely on version-specific compiler features. FESCo did consider rebuilding all of Fedora 22 with the updated compiler, but ultimately decided against it. There are several significant improvements to be found, however, most notably support for the OpenACC 2.0a specification and support for OpenMP 4.0 offloading. Both allow programs to more easily utilize hardware-acceleration devices like GPUs. GCC 5 also improves C++11 support and adds initial support for C++14.
Another under-the-hood improvement is an update to Glibc's Unicode data, from version 5.1 to version 7.0. Users who rely on European languages and locales may not notice the change, but it will bring long-awaited improvements to character support to a number of Indic, Asian, and Australasian languages. "Long-awaited" is quite literal in this case; Unicode 5.1 was released in April 2008.
Fedora 22 will also migrate a few important components from running X.Org to running on Wayland by default. First is the GNOME Display Manager (GDM) login screen, which will now run on Wayland by default, but can fall back to X in the event of a failure. Since the login screen is isolated from normal user sessions, this change will hopefully allow Fedora to field-test Wayland support without putting its users' sessions at any significant risk.
Second, Fedora 22 will move input devices from the old set of X.Org drivers to libinput on top of Wayland. For now, that change will only affect Fedora Workstation, and the input stack will provide an X.Org compatibility layer through the xorg-x11-drv-libinput driver, but it is an important change for the long term. Libinput provides a unified input model that functions the same regardless of the device; X.Org needed separate drivers for each device class (e.g., keyboard, trackpad, mouse, tablet).
Speaking of Fedora Workstation, one of the more significant outcomes of the Fedora.next effort was the introduction of multiple Fedora "products"—namely, Server, Workstation, and Cloud versions of Fedora, each of which is planned and managed by a distinct team. Fedora 21 was the first release to implement the products model, but the Fedora 22 release may be the first to really show how the new model will differentiate between the offerings.
For example, the Server product working group has added a new database server role for Fedora 22. This is a deployment mechanism that lets administrators easily set up a Fedora 22 box to act as a database server, which Fedora has never targeted before. Obviously, users have been able to configure Fedora machines to run databases, but it is hoped that by paying special attention to the packaging and configuration needs, the distribution can attract more server administrators. The deployment scheme uses the Cockpit administration tool for setup and database management.
Two other product-driven changes are still under development, but may yet be ready when Fedora 22 is released. The first is support for rpm-ostree, which implements Colin Walters's OSTree, an image-based means to deploy a Linux distribution. Rather than installing individual RPM packages, OSTree builds a new system image on a remote server for any requested set of changes, then deploys that image to the machine. The result is an update system that allows easy rollbacks, rapid replication across multiple machines, and atomic updates.
Rpm-ostree support is a dependency for the other change, support for Red Hat's Project Atomic deployment scheme. Project Atomic takes the image-based, atomic-update model of OSTree and tunes it to deploy Linux containers. The change slated for Fedora 22 is to allow a Fedora Cloud instance to be deployed as the Atomic "host" system on which containers will run. In addition to rpm-ostree support, this change involves ensuring that Fedora Cloud images will run under VirtualBox and libvirt/qemu as well as ensuring that they will run the latest Docker containers.
There are, of course, many other changes that users should look forward to in Fedora 22—it will be the first release to support the Elasticsearch search engine, for example, and users on UEFI machines will be able to update their Secure Boot blacklists. On the other hand, users who were following the password-strength debate of the last few months may note that the changes proposed to resolve that issue have been deferred to Fedora 23.
So far, the Fedora products model seems to working well for the distribution; it will be interesting to see where things head next after Fedora 22 is out the door. With the distribution looking for ways to attract new users, more tailoring of the individual products could certainly prove to be a plus.
Brief items
Distribution quotes of the week
And he told that there once was a mighty king in the land of ancient desktops, called KDE- The King of DEsktops!
And though the name and places have changed, its stories are still told and memories recalled.
As it is known, there comes a point in time, where all things present must become a thing of the past to make way for the new and shiny.
The King is dead – long live the King!
PC-BSD 10.1.2 Released
The PC-BSD team has announced the release of PC-BSD 10.1.2. This version features a new PersonaCrypt Utility. The Utility allows users to move $HOME to an encrypted USB Drive. Additionally, it includes a Stealth Mode that allows a user to login to a blank $HOME directory, which is encrypted with a one-time GELI key, and later discarded at logout.
Distribution News
CentOS
CentOS-7 disk images for AArch64 Platforms
A disk image of CentOS 7 is available for the AArch64 platform. "This allows a vendor to bypass the installer or to edit the disk image before booting in order to test kernel modules or options. It is intended for development purposes only, and will only continue through the alpha and beta test phases."
Debian GNU/Linux
Bits from the DPL - May 2015
Neil McGovern, the new Debian project leader (DPL), will continue with monthly bits from the DPL. The May edition covers the Jessie release, interviews, libdvdcss, ZFS on Linux, and more.
openSUSE
Tumbleweed moves to Plasma 5.3 and a new release of KDE Applications
KDE Plasma 5.3 has landed in Tumbleweed. "The packages being made available today include a switch to Plasma 5 as default desktop. The openSUSE KDE team deems this release stable for day-to-day usage and a considerable step up from the Plasma 4.x series. While the KDE community continues to maintain the 4.x series for a few more months, by the end of this year, all efforts will be directed toward the Plasma 5 series. Applications are porting in a swift manner with even the KDE PIM team eyeballing a Frameworks 5 release around August and half of the KDE Applications are now Frameworks 5 based."
Newsletters and articles of interest
Distribution newsletters
- DistroWatch Weekly, Issue 610 (May 18)
- 5 things in Fedora this week (May 15)
- Tumbleweed - Review of the week (May 18)
- Ubuntu Weekly Newsletter, Issue 417 (May 17)
Stripped to the core and full of Xfce: Xubuntu Linux loses it (The Register)
The Register reviews Xubuntu Core. "The "core" name is a little confusing since Ubuntu proper recently began shipping Ubuntu Core, a lightweight version of Ubuntu optimized for container-based environments like Docker. Xubuntu core is unrelated and derived from Xubuntu, not Ubuntu Core. Xubuntu core is also not, like many distros using words like "core,” aimed at the server or Docker instances. Xubuntu core remains focused on the desktop, but wants to deliver a lightweight desktop ready for customization, rather than a complete, ready-to-use desktop like Xubuntu proper."
Page editor: Rebecca Sobol
Development
PostgreSQL: the good, the bad, and the ugly
The PostgreSQL development community is working toward the 9.5 release, currently planned for the third quarter of this year. Development activity is at peak levels as the planned feature freeze for this release approaches. While this activity is resulting in the merging of some interesting functionality, including the long-awaited "upsert" feature (described below), it is also revealing some fault lines within the community. The fact that PostgreSQL lacks the review resources needed to keep up with its natural rate of change has been understood for years; many other projects suffer from the same problem. But the pressures on PostgreSQL seem to be becoming more acute, leading to concerns about fairness in the community and the durability of the project's cherished reputation for high-quality software.
The good
The news from PostgreSQL is certainly not all bad, though; the project continues to grow and to add interesting new features. One feature that was recently merged for the 9.5 release fills a longstanding gap: the lack of an "upsert" command.
The SQL standard provides two basic commands for placing data into a table. The INSERT command adds new rows, while UPDATE will modify the data stored in existing rows. What is missing is a way to modify a row if it exists, but to add a new row otherwise; that leads to the need for program logic to check for the existence of the row in question before choosing the correct command to make the change. Many relational database systems contain an "upsert" mechanism that combines INSERT and UPDATE in this way, but PostgreSQL has always lacked this feature.
Adding this functionality is harder than one might expect. One can start with the question of what it means for a row to be already present in a table. A simple answer is that the table has a primary key and that a row containing that primary key already exists, but one could imagine more subtle approaches. Once the problem of defining "presence" has been solved, there is the little problem of dealing with race conditions; the operation as a whole needs to be atomic or bad things will surely result. The challenges involved in getting this feature right have kept it out of PostgreSQL so far.
With 9.5, though, that situation will change — though there is still no UPSERT command. Instead, the new syntax, as implemented by Peter Geoghegan, looks like this:
INSERT INTO table_name <insert stuff>
ON CONFLICT conflict_target conflict_action;
The conflict_target is, in its simplest form, the name of a column for which a uniqueness requirement would be violated if the row were to be inserted directly. The conflict_action, instead, can take either of two forms: either DO NOTHING (which causes the insert operation for the conflicting row to be silently aborted) or DO UPDATE, which provides an UPDATE command to execute instead of the INSERT. There are, of course, numerous details that have been skipped over here; see the above-linked document for the full story.
This feature went through a number of cycles of review and change; it was finally committed to the PostgreSQL mainline on May 8. Once the 9.5 release is available, a famous gap in PostgreSQL functionality will be no more.
The bad
On May 9, PostgreSQL developer Robert Haas asked for suggestions toward the solution of some nasty data-corruption issues associated with the "multixact" feature, which is an internal mechanism supporting the use of multiple locks on any given row in order to increase concurrency. There are, it seems, a number details involved with the multixact feature that have not yet been thoroughly worked out. That might be acceptable for a feature planned for the 9.5 release, but, unfortunately, the relevant multixact changes were merged during the 9.3 development cycle. The 9.3 release happened in September 2013, but the fallout from that particular change is still being dealt with.
After Robert and others dealt with the latest round of problems, Bruce Momjian started a discussion on how this particular patch should have been dealt with. He suggested that perhaps the feature should have been reverted, but that didn't happen.
The ensuing discussion made it clear that there is no single thing that can be fixed to avoid the possibility of similar problems in the future. Bruce feels that there are not enough people jumping in to fix bugs when they come around. Joshua Drake pointed his finger at the PostgreSQL release cycle which is, he says, too short and too oriented toward the addition of new features. The one-year cycle used by the project is already long relative to many others, though. Tom Lane noted that the ratio of developers paid to fix bugs to those paid to add new features is too low, making it hard to get a problematic feature fixed. Robert attributed the bulk of the problems to the complexity of the particular patch involved.
Whatever the reason, there is concern within the PostgreSQL community that its well-earned reputation for low bug rates is at risk. That concern carried over into Robert's message on the remainder of the schedule for the 9.5 development cycle. While he is glad to see stuff being pushed in under the deadline, some of it worries him, too:
He noted that he is already worried about some of the patches, upsert among them, that have gone in for 9.5. Bruce, despite his concerns mentioned above, still seems to be more worried about the prospect that interesting features will be left out of 9.5; he has encouraged reviewers to push to get as many of them in as possible. But the mood of the community as a whole appears to be moving in the direction of letting questionable features wait rather than rushing them into the 9.5 release.
The ugly
That mood can be seen in Tom's posting of the remaining open items and his suggested disposition of each, but it turned a bit uglier as people read the list and saw what was proposed. In particular, Tom recommended that the grouping sets feature should be left out this time around. Grouping sets allow for more flexible grouping of results from a query; the feature has been under development for some time.
The specific problem in this case is that Tom had stated some months ago that he would be the one to shepherd that patch set through the review process. His role in the project is such that, once he lays claim to a patch in that way, other developers are unlikely to work on it; why bother, when the patch appears to be in the hands of one of the community's top developers? In this case, though, Tom never found the time to help get the patch into shape for merging; since nobody else did either, the patch languished. When he then said that the patch should not go into 9.5 because it was not ready, that led to charges of unfairness and calls to give the patch another chance.
This episode has made it clear that some developers, at least, are worried about how the development process looks to PostgreSQL contributors. As Josh Berkus put it:
Andres Freund observed that the grouping sets patch poses a relatively low risk to the stability of PostgreSQL as a whole; it is highly unlikely to cause data corruption problems. There have been a number of other patches, he said, that have gone into 9.5 with less review and in worse shape. Keeping this patch out, he said, is likely to worsen the perception of the development community as a whole:
In the end, Andres committed the grouping sets patch on May 16, so this feature will indeed be a part of the 9.5 release, but the incident seems to have left a bad taste in some developers' mouths. Nobody seems to believe that Tom intentionally stalled the patch, but they would have liked to see this story play out in a different way.
In general, it seems that the "commitfest" process used by the PostgreSQL community since the 8.4 release is showing some signs of age. Andres suggested that commitfests are frustrating for both reviewers and contributors while blocking good patches and failing to keep the bad ones out. He suggested a more direct "judgefest" process that would try to reach quick decisions on patches without allowing them to languish, but there has not been a lot of uptake for this idea on the list. This is not the first time the commitfest model has been called into question, of course, but the voices seem to be getting louder.
Development-process issues seem to come to the fore in just about every successful project at some point; a process that worked at one point in a project's evolution may well be unsuited to the next stage. In the end, a project is not just hacking the code; it must also hack the process by which it produces that code. That can make for some messy public discussions, but most projects come out of process-hacking phases as more smoothly functioning, better-tuned machines. PostgreSQL is not working badly now by any objective standard; with luck, its community will find a way to make it work even better.
Brief items
Quotes of the week
Rust 1.0 released
Version 1.0 of the Rust language has been released. "The 1.0 release marks the end of that churn. This release is the official beginning of our commitment to stability, and as such it offers a firm foundation for building applications and libraries. From this point forward, breaking changes are largely out of scope (some minor caveats apply, such as compiler bugs). That said, releasing 1.0 doesn’t mean that the Rust language is “done”. We have many improvements in store. In fact, the Nightly builds of Rust already demonstrate improvements to compile times (with more to come) and includes work on new APIs and language features, like std::fs and associated constants."
20 years of Qt
Lars Knoll marks the 20th anniversary of the Qt toolkit on the Qt blog. "From the beginning, Qt has been released with both open source and commercial licensing options. Over the years, we have worked on expanding this model, and nowadays, Qt is actually developed as an open source project. In this sense Qt is actually in a rather unique position, having a strong ecosystem with passionate people, as well as a commercial entity behind it, which backs up and funds most of the development."
PyCA 0.9 released
Version 0.9 of the PyCA cryptography library has been released. Among the many improvements are support for several additional extensions for X.509 certificates, support for the SECP256K1 elliptic curve, and support for DER serialization of keys. Also noteworthy is that this release removes support for Python 3.2 and deprecates support for Python 2.6.
Twisted 15.2 available
Version 15.2 of the Twisted framework is available. This release includes "a brand-new, feature-rich logging framework
" called twisted.logger, Unicode support in file paths, and Python 3.4 support for all Python 3–ported modules—to which there are several new additions.
Newsletters and articles
Development newsletters from the past week
- What's cooking in git.git (May 19)
- GNU Toolchain Update (May 18)
- LLVM Weekly (May 18)
- OCaml Weekly News (May 19)
- Perl Weekly (May 18)
- PostgreSQL Weekly News (May 17)
- Python Weekly (May 14)
- Ruby Weekly (May 14)
- This Week in Rust (May 18)
- Tor Weekly News (May 14)
- Wikimedia Tech News (May 18)
Goodbye, Pi. Hello, C.H.I.P. (Linux Journal)
Linux Journal takes a look at the C.H.I.P. mini-computer, an open software and hardware device that comes with a Debian-based OS. "The official public release is scheduled for next year, but crowdfunding backers will be able to land a "Kernel Hacker" package this September. This package is aimed at Linux developers who want to help to contribute to kernel modifications for the C.H.I.P. before its final release."
Page editor: Nathan Willis
Announcements
Brief items
LibrePlanet forever! Watch five sessions from 2015 online
Videos for 5 LibrePlanet sessions are available, including "Stand up for the GNU GPL!" by Karen Sandler, "Free software, free hardware, and other things" by Richard Stallman, "Let's encrypt" by Seth Schoen, "Style or substance? Free software is totally the 80's" by Deb Nicholson, and "Federation and GNU" by Chris Webber.
Articles of interest
3 big lessons I learned from running an open source company (Opensource.com)
Over at Opensource.com, Lucidworks co-founder and CTO Grant Ingersoll writes about lessons he has learned from running an open-source company. "You might ask, 'Why not open source it all and just provide support?' It's a fair question and one I think every company that open sources code struggles to answer, unless they are a data company (e.g., LinkedIn, Facebook), a consulting company, or a critical part of everyone's infrastructure (e.g., operating systems) and can live off of support alone. Many companies start by open sourcing to gain adoption and then add commercial features (and get accused of selling out), whereas others start commercial and then open source. Internally, the sales side almost always wants "something extra" that they can hang their quota on, while the engineers often want it all open because they know they can take their work with them."
How to Make Money from Open Source Platforms (Linux.com)
Over at Linux.com, John Mark Walker examines why companies aren't making money on pure open source ventures. "It is not that there is no money in selling open source software, but rather that the business models have shifted. Whereas, under the old proprietary world, a larger percentage of money went to pure software vendors, now that money has spread among a larger spectrum of companies and industries; lots of people get paid to work on or with open source software, but an increasing number of them don’t work for software vendors, per se. In addition to looking in all the wrong places, the current investment model is suspicious of an open source approach. The vast majority of venture capitalists, especially in Silicon Valley, are very risk averse and shy away from open source products that, in their view, will not give as large a return on their investment. In order to secure the funding required to scale a company, investors will frequently require that the startup company include proprietary bits as tools to increase revenue and margins. These two factors - diffusion of revenue and risk-averse investors - combine to both give a false impression and, in part due to the false impression, prevent pure open source software vendors from getting funding."
Calls for Presentations
CFP Deadlines: May 21, 2015 to July 20, 2015
The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.
| Deadline | Event Dates | Event | Location |
|---|---|---|---|
| May 23 | August 22 August 23 |
Free and Open Source Software Conference | Sankt Augustin, Germany |
| May 23 | May 23 May 25 |
Wikimedia/MediaWiki European Hackathon | Lyon, France |
| May 31 | October 2 October 4 |
PyCon India 2015 | Bangalore, India |
| June 1 | November 18 November 22 |
Build Stuff 2015 | Vilnius, Lithuania |
| June 1 | July 3 July 5 |
SteelCon | Sheffield, UK |
| June 5 | August 20 August 21 |
Linux Security Summit 2015 | Seattle, WA, USA |
| June 6 | September 29 September 30 |
Open Source Backup Conference 2015 | Cologne, Germany |
| June 11 | June 25 June 28 |
Linux Vacation Eastern Europe 2015 | Grodno, Belarus |
| June 15 | August 15 August 22 |
DebConf15 | Heidelberg, Germany |
| June 15 | September 24 | PostgreSQL Session 7 | Paris, France |
| June 15 | November 17 November 18 |
PGConf Silicon Valley | San Francisco, CA, USA |
| June 17 | October 5 October 7 |
LinuxCon Europe | Dublin, Ireland |
| June 17 | October 5 October 7 |
Embedded Linux Conference Europe | Dublin, Ireland |
| June 19 | August 20 | Tracing Summit | Seattle, WA, USA |
| June 28 | August 28 September 3 |
ownCloud Contributor Conference | Berlin, Germany |
| June 28 | November 11 November 13 |
LDAP Conference 2015 | Edinburgh, UK |
| June 30 | November 16 November 19 |
Open Source Monitoring Conference 2015 | Nuremberg, Germany |
| July 5 | October 27 October 29 |
Open Source Developers' Conference | Hobart, Tasmania |
| July 15 | October 8 October 9 |
CloudStack Collaboration Conference Europe | Dublin, Ireland |
| July 15 | October 27 October 30 |
OpenStack Summit | Tokyo, Japan |
| July 15 | November 6 November 8 |
Jesień Linuksowa 2015 | Hucisko, Poland |
| July 17 | October 2 October 3 |
Ohio LinuxFest 2015 | Columbus, OH, USA |
| July 19 | September 25 September 27 |
PyTexas 2015 | College Station, TX, USA |
If the CFP deadline for your event does not appear here, please tell us about it.
Upcoming Events
EuroPython 2015: Come with your partners
EuroPython will take place July 20-26 in Bilbao, Spain. The organizers have announced the EuroPython Partner Program, which offers interesting tours for EuroPython attendees and their partners. The registration deadline for the tours is June 20, seats are limited.Next year's Intl. Day Against DRM: Are you in?
The FSF's DefectiveByDesign campaign reports that the 2015 International Day Against DRM was the biggest ever and urges everyone to help make the 2016 edition better still.Events: May 21, 2015 to July 20, 2015
The following event listing is taken from the LWN.net Calendar.
| Date(s) | Event | Location |
|---|---|---|
| May 18 May 22 |
OpenStack Summit | Vancouver, BC, Canada |
| May 19 May 21 |
SAMBA eXPerience 2015 | Goettingen, Germany |
| May 20 May 22 |
SciPy Latin America 2015 | Posadas, Misiones, Argentina |
| May 21 May 22 |
ScilabTEC 2015 | Paris, France |
| May 23 May 24 |
Debian/Ubuntu Community Conference Italia - 2015 | Milan, Italy |
| May 23 May 25 |
Wikimedia/MediaWiki European Hackathon | Lyon, France |
| May 30 May 31 |
Linuxwochen Linz 2015 | Linz, Austria |
| June 1 June 2 |
Automotive Linux Summit | Tokyo, Japan |
| June 3 June 5 |
LinuxCon Japan | Tokyo, Japan |
| June 3 June 6 |
Latin American Akademy | Salvador, Brazil |
| June 5 June 7 |
PyCon APAC 2015 | Taipei, Taiwan |
| June 8 June 10 |
Yet Another Perl Conference 2015 | Salt Lake City, UT, USA |
| June 10 June 13 |
BSDCan | Ottawa, Canada |
| June 11 June 12 |
infoShare 2015 | Gdańsk, Poland |
| June 12 June 14 |
Southeast Linux Fest | Charlotte, NC, USA |
| June 16 June 20 |
PGCon | Ottawa, Canada |
| June 22 June 23 |
DockerCon15 | San Francisco, CA, USA |
| June 23 June 26 |
Red Hat Summit | Boston, MA, USA |
| June 23 June 25 |
Solid Conference | San Francisco, CA, USA |
| June 23 June 26 |
Open Source Bridge | Portland, Oregon, USA |
| June 25 June 28 |
Linux Vacation Eastern Europe 2015 | Grodno, Belarus |
| June 25 June 26 |
Swiss PostgreSQL Conference | Rapperswil, Switzerland |
| June 26 June 27 |
Hong Kong Open Source Conference 2015 | Hong Kong, Hong Kong |
| June 26 June 28 |
FUDCon Pune 2015 | Pune, India |
| July 3 July 5 |
SteelCon | Sheffield, UK |
| July 4 July 10 |
Rencontres Mondiales du Logiciel Libre | Beauvais, France |
| July 6 July 12 |
SciPy 2015 | Austin, TX, USA |
| July 7 July 10 |
Gophercon | Denver, CO, USA |
| July 15 July 19 |
Wikimania Conference | Mexico City, Mexico |
| July 18 July 19 |
NetSurf Developer Weekend | Manchester, UK |
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol