Sizing up the ColorHug open source colorimeter

Having covered the open source graphics software community for several years, I was immediately intrigued when I learned about the ColorHug project. ColorHug is a spare-time effort by Red Hat's Richard Hughes to build an open source monitor calibration device. I am still resigned to the fact that most people's eyes will glaze over whenever you utter the phrase "color management," but with ColorHug, at least interested parties have a simple and affordable option — and as with any good open source project, there are hints at more things still to come.

Color me good

For the benefit of the uninitiated, the ColorHug is a tristimulus colorimeter. That means it can measure the amount of RGB light hitting its sensor; by displaying a carefully-chosen set of color swatches on the screen, it can characterize the display's output and generate an ICC profile. You can then load the ICC profile into the OS color management system, and enjoy balanced output. The ColorHug sports a 64-pixel light sensor, and like other colorimeters uses filters to read the red, green, and blue values separately. Unlike many other colorimeters, the ColorHug also converts the readings into a standardized XYZ color value in the firmware, and can correct the output to account for the different illumination characteristics (e.g., varying backlights and primary colors) found in different display types.

Unlike essentially every other colorimeter, the ColorHug is supported only under Linux — it ships with a Fedora-based live CD to perform the calibration, although the resulting ICC profile can be used by any operating system. That aspect of the project receives little attention, but it is significant. There are other colorimeters supported by open source software projects like Argyll (which handles underlying calibration tasks), but there is no official Linux support. On top of that, the existing Windows and OS X client software is usually restrictive, with limits on the number of copies that can be installed, requiring email registration, and so on. Hughes made a point of designing the ColorHug to be fast and accurate hardware at a low price point (currently £60), but the software support is equally significant. It even gets free firmware updates, which could make for some interesting developments down the road.

How it works

Physically the ColorHug is a small black puck about the size of a matchbox, with a sensor aperture on one side and a mini-USB port on the end. It comes with a mini-to-fullsize USB cable and the aforementioned live CD. Hughes is also the author and maintainer of GNOME Color Manager (g-c-m) and colord, so that software is used by the live CD to detect attached displays and colorimeters, as well as to step the user through the process of creating a display profile. G-c-m should automatically recognize your displays and list them in its main window; to start creating a profile you simply select one and click the "Create Profile" button.

The application will ask you to name your new profile (although it maintains a time-sorted list of the resulting profiles in the window, too, so the name is primarily for your benefit) and select the accuracy at which to analyze the monitor. The approximate measurement times listed — up to 20 minutes for the most thorough — are very conservative when it comes to the ColorHug; I cannot think of any reason to create a low-accuracy profile other than curiosity. I tested the ColorHug against a Pantone Huey (a common proprietary colorimeter), and creating an accurate profile took well under 10 minutes for each device, with a slight edge going to the ColorHug.

Whatever accuracy you choose, when you click the "Start" button the application launches into a sequence of color patches displayed in the center of the screen. You must have the ColorHug centered on the display area and keep it still for the duration of the process. That proved to be the trickiest part of the formula for me; the USB cable is on the thick side, and the ColorHug weighs so little that the cable pulls it out of position if you don't take care to wedge it in place with a nearby object. Even then, profiling a laptop is easier because you can maneuver the display to lie flat. Back when CRTs were the norm, proprietary colorimeters often used suction cups to stick to the front of the monitor. That is clearly a bad idea for LCDs, but some form of clip to hang the device in place against a vertical surface might be a nice addition in the future.

Calibration example
Before	After

After a few minutes of silent color-swatch juggling, g-c-m announces that it is finished, and you can instantly apply the freshly baked ICC profile to your display. This is the point where you might notice trouble. Early versions of the software asked you to select the white point of the monitor during the first step, and the wording encouraged you to select the daylight-balanced D65, but that resulted in profiles skewed sharply into the red part of the spectrum. The correct choice was to use the display's "Native" white point, and updated versions of the software and live CD now choose that setting automatically.

The other tricky bit is that it might be hard to find the actual ICC profile files if you are running the live CD. They are located in ~/.local/share/icc/, so you need to copy them to another storage location in order to load them into your normal desktop environment and reap the benefits. However, you still may find the newly-adjusted profile looks different than you expect. The truth is that most LCDs are factory set to a very cool-blue white point because that appears brighter to the human eye. But brighter is not accurate; I immediately noticed that the Fedora live CD's aquatic background image had a lot more color variation when I applied the new ICC profile; without it the entire ocean scene was deep blue. Your eyes will adjust to the change. More importantly, if you profile multiple displays, your eyes will not have to adjust when you switch between them. You can see "before" and "after" photos above ("before" is on the left). Of course, how much of a difference you see depends on your own display as well, but hopefully it is at least clear that the grays are much bluer in the "before" shot.

Using the live CD is actually pretty important; Hughes updates the client code and utilities frequently and his updated ISOs are the only officially guaranteed path to making all of the pieces work together. Your distribution might not package the most recent version of Argyll or other dependencies for several months.

Auxiliary bits

Display profiling is the ColorHug's primary purpose, but the project has other interesting infrastructure built in as well. For instance, it is firmware-upgradeable: the live CD comes with a GUI firmware updater that checks the project web site for a new release, and allows you to download and flash it. The first time I tried that, it did not work. But the ColorHug is also designed with a separate firmware bootloader to prevent accidentally bricking the device. The ColorHug site explains how to recover from a bad flash with command line utilities included on the live CD, and I am happy to say that they worked.

You can also use the ColorHug to calculate a CCMX color transformation matrix, if you have access to a separate spectrophotometer. A spectrophotometer is a different class of calibration device from a colorimeter; rather than using a light sensor to measure RGB values, it measures the amount of light output by a display across the full visual spectrum, in tiny increments. They are pretty expensive in practice, but if you can wrestle one away from a friend, you can use it to create a CCMX matrix that will be shared with the rest of the community. These matrices are most valuable for non-standard display types; most LCD screens work fine with the standard sRGB matrix shipping in the ColorHug. But mobile phones, TVs, and other devices do differ, and the project is interested in collecting as many good samples at it can, which will then be included in future firmware updates. You can grab and install new CCMXes with the colorhug-ccmx tool included on the CD.

On the ColorHug mailing list, there was a thread about using other colorimeters to generate CCMX data. The process relies on Argyll, and author Graeme Gill confirmed that the Argyll tools can create a matrix from two colorimeters. I asked Hughes what the relative value of such a colorimeter-to-colorimeter matrix is, and he likened it to trying to adjust your speedometer by driving alongside another car: not totally useless, but definitely not reliable enough to recommend.

Finally, the G-c-m build on the live CD includes tools for examining ICC profiles themselves. In the version of the CD I tested, they qualify as interesting but not rigorous. For example, you can show a 3D rendering of each profile as a solid space, but you cannot overlay to profiles to compare them. I hope that that will improve in the future, the other major Linux color management system, Oyranos, has a very nice profile inspection tool called ICC Examin, so there is hope.

Speaking of Oyranos, just because colord is used by the live CD, that does not mean that the ColorHug's ICC profiles are usable only with colord on a GNOME system. The profiles are bound to the hardware, whichever software is used to load them. Oyranos developer Kai-Uwe Behrmann is also working on a set of enhancements to ICC file metadata via the OpenICC project which should make interoperability better in the future, but you can simply install the profiles on an Oyranos system and take advantage of them now. You can also submit them to the taxi service, which collects user-generated ICC profiles in an effort to build a more reliable set of real-world options for users who cannot generate their own profiles. You can even use the profiles in Mac OS X and Windows if you multi-boot.

The future

Automatically installing the profiles created by the live CD is a possible enhancement that has come up on the ColorHug mailing list, although for now it is not high-priority. More interesting is the possibility of extending the hardware's usefulness — such as to CRT displays and projectors. Supporting CRTs requires different color-measurement timing, due to that hardware's refresh rate.

Projectors are another beast altogether. There are proprietary colorimeters intended for use with projectors, and technically the ColorHug hardware can be used with them as well — but it is tricky. There is an aperture disk in front of the ColorHug light sensor; removing it allows the sensor to see a larger sample area, which would be useful for projectors, but the details of placing and orienting the device have yet to be worked out. Similarly, removing the aperture disk would also make the device useful for profiling the ambient light in a room, but here again the details have yet to be worked out. Hughes and Gill have discussed the issues on the mailing list — including whether or not a light diffuser would be required, but it is possible that future versions of the software will support these or other new functions.

Looking even further out, Hughes has floated the idea of building an open source spectrophotometer as a sequel or companion piece to the ColorHug. There was interest in the idea on the mailing list, but most thought such a device would be useful only if it was designed to help profile reflective light — in other words, printers. Not all spectrophotometers do so; the hardware challenge is significantly steeper, because in addition to making precise measurements at specific wavelengths, a reflective light spectrometer also needs to provide a precise illumination source, which is a tricky prospect for any project. Hughes indicated that such a device was a 2013-or-later project, and it would cost at least four or five times as much as the ColorHug.

Of course, the ColorHug proves that there is at least some market for open source color-measurement hardware, which few people would have predicted before the project launched. In fact, according to Hughes' talk at Libre Graphics Meeting 2012, he can barely keep up with demand for the devices. The tale of designing, manufacturing, and iterating the devices is interesting for anyone looking for real-world information on open hardware projects; hopefully slides or a recording of the session will be available soon. As an open hardware device, you can download the gEDA schematics and the firmware for the ColorHug from the same repository that hosts the client software. But for everyone who is merely interested in getting good results from a monitor, the ColorHug is a good buy on its own. It would probably be a good buy even if it was closed source: it is fast, inexpensive, and accurate. But the fact that you can tweak and modify it only adds to the value.

Comments (28 posted)

Relicensing and rebasing LibreOffice

Projects the size of LibreOffice often tend to get a little unwieldy; the size of the code is such that even seemingly trivial tasks like removing dead code can take a long time. Considering the sheer size of the project and the fact that its copyright ownership is distributed, it would be natural to doubt the sanity of anybody proposing to simultaneously move 1.5 years worth of work to a new base and adopt a new license. But that is just what LibreOffice has in mind.

LibreOffice is based on the original OpenOffice.org project; it inherited its current LGPLv3 license from there. The project is clearly happy with copyleft licensing, but it occasionally shows signs of wanting a bit more flexibility than LGPLv3 sometimes provides. LibreOffice has, since the beginning, accepted all changes under a dual license, mixing LGPLv3 with version 2 of the Mozilla Public License. But the LibreOffice project does not own the original OpenOffice.org code, so it must distribute its work under LGPLv3, essentially dropping the MPLv2 license from the changes that have been made since the project began.

Something interesting happened a while after LibreOffice launched, though: Oracle donated the code to the Apache Software Foundation (ASF) which, after some work, has released it under the Apache License. That is a permissive license, which is not something LibreOffice is interested in. But the Apache License is compatible with MPLv2; a derived product containing code under both licenses can be distributed as an MPLv2-licensed whole. Thus, LibreOffice has concluded:

In other words, LibreOffice intends to toss out its original OpenOffice.org base and move its changes over to the newly released, Apache licensed version; the result will be a LibreOffice that can be distributed under the MPL, which the project sees as being worthwhile:

Getting there will be an interesting process, though. The code given to the ASF is substantially the same as the code LibreOffice started with, but, as some Apache contributors have taken pains to point out, that code is not available under the Apache License. Contrary to the text quoted above, Oracle did not relicense the code; that code was, instead, given to the ASF, which was given the freedom to release it under a new license. The only code that is actually available under the Apache License is that found in the Apache OpenOffice 3.4 release. Anything that has not been officially released by Apache has not truly been relicensed.

Indeed, it has been claimed that code found in the Apache repositories—and, thus, currently distributed by Apache—is not necessarily available under the Apache License. So LibreOffice will have to start with the official release, which has had a lot of components carved out of it. Much of that code has been replaced with permissively-licensed code, but not all of it. There are also several chunks of code that Apache OpenOffice may relicense and integrate into a future release, but that has not happened yet. LibreOffice developers have requested help in clarifying the license status of that code, but answering those queries has not been a high priority for Apache OpenOffice, especially while the latter was working flat-out to make its own first release.

So the relicensing of LibreOffice is not going to be a quick task. The project must start with, and limit itself to, code that is known to be available under the Apache license. Then it will be necessary to rebase the large number of changes made over the lifetime of the LibreOffice project onto the new release; the project thinks that will be relatively easy, because Apache OpenOffice has, thus far, not made a lot of changes:

There remains the task of filling in all of the parts that Apache OpenOffice simply removed from the code base or has not yet gotten around to releasing under the Apache License. How easy that will be is not entirely clear; it may well be that, even after rebasing the project, LibreOffice may not be able to make a fully MPL-licensed release for some time. There are also little details like getting the license headers right, but they seem minor by comparison.

In summary, the LibreOffice project appears to be setting itself up for a fair amount of work. License changes are hard, especially when one does not necessarily have the cooperation of the copyright holders; it is only possible this time around through the surprising combination of the Apache relicensing and the LibreOffice project's prescient-seeming decision to require LGPL/MPL dual licensing on all contributions. The reward from all of this work, the project hopes, will be more flexible licensing that helps it to compete with the project upon which it is rebasing. All told, the interesting interplay between these two projects seems destined to continue for the foreseeable future.

Comments (45 posted)

SFC expands license compliance efforts

To a large extent, BusyBox has been the "poster child" for GPL enforcement, at least in the US. That may now be changing with the announcement that the Software Freedom Conservancy (SFC) has signed up multiple Linux kernel and Samba developers whose copyrights can be used in license compliance efforts. That expands the scope of license enforcement activities while also removing the need to use the controversial GPLv2 death penalty threat to ensure compliance with the kernel's license—SFC should be able to go directly at kernel enforcement now.

Copyright holders step up

There are three parts to the announcement. First off, the Samba project, which is an SFC member project, has moved its existing compliance efforts over to SFC. Secondly, a new "GPL Compliance Project for Linux Developers" has been created for Linux kernel copyright holders to allow their contributions to be used in enforcing the GPL on that code. So far, seven copyright holders have joined the project. Lastly, several other SFC member projects (Evergreen, Inkscape, Mercurial, Sugar Labs, and Wine) have requested that the organization handle any compliance issues for their code.

The kernel copyright holders that have joined the project are—somewhat curiously—not listed, other than Matthew Garrett, who was the first. In an email exchange with SFC executive director Bradley M. Kuhn, he said that he asked Garrett to "officially speak on behalf of this program", but that the others are welcome to speak up if they wish. Essentially, Kuhn is shielding those developers from having to handle questions if they don't want to, which is part of the mission of SFC: "we take care of boring work for Free Software projects so the developers can focus on the developing the code".

Furthermore, Kuhn recognizes that GPL compliance activities are sometimes controversial in the community. So, he is happy to deflect any criticism in his direction:

Similarly, Kuhn asked Jeremy Allison to be the contact person for the Samba copyright holders, who were also unnamed. There were nine of those before the announcement, but Kuhn said that another stepped forward after it went out. Part of the idea behind the announcement is to do just that: attract more copyright holders, particularly kernel copyright holders, to the compliance efforts. Kuhn also expects that some of the copyright holders may make themselves known in "comment threads, blogs and the like".

Copyright coverage

One question that arises when only a subset of the copyrights in a body of code are being used is whether those copyrights are sufficient to be used for compliance purposes. For Samba, there is little question in Kuhn's mind that Allison's contributions would be sufficient by themselves: "Jeremy's copyrights *alone* are so extensively spread across the many Samba codebases for multiple decades that it's probably impossible to have a working Samba binary without including very substantial portions of Jeremy's copyright works". But he is happy to have other Samba developers on board "to show their solidarity and so that they could give input into the process of the compliance activity".

On the kernel side, Garrett himself noted that his copyrights might not be useful for enforcement activities: "Most of the past work I've done is in bits of the kernel that are rarely present in infringing devices, and most of my recent work is owned by my employer rather than me." But that was an off-the-cuff blog comment that Kuhn said "painted a substantially bleaker outlook than the reality" once SFC started investigating Garrett's copyrights. Beyond that, Kuhn said, the addition of the other copyright holders has further strengthened SFC's hand:

Kuhn said that SFC is now going through the laborious process of registering the copyrights of all of those involved. The basic methodology is "going through git logs, finding commits by a particular developer, verifying their copyright claim to that commit, and then preparing the source code in a form that the copyright office can grok", he said. That will take some time, but, once it's done, those registrations will be searchable by those interested at the US copyright office.

But seven copyright holders is only a tiny fraction of the thousands of contributors to the Linux kernel over the years. To understand the scope of the copyrights that can be enforced, we would need to know just who has signed up with SFC. Until the copyright registration process is completed—or a lawsuit filed—we won't really have a good feel for that it seems.

Compliance, not litigation

When asked about any litigation on the horizon, both Kuhn and Allison were quick to point out that SFC only uses lawsuits as a last resort. The policy has always been to file suit only "when a company simply refuses to respond to repeated requests from Conservancy about compliance problems", Kuhn said, while Allison called lawsuits "a 'LISTEN TO ME !!!!' action" for those who are non-responsive.

In fact, Kuhn said, "the average is less than one lawsuit every three years", though SFC has been accused of being "overly litigious" by some. By way of contrast, SFC deals with more than a hundred minor compliance issues (e.g. some kind of compliance question) each year. In addition, situations that "required complex negotiation" number in the twenties each year, he said. There is also a certain amount of educational work that SFC does, including Kuhn speaking publicly about the issue at various conferences "which (hopefully) reaches a wide audience and (again, hopefully) raises the level of understanding" about compliance for companies beyond those SFC interacts with directly.

The announcement essentially puts companies on warning that there are "cops on the beat", Allison said:

Plans

SFC will be consulting with the copyright holders to determine the compliance steps that are to be taken. The organization is not looking to go its own way, but rather to work with the stakeholders to find a path that is acceptable to all. As the announcement put it:

That attitude may be helpful in bringing in other developers that might be leery of just appointing SFC as their copyright agent. By making it clear that they will be consulted and are able to help define the strategy and tactics, SFC is clearly trying to alleviate fears in the hopes of bringing in more copyright holders.

The addition of kernel and Samba copyrights (and, likely to a lesser degree, those of the other SFC member projects) will clearly expand the number and kinds of license compliance offenders the organization can target. But Kuhn does not see SFC spending any more time on compliance than it has in the past. It is third on the list of SFC activities as reported in its 2010 Form 990 [PDF], and Kuhn does not see that changing as SFC "doesn't want license compliance to be our sole or even primary focus".

That said, there are a number of things that need to be done. Kuhn said that SFC will be giving the non-responsive companies in its queue "another go-around" in hopes that the announcement will spur them to start engaging. That queue, which numbers around 250 violations, will also need to be re-prioritized based on product age and availability, he said.

Compliance is an important issue for our communities. It is not just that some are completely ignoring the—pretty minimal in the grand scheme—requirements that we put on our freely available code, but also that violators are getting an advantage on their compliant competitors. As Allison put it: "The people who do think about compliance are subsidising their competitors who don't. I think that's unfair."

With this announcement, SFC now has more weapons in its arsenal so that it can reduce that problem, while still working within the community it serves. Compliance may be controversial in some circles, but choosing a license that places restrictions on distribution—as the GPL does—doesn't really make sense unless it is enforced. Those who would rather not see SFC (or others) push compliance on companies might be best served by more permissive licenses.

Comments (7 posted)

TACK: TLS key pinning for everyone

Moxie Marlinspike and Trevor Perrin have created a TLS enhancement called Trust Assertions for Certificate Keys (TACK) that is designed to protect users against SSL certificate fraud with minimal overhead or infrastructure. The TACK proposal has been submitted to the IETF, as has a competing key-pinning mechanism created by Google's Chrome team. Although the two systems share broad similarities, their implementations are quite different.

The trouble

TACK is designed to combat flaws with the certificate authority (CA) trust chain model used by applications to check the validity of a remote site's TLS public key. The entire CA chain can be undermined by one bad CA, but there are other difficulties, too. First, any given hostname (e.g., www.example.net) can map to several different servers, with several certificate and thus several different CA chains deployed at once (e.g., in different jurisdictions) — which makes it harder to verify a connection. Second, the CA chain itself may change at any time, due to CA actions that are outside the control of the www.example.net site administrators.

There have been plenty of previous approaches to circumventing CA problems (including Marlinspike's own Convergence). "Certificate pinning" is a client-side technique in which the browser remembers (or "pins") a known-good SSL certificate across multiple browser sessions, and alerts the user if the certificate suddenly seems to change. The TACK proposal argues that certificate pinning is flawed because it binds the trust not to the server's TLS key but to the certificate, which includes the key plus the CA's signature and various other metadata. Thus, by extension, pinning the entire certificate ties trust of the server's TLS key to trust of the CA.

Google Chrome added a key-pinning feature (which tracks known-good TLS keys, even though it is often erroneously referred to as "certificate pinning") with Chrome 13 in mid-2011. But the TACK project says that this, too, has its flaws. First, Chrome's key-pinning uses a whitelist approach, with the browser designating specific CAs as trustworthy. But as long as the CA chain includes a whitelisted CA anywhere in the chain, the whole chain is trusted. Second, if a server issues a new TLS key, it does not roll out to users until a new version of Chrome is released.

TACK

In TACK, the identity-verification mechanism is also key-pinning, but rather than pinning a CA key, the browser pins a special "TACK" data structure created and controlled independently by the web site's administrators. The TACK includes a cryptographic signature of the site's TLS key, plus a public encryption key that the browser can use to verify the signature. Obviously the TACK public key and the TLS public key need to be different. In addition to this basic identity information, the TACK also includes an expiration date and a "generation" counter that gets incremented every time the site administrator creates a new TACK (so that old TACKs can be revoked if necessary). The final piece of the puzzle is that TACK is an extension to TLS itself, rather than an extension to HTTP as in Google's pinning proposal.

In practice, when the client application initiates a TLS connection, it requests a TACK from the server as part of the handshake step. When the server sends a TACK back in return, the client checks the generation information against its current cache for the site (if it has one), checks the expiration date, and validates the signature. If everything checks out, the TACK is "pinned" locally for future reference. The client can still choose to verify the CA chain or use other means to cross-check the identity of the server.

When the client pins a TACK, it is supposed to assign a cache lifespan to it equal to the amount of time between when it first saw the TACK and the present — up to a maximum of 30 days. In other words, a TACK that was first encountered seven days ago gets pinned for another seven days, while a TACK from one year ago gets the 30-day maximum. In the maximum-lifespan case, however, the TACK will only remain in the local cache if the user has returned to the site at least once every 30 days; otherwise it will have expired on the 31st day and a new pin will be created.

The big drawback of the TACK mechanism is that, like other client-side key-pinning methods, it does not protect the user against a man-in-the-middle (MITM) attack during the very first connection attempt to the remote server. It relies on the client having known-good information pinned locally. On the other hand, the TACK generation, cache lifespan, and expiration date features are intended to mitigate the damage that a MITM attack can inflict.

The worst-case scenario is a bad pin that lingers for 30 days, which is half the time the rogue google.com certificate issued by DigiNotar lingered in 2011 — and it could have lasted much longer had it not been discovered. Another advantage of TACK's design is that in the worst-case scenario, the server administrator can single-handedly push out an update, rather waiting on the CA or another party.

But TACK's developers also note that there is no reason that TACKs must only be retrieved "live" by the browser. Search engines could index well-known TACKs and browser-makers could pre-load lists of them into their applications. The TACK team has posted example TACK code for Apache and OpenSSL, plus a Python tool for making TACK requests.

Google, pins, and the future

Google submitted its own key-pinning proposal to the IETF, based on an evolution of its original feature in Chrome. The Google proposal is similar in many ways to TACK. For starters, unlike the built-in pinning currently found in Chrome, it allows each client application to check and cache its own pins rather than depend on a vendor-supplied list. It also pins individual site keys rather than CA keys, so it is independent of the CA chain-of-trust.

Google's pin format includes a max-age field, which takes the place both of TACK's expiration date and its maximum cache-lifespan criteria. Google's proposal does not track generations, however. To recover from a lost or stolen pin-signing key, it instead mandates that server administrators keep a "Backup Pin" — a second, un-deployed signing key that the administrators must store offline. The key fingerprint of the Backup Pin is published by the server, though, so that clients will recognize it if it needs to be deployed. Presumably when the Backup Pin is deployed, the administrators will then need to create a new, backup backup Pin.

There are other details that differ between the proposals (such as what signature algorithms are required), but the most fundamental difference is that Google's pinning proposal is an extension to HTTP: servers send their pinnable key signatures in a new HTTP header, rather than as part of the TLS handshake. TACK's proponents argue that because the CA weakness is fundamentally a TLS problem, it should be fixed in TLS itself, and that building an HTTP-based solution leaves other important protocols (e.g., IMAP and POP3) unprotected. HTTP proponents argue that an HTTP solution is easier to deploy due to the difficulty of setting up a TLS stack, and that HTTP protects the most attack targets.

As of today, TACK is being discussed by the IETF's TLS Working Group. It is a new enough proposal that it is likely to see revisions before it advances any further in the IETF standards process. Google's key pinning proposal was accepted as an Internet-Draft by the Web Security Working Group in December. Whichever makes it to broad browser and server acceptance first, it is fair to say that users will start hearing more about "pinning" in one form or another — but they still should not expect the CA system to disappear any time soon.

Comments (2 posted)

Security quotes of the week

-- Charlie Stross

-- Lauren Weinstein

-- Vint Cerf worries about the future of the internet

-- Joseph Bonneau

-- Ted Ts'o

Comments (53 posted)

SSL fix flags forged certificates before they're accepted by browsers (Ars Technica)

Over at Ars Technica, Dan Goodin writes about Trust Assertions for Certificate Keys (TACK), a proposed extension to SSL/TLS designed to discover fake certificates before they are accepted. "The opt-in system works by allowing SSL sites to sign valid SSL certificates, the domain name, and an expiration date with a TACK key. Once an end user has visited the site a few times using a TACK-compatible browser, a 'pin' for that site is activated on the user's computer. If the end user later encounters a forged certificate for that same site—as was the case when DigiNotar was breached—the browser will reject the session and return a warning to the user." One of TACK's co-creators is Moxie Marlinspike, who proposed the Convergence alternative certificate-management framework in 2011.

Comments (21 posted)

Android Malware Genome Project launched (The H)

The H covers the debut of the Android Malware Genome Project by researchers from North Carolina State University. The team "has already collected more than 1,200 samples of Android malware, including GingerMaster and DroidKungFu, and has organised them into various malware families. [Xuxian] Jiang told Dark Reading that 'the purpose is to engage the research community to better our understanding of mobile threats and develop effective solutions against them.'" Access to the data set, however, is restricted.

Comments (2 posted)

Implementing UEFI Secure Boot in Fedora

On his blog, Matthew Garrett writes about plans for supporting UEFI secure boot in Fedora 18. In it he covers signing the first-stage bootloader with a Microsoft key, while signing GRUB 2, the kernel, modules, etc. with a Fedora key. It is a compromise to try to avoid problems for users who want to boot Linux on Windows 8 hardware. "The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key."

Comments (107 posted)

chromium: multiple vulnerabilities

Package(s):	chromium	CVE #(s):	CVE-2011-3103 CVE-2011-3104 CVE-2011-3105 CVE-2011-3106 CVE-2011-3107 CVE-2011-3108 CVE-2011-3109 CVE-2011-3111 CVE-2011-3115
Created:	May 29, 2012	Updated:	May 31, 2012
Description:	From the Gentoo advisory: Multiple vulnerabilities have been discovered in Chromium and V8. A context-dependent attacker could entice a user to open a specially crafted web site or JavaScript program using Chromium or V8, possibly resulting in the execution of arbitrary code with the privileges of the process, or a Denial of Service condition.
Alerts:	Gentoo 201205-04 chromium 2012-05-27

Comments (none posted)

cobbler: remote code execution

Package(s):	cobbler	CVE #(s):	CVE-2012-2395
Created:	May 29, 2012	Updated:	July 3, 2012
Description:	From the Novell bugzilla: David Black has recently reported a remote code execution flaw in cobbler.
Alerts:	SUSE SUSE-SU-2012:0814-1 cobbler 2012-07-03 openSUSE openSUSE-SU-2012:0655-1 cobbler 2012-05-29

Comments (none posted)

dokuwiki: cross-site scripting/request forgery

Package(s):	dokuwiki	CVE #(s):	CVE-2012-2129 CVE-2012-2128
Created:	May 29, 2012	Updated:	August 13, 2012
Description:	From the Red Hat bugzilla: A cross-site scripting (XSS) and cross-site request forgery (CSRF) flaws were found in the way DokuWiki, a standards compliant, simple to use Wiki, performed sanitization of the 'target' parameter when preprocessing edit form data. A remote attacker could provide a specially-crafted URL, which once visited by a valid DokuWiki user would lead to arbitrary HTML or web script execution in the context of logged in DokuWiki user.
Alerts:	Mageia MGASA-2012-0207 dokuwiki 2012-08-12 Fedora FEDORA-2012-6630 dokuwiki 2012-06-12 Fedora FEDORA-2012-6628 dokuwiki 2012-05-27

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2012-2375
Created:	May 29, 2012	Updated:	December 19, 2012
Description:	From the Red Hat bugzilla: The fix for CVE-2011-4131 was not complete. Malicious NFS server could still crash the clients when returns more than 2 GETATTR bitmap words in response to the FATTR4_ACL attribute request.
Alerts:	Red Hat RHSA-2014:0284-01 kernel 2014-03-11 Scientific Linux SLSA-2013:1645-2 kernel 2013-12-16 Oracle ELSA-2013-1645 kernel 2013-11-26 Red Hat RHSA-2013:1645-02 kernel 2013-11-21 Red Hat RHSA-2013:0566-01 kernel-rt 2013-03-06 Oracle ELSA-2013-2507 kernel 2013-02-28 Oracle ELSA-2012-2047 kernel 2012-12-20 Oracle ELSA-2012-2047 kernel 2012-12-20 Oracle ELSA-2012-1580 kernel 2012-12-19 Scientific Linux SL-kern-20121219 kernel 2012-12-19 CentOS CESA-2012:1580 kernel 2012-12-19 Red Hat RHSA-2012:1580-01 kernel 2012-12-18 Ubuntu USN-1530-1 linux-ti-omap4 2012-08-10 Ubuntu USN-1499-1 linux-ti-omap4 2012-07-08 Ubuntu USN-1494-1 linux-ti-omap4 2012-07-02 Ubuntu USN-1490-1 linux-lts-backport-natty 2012-06-29 Ubuntu USN-1489-1 linux-lts-backport-oneiric 2012-06-29 Ubuntu USN-1488-1 linux 2012-06-29 Ubuntu USN-1487-1 linux 2012-06-29 Ubuntu USN-1486-1 linux 2012-06-29 SUSE SUSE-SU-2012:0789-1 Linux kernel 2012-06-26 Fedora FEDORA-2012-8359 kernel 2012-05-27

Comments (none posted)

kernel: privilege escalation

Package(s):	kernel	CVE #(s):	CVE-2012-2136
Created:	May 30, 2012	Updated:	November 5, 2012
Description:	From the Red Hat advisory: It was found that the data_len parameter of the sock_alloc_send_pskb() function in the Linux kernel's networking implementation was not validated before use. A local user with access to a TUN/TAP virtual interface could use this flaw to crash the system or, potentially, escalate their privileges. Note that unprivileged users cannot access TUN/TAP devices until the root user grants them access.
Alerts:	Oracle ELSA-2013-1645 kernel 2013-11-26 SUSE SUSE-SU-2012:1391-1 Linux kernel 2012-10-24 Ubuntu USN-1598-1 linux 2012-10-09 openSUSE openSUSE-SU-2012:1439-1 kernel 2012-11-05 Ubuntu USN-1529-1 linux 2012-08-10 Ubuntu USN-1539-1 linux-lts-backport-oneiric 2012-08-14 Ubuntu USN-1538-1 linux-lts-backport-natty 2012-08-14 Ubuntu USN-1535-1 linux 2012-08-10 Ubuntu USN-1533-1 linux 2012-08-10 Ubuntu USN-1531-1 linux 2012-08-10 Ubuntu USN-1530-1 linux-ti-omap4 2012-08-10 Ubuntu USN-1514-1 linux-ti-omap4 2012-08-10 Ubuntu USN-1534-1 linux-ec2 2012-08-10 Ubuntu USN-1532-1 linux-ti-omap4 2012-08-10 Red Hat RHSA-2012:1087-01 kernel 2012-07-17 openSUSE openSUSE-SU-2012:0812-1 kernel 2012-07-03 Oracle ELSA-2012-0862 kernel 2012-07-02 Oracle ELSA-2012-2022 kernel 2012-07-02 Oracle ELSA-2012-2022 kernel 2012-07-02 openSUSE openSUSE-SU-2012:0799-1 kernel 2012-06-28 SUSE SUSE-SU-2012:0789-1 Linux kernel 2012-06-26 Oracle ELSA-2012-2021 kernel 2012-06-23 Oracle ELSA-2012-2021 kernel 2012-06-23 openSUSE openSUSE-SU-2012:0781-1 kernel 2012-06-22 Oracle ELSA-2012-0743 kernel 2012-06-21 Oracle ELSA-2012-2020 kernel 2012-06-21 Oracle ELSA-2012-0690 kernel 2012-05-31 Red Hat RHSA-2012:0690-01 kernel 2012-05-29 Scientific Linux SL-kern-20120619 kernel 2012-06-19 CentOS CESA-2012:0743 kernel 2012-06-19 Red Hat RHSA-2012:0743-01 kernel 2012-06-18 Scientific Linux SL-kern-20120531 kernel 2012-05-31 CentOS CESA-2012:0690 kernel 2012-05-29

Comments (none posted)

mailman: information disclosure

Package(s):	mailman	CVE #(s):	CVE-2002-0389
Created:	May 29, 2012	Updated:	May 31, 2012
Description:	From the CVE entry: Pipermail in Mailman stores private mail messages with predictable filenames in a world-executable directory, which allows local users to read private mailing list archives.
Alerts:	Scientific Linux SLSA-2015:1417-1 mailman 2015-08-03 Oracle ELSA-2015-1417 mailman 2015-07-29 Red Hat RHSA-2015:1417-01 mailman 2015-07-22 openSUSE openSUSE-SU-2012:0660-1 mailman 2012-05-29

Comments (none posted)

net-snmp: denial of service

Package(s):	net-snmp	CVE #(s):	CVE-2012-2141
Created:	May 24, 2012	Updated:	April 8, 2013
Description:	From the Ubuntu advisory: It was discovered that Net-SNMP incorrectly performed entry lookups in the extension table. A remote attacker could send a specially crafted request and cause the SNMP server to crash, leading to a denial of service.
Alerts:	Gentoo 201409-02 net-snmp 2014-09-01 Mandriva MDVSA-2013:049 net-snmp 2013-04-05 CentOS CESA-2013:0124 net-snmp 2013-01-09 Oracle ELSA-2013-0124 net-snmp 2013-01-12 Fedora FEDORA-2012-16659 net-snmp 2012-10-31 Fedora FEDORA-2012-16662 net-snmp 2012-10-31 Scientific Linux SL-net--20130116 net-snmp 2013-01-16 CentOS CESA-2012:0876 net-snmp 2012-07-10 Scientific Linux SL-net--20120709 net-snmp 2012-07-09 Oracle ELSA-2012-0876 net-snmp 2012-07-02 Mageia MGASA-2012-0128 net-snmp 2012-06-27 Mandriva MDVSA-2012:099 net-snmp 2012-06-21 openSUSE openSUSE-SU-2012:0659-1 net-snmp 2012-05-29 Red Hat RHSA-2012:0876-04 net-snmp 2012-06-20 Ubuntu USN-1450-1 net-snmp 2012-05-23

Comments (none posted)

pidgin: multiple vulnerabilities

Package(s):	pidgin	CVE #(s):	CVE-2012-2214 CVE-2012-2318
Created:	May 29, 2012	Updated:	March 15, 2013
Description:	From the Mandriva advisory: Multiple vulnerabilities have been discovered and corrected in pidgin: A series of specially crafted file transfer requests can cause clients to reference invalid memory. The user must have accepted one of the file transfer requests (CVE-2012-2214). Incoming messages with certain characters or character encodings can cause clients to crash (CVE-2012-2318).
Alerts:	Oracle ELSA-2013-0646 pidgin 2013-03-14 Scientific Linux SL-pidg-20120719 pidgin 2012-07-19 Oracle ELSA-2012-1102 pidgin 2012-07-20 CentOS CESA-2012:1102 pidgin 2012-07-19 CentOS CESA-2012:1102 pidgin 2012-07-19 Red Hat RHSA-2012:1102-01 pidgin 2012-07-19 openSUSE openSUSE-SU-2012:0866-1 pidgin 2012-07-11 Ubuntu USN-1500-1 pidgin 2012-07-09 SUSE SUSE-SU-2012:0782-1 finch, libpurple and pidgin 2012-06-22 Fedora FEDORA-2012-8669 pidgin 2012-06-10 Fedora FEDORA-2012-8686 pidgin 2012-06-10 Mandriva MDVSA-2012:082 pidgin 2012-05-28 Fedora FEDORA-2012-8687 pidgin 2012-06-03

Comments (none posted)

python: insecure file creation

Package(s):	python	CVE #(s):	CVE-2011-4944
Created:	May 30, 2012	Updated:	October 18, 2012
Description:	From the Novell bugzilla: python distutils first creates ~/.pypirc and then calls chmod() to restrict permissions. This allows for a time window where the file is readable by others.
Alerts:	Mandriva MDVSA-2013:117 python 2013-04-10 Ubuntu USN-1615-1 python3.2 2012-10-23 Ubuntu USN-1616-1 python3.1 2012-10-24 Ubuntu USN-1613-1 python2.5 2012-10-17 Ubuntu USN-1613-2 python2.4 2012-10-17 Ubuntu USN-1596-1 python2.6 2012-10-04 Ubuntu USN-1592-1 python2.7 2012-10-02 Mageia MGASA-2012-0170 python 2012-07-19 Mageia MGASA-2012-0169 python 2012-07-19 Mandriva MDVSA-2012:096-1 python 2012-07-02 Mandriva MDVSA-2012:096 python 2012-06-20 Mandriva MDVSA-2012:097 python 2012-06-20 openSUSE openSUSE-SU-2012:0667-1 python 2012-05-30 CentOS CESA-2012:0744 python 2012-06-18 Scientific Linux SL-pyth-20120618 python 2012-06-18 CentOS CESA-2012:0745 python 2012-06-18 Red Hat RHSA-2012:0745-01 python 2012-06-18 Red Hat RHSA-2012:0744-01 python 2012-06-18 Oracle ELSA-2012-0745 python 2012-06-19 Oracle ELSA-2012-0744 python 2012-06-19 Scientific Linux SL-pyth-20120618 python 2012-06-18

Comments (none posted)

python-tornado: HTTP header injection

Package(s):	python-tornado	CVE #(s):	CVE-2012-2374
Created:	May 29, 2012	Updated:	June 18, 2012
Description:	From the CVE entry: CRLF injection vulnerability in the tornado.web.RequestHandler.set_header function in Tornado before 2.2.1 allows remote attackers to inject arbitrary HTTP headers and conduct HTTP response splitting attacks via crafted input.
Alerts:	Fedora FEDORA-2012-8194 python-tornado 2012-05-29 openSUSE openSUSE-SU-2012:0755-1 python-tornado 2012-06-18 Fedora FEDORA-2012-8205 python-tornado 2012-05-29 Fedora FEDORA-2012-8217 python-tornado 2012-05-29

Comments (none posted)

request-tracker3.8: multiple vulnerabilities

Package(s):	request-tracker3.8	CVE #(s):	CVE-2011-2082 CVE-2011-2083 CVE-2011-2084 CVE-2011-2085 CVE-2011-4458 CVE-2011-4459 CVE-2011-4460
Created:	May 25, 2012	Updated:	September 17, 2012
Description:	From the Debian advisory: CVE-2011-2082: The vulnerable-passwords scripts introduced for CVE-2011-0009 failed to correct the password hashes of disabled users. CVE-2011-2083: Several cross-site scripting issues have been discovered. CVE-2011-2084: Password hashes could be disclosed by privileged users. CVE-2011-2085: Several cross-site request forgery vulnerabilities have been found. If this update breaks your setup, you can restore the old behaviour by setting $RestrictReferrer to 0. CVE-2011-4458: The code to support variable envelope return paths allowed the execution of arbitrary code. CVE-2011-4459: Disabled groups were not fully accounted as disabled. CVE-2011-4460: SQL injection vulnerability, only exploitable by privileged users.
Alerts:	Debian DSA-2480-4 request-tracker3.8 2012-09-15 Debian DSA-2480-3 request-tracker3.8 2012-06-07 Fedora FEDORA-2012-8339 rt3 2012-06-02 Fedora FEDORA-2012-8363 rt3 2012-06-02 Fedora FEDORA-2012-8290 rt3 2012-06-01 Debian DSA-2480-1 request-tracker3.8 2012-05-24 Debian DSA-2480-2 request-tracker3.8 2012-05-29

Comments (none posted)

strongswan: authentication bypass

Package(s):	strongswan	CVE #(s):	CVE-2012-2388
Created:	May 31, 2012	Updated:	April 30, 2013
Description:	From the Debian advisory: An authentication bypass issue was discovered by the Codenomicon CROSS project in strongSwan, an IPsec-based VPN solution. When using RSA-based setups, a missing check in the gmp plugin could allow an attacker presenting a forged signature to successfully authenticate against a strongSwan responder.
Alerts:	Debian DSA-2483-1 strongswan 2012-05-31 Fedora FEDORA-2012-8821 strongswan 2012-06-10 Fedora FEDORA-2012-8815 strongswan 2012-06-10 openSUSE openSUSE-SU-2012:0691-1 strongswan 2012-06-04

Comments (none posted)

wireshark: denial of service

Package(s):	wireshark	CVE #(s):	CVE-2012-2392 CVE-2012-2393 CVE-2012-2394
Created:	May 29, 2012	Updated:	July 11, 2012
Description:	From the openSUSE advisory: This update is a maintenance release of Wireshark. It fixes some vulnerabilities when dissecting certain protocols. As packages for these protocols may be received over the network, an attacker may trigger infinite or large loops or crashes of the dissector.
Alerts:	Scientific Linux SLSA-2013:1569-2 wireshark 2013-12-09 Oracle ELSA-2013-1569 wireshark 2013-11-26 Red Hat RHSA-2013:1569-02 wireshark 2013-11-21 Mandriva MDVSA-2013:055 wireshark 2013-04-05 Mageia MGASA-2012-0206 wireshark 2012-08-12 Fedora FEDORA-2012-10175 wireshark 2012-07-10 Mageia MGASA-2012-0134 wireshark 2012-06-27 openSUSE openSUSE-SU-2012:0657-1 wireshark 2012-05-29

Comments (none posted)

xinetd: service disclosure flaw

Package(s):	xinetd	CVE #(s):	CVE-2012-0862
Created:	May 29, 2012	Updated:	October 9, 2013
Description:	From the Red Hat bugzilla: Thomas Swan reported a service disclosure flaw in xinetd. xinetd allows for services to be configured with the TCPMUX or TCPMUXPLUS service types, which makes those services available on port 1, as per RFC 1078 [1], if the tcpmux-server service is enabled. When the tcpmux-server service is enabled, xinetd would expose _all_ enabled services via the tcpmux port, instead of just the configured service(s). This could allow a remote attacker to bypass firewall restrictions and access services via the tcpmux port.
Alerts:	openSUSE openSUSE-SU-2014:0517-1 xinetd 2014-04-11 openSUSE openSUSE-SU-2014:0494-1 xinetd 2014-04-08 Scientific Linux SLSA-2013:1302-1 xinetd 2013-10-09 Oracle ELSA-2013-1302 xinetd 2013-10-02 Red Hat RHSA-2013:1302-01 xinetd 2013-09-30 CentOS CESA-2013:0499 xinetd 2013-03-09 Scientific Linux SL-xine-20130228 xinetd 2013-02-28 Mandriva MDVSA-2013:057 xinetd 2013-04-08 Oracle ELSA-2013-0499 xinetd 2013-02-25 Red Hat RHSA-2013:0499-02 xinetd 2013-02-21 Mandriva MDVSA-2012:155-1 xinetd 2012-10-02 Mandriva MDVSA-2012:155 xinetd 2012-09-28 Fedora FEDORA-2012-8041 xinetd 2012-05-29 Fedora FEDORA-2012-8061 xinetd 2012-05-29

Comments (none posted)

Kernel release status

The 3.5 merge window remains open as of this writing, so there is no current development kernel. The flow of patches into the mainline continues; see the separate article below for a summary of what has been merged thus far.

Stable updates: there have been no stable updates released in the last week. The 3.0.33, 3.2.19, 3.3.8, and 3.4.1 updates are all in the review process as of this writing; they could all be released at any time.

Comments (none posted)

Quotes of the week

— Mel Gorman

— Jens Axboe

— Alan Cox

Comments (none posted)

klibc 2.0 released

Version 2.0 of the minimal C library klibc has been released. "A bit delayed due to kernel.org breakin, but development is kicking in again. The 2.0 branch saw boot time tests and deployments in Debian, so we are quite certain it should work for the most out of you, if not please let us know." The biggest change appears to be proper support for buffered I/O in the stdio functions.

Full Story (comments: 1)

3.5 merge window part 2

The 3.5 merge window continued in full force after last week's summary, with another 4,000 non-merge changesets pulled into the mainline since then; there have now been over 8,600 changes merged for 3.5, and the merge window is not done yet. The most significant user-visible changes are:

• The autosleep patch set, implementing Android-style opportunistic suspend (with a different API) has been merged. Associated with this work is a new epoll flag (EPOLLWAKEUP) which causes a wakeup event to be activated, preventing suspend when an event is available for processing.

• The user namespace rework gets the kernel closer to being able to safely run processes as root within a container.

• RAID10 arrays managed by the MD layer can now be reshaped.

• After years of attempts, the uprobes subsystem has been merged. See this article for more information on the version of uprobes that was merged for 3.5.

• The tmpfs filesystem now supports hole punching and the SEEK_DATA and SEEK_HOLE lseek() options.

• The removal of old code continues; victims include Microchannel bus support, legacy CRIS RTC drivers, the imxmmc driver, the swap token code, and the lumpy reclaim mechanism.

• New drivers include:
  • Systems and processors: Renesas SH7264, SH7269, and SH7734 processors, ST SPEAr13xx processors, Atheros DB120 reference boards, and Lantiq FALCON processors.

  • Audio: Freescale MC13783 codecs, Cirrus Logic CS42L52 low power stereo codecs, LAPIS Semiconductor ML26124 codecs, TI LM49453 codecs, and ST Ericsson Ux500-based audio platform devices.

  • Block: Cirrus Logic EP93xx PATA controllers.

  • Graphics: Aspeed Technologies AST 2000, 2100, 2200, 2150 and 2300 chips, MGA G200 server engines, and QEMU-emulated Cirrus GPUs.

  • Input: I2C-based Wacom tablets, National Semiconductor LM8333 keypad controllers, Dialog DA9052/DA9053 touchscreen controllers, and Synaptics NavPoint touchpads on PXA27x SSP ports.

  • Miscellaneous: STA2X11 "ConneXt" I/O hubs, Power 7+ Nest crypto accelerators, Texas Instruments INA219 and INA226 power management chips, Intel Atom E6xx watchdogs, Intel MSIC mixed signal gpio controllers, RICOH RC5T583 GPIO controllers, Samsung Exynos I/O memory management units, and Dialog DA9052 watchdogs.

  • Multi-function chipsets: Maxim Semiconductor MAX77693 PMICs, Intel ICH LPC bridges, ST Microelectronics ConneXt (STA2X11) I/O hubs, and National Semiconductor / TI LM3533 lighting power chips.

  • Network: NXP PN544 NFC controllers.

  • Video4Linux: Infineon TUA 9001 silicon tuners, Afatech AF9033 DVB-T demodulators, Afatech AF9035 based DVB USB receivers, Fitipower FC0011 silicon tuners, LG Electronics LG216x-based ATSC-MH demodulators, Fitipower FC0012 and FC0013 silicon tuners, STA2x11 video input ports, and SMIA++/SMIA-compliant sensors.

Changes visible to kernel developers include:

• The kernel's exception table can now be sorted at build time, speeding the boot process somewhat.

• The ALSA core now supports "dynamic PCM" devices, being audio devices split into front and back ends which allow arbitrary routing of audio data between the front and back end devices.

• The contiguous memory allocator patch set, designed to make life easier on systems where large chunks of physically-contiguous memory are needed on occasion, has been merged at last. The same pull included a complete rework of the ARM DMA mapping subsystem, adding CMA support and support for I/O memory management units.

• The DMA buffer sharing subsystem has gained support for mapping buffers into user space. Also added is a new dma_buf_vmap() function for mapping buffers (using the vmalloc() area) into kernel space.

• <asm/word-at-a-time.h> has been significantly reworked (by Linus) to be more efficient on all architectures; strnlen_user() has then been rewritten to use it in a generic manner.

• The LED subsystem now supports one-shot timed operation; see ledtrig-transient.txt for details.

• The error detection and correction (EDAC) subsystem has been massively reworked to better handle contemporary processors and memory controllers.

As of this writing, the 3.5 merge window has a few more days left to run. The final article in this series will come out once the merge window has closed.

Comments (1 posted)

Uprobes in 3.5

Uprobes is a kernel patch with a long story and many contentious discussions behind it. This code has its roots in utrace, a user-space tracing and debugging API that was first covered here in early 2007. Utrace ran into various types of opposition (only partly related to its own origin in SystemTap) and has never been merged, but a piece of it lives on in the form of uprobes, which is charged with the placement of probes into user-space code. After several mailing-list rounds of its own, uprobes was finally merged for the 3.5 kernel development cycle. Just how this facility will be used remains to be seen, however.

At the core of uprobes is this function:

    #include <linux/uprobes.h>

    int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc);

The inode structure represents an executable file; the probe is to be placed at offset bytes from the beginning. The uprobe_consumer structure tells the kernel what is to be done when a process encounters the probe; it looks like:

    struct uprobe_consumer {
	int (*handler) (struct uprobe_consumer *self, struct pt_regs *regs);
	bool (*filter) (struct uprobe_consumer *self, struct task_struct *task);
	struct uprobe_consumer *next;
    };

The filter() function is optional; if it exists, it determines whether handler() is called for each specific hit on the probe. The handler returns an int, but the return value is ignored in the current code.

Since probes are associated with files, they affect all processes that run code from those files. A special copy is made of the page to contain the probe; in that copy, the instruction at the specified offset is copied and replaced by a breakpoint. When the breakpoint is hit by a running process, filter() will be called if present, and handler() will be run unless the filter said otherwise. Then the displaced instruction is executed (using the "execute out of line" mechanism described in this article) and control returns to the instruction following the breakpoint.

Uprobes thus implements a mechanism by which a kernel function can be invoked whenever a process executes a specific instruction location. One could imagine a number of things that said kernel function could do; there has been talk, for example, of using uprobes (and, perhaps someday, something derived from utrace) as a replacement for the much-maligned ptrace() system call. Tools like GDB could place breakpoints with uprobes; it might even be possible to load simple filters for conditional breakpoints into the kernel, speeding their execution considerably. Uprobes could also someday be a component of a Dtrace-like dynamic tracing functionality. For now, though, the interfaces for that kind of feature have not been added to the kernel; none have even been proposed.

What the current implementation does have is integration with the perf events subsystem. New dynamic "events" can be added to any file location via an interface similar to that used for dynamic kernel tracepoints. In particular, there is a new file called uprobe_events in the tracing directory (/sys/kernel/debug/tracing/ on most systems) that is used to add and remove events. As an example, a line like:

    echo 'p:bashme /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events

would place a new event (called "bashme") at location 0x4245c0 in the bash executable. The event would then appear with all other events in /sys/kernel/debug/tracing/events, in the uprobes subdirectory. Like other events, it is not actually turned on until its enabled attribute is set. See Documentation/trace/uprobetracer.txt for details on the interface at this level.

Placing uprobes is, by default, a privileged operation requiring the CAP_SYS_ADMIN capability. One can remove the privilege requirement by setting the perf_paranoid sysctl knob to -1, but doing so will allow the placement of dynamic tracepoints anywhere in the system, in kernel or user space. Thus, one need not be overly paranoid to leave perf_paranoid at its default setting.

The perf tool has been enhanced to make working with dynamic user-space tracepoints easy. One can, for example, set a tracepoint at the entry to the C library's malloc() implementation with:

    perf probe -x /lib64/libc.so.6 malloc

That tracepoint can then be treated like any other event understood by perf. See the explanatory text from Ingo Molnar's pull request for examples of what can be done.

Most kernel patches are conceived, implemented, reviewed, and merged into the mainline over a fairly short period of time. But some of them seem to languish for years without making much progress. Uprobes was such a patch set. It must have been frustrating for the developers to keep revising and posting this code, only to see it shot down over and over again. But the kernel community can be supportive of developers who show both persistence and a willingness to listen to criticism. The result, in this case, is a user-space probing mechanism that has been simplified, made more robust, and integrated into the existing events infrastructure. Hopefully it was worth the wait.

Comments (6 posted)

Atime and btrfs: a bad combination?

Unix and Unix-like systems have traditionally recorded the time of last access for each file in the system. This practice has fallen partially out of favor over the last decade for a simple reason: writing the last-accessed time ("atime") takes up a lot of I/O bandwidth when lots of files are being read; see this article from 2007, for example. The worst of the atime-related problems have long since been mitigated by moving to the "relatime" mount option by default; relatime only updates atime a maximum of once per day for unchanging files. But now it seems that atime recording can be especially problematic with the btrfs filesystem, and relatime may not help much.

One of the core design features of btrfs is its copy-on-write nature. Blocks on disk are never modified in place; instead, when it becomes necessary to commit a change, the affected block is copied and rewritten into a new location. Copy-on-write applies to metadata as well as data; if a file's metadata (such as its last-accessed time) is changed, the block containing that metadata will be copied to a new spot. So, on btrfs, an operation that reads a lot of files (creating a tar archive, say, or a recursive grep) can, through atime updates, cause the copying and rewriting of a lot of metadata blocks.

Needless to say, performance is not improved this way, but that is not where the big problem comes in. As Alexander Block pointed out, the real problem has to do with the interaction between atime, copy-on-write, and snapshots.

Btrfs provides a fast snapshotting feature that can create a copy of the state of the filesystem at a specific time. When a snapshot is created, it shares all data and metadata with the "trunk" filesystem. Should a file be changed, the resulting copy-on-write operation separates the trunk from the snapshot, keeping both versions of the data available. So snapshots can be thought of as being nearly free as long as the filesystem remains relatively quiet. Snapshots will share data and metadata, so they do not require a lot of additional space.

Atime updates change the situation, though. If somebody takes a snapshot of a filesystem, then performs a recursive grep on that filesystem, the last-access time of every file touched may be updated. That, in turn, can cause copy-on-write operations on each file's inode structure, with the result that many or all of the inodes in the filesystem may be duplicated. That can increase the space consumption of the filesystem considerably; Alexander posted an example where a recursive grep caused 2.2GB of free space to disappear. That is a surprising result for what is meant to be a read-only operation.

Once upon a time, when disk capacities were measured in megabytes, it was said that the only standard feature of Unix systems was the message of the day telling users to clean up their files. Atime was often used by harried system administrators trying to recover some disk space; they would scan for infrequently-accessed files and, depending on how desperate the situation was and how powerful their users were, either send lists of unused files to users or simply back those files up to tape and delete them. It is somewhat ironic that a feature meant (among other things) to help keep disk space free has now, on next-generation filesystems, become part of the problem.

It's worth noting that the relatime option (which only updates atime once per day unless the file has been modified since the last atime update) is of little help here. It only takes one atime update to force an inode to be rewritten and unshared with any snapshots. So the fact that such updates are limited to one per day offers little in the way of consolation.

Users are also unlikely to be consoled by one other aspect of the problem pointed out by Alexander: since reading data can consume space in the filesystem, read operations might fail with "no space available" errors on an overflowing filesystem. That may make it difficult or impossible to fix the problem by copying data out of a full filesystem. By the time that happens, a typical user could be forgiven for thinking that, perhaps, they don't need last-accessed time tracking at all.

Along those lines, Alexander suggested that it might be a good idea to default to "noatime" (which turns off atime recording entirely) for btrfs mounts, even if that means that btrfs would then behave differently than other filesystems. That idea was quickly shot down for a simple reason: there are still applications that actually need the atime information to function correctly. The classic example is the mutt email client which, in the absence of atime, cannot tell whether a mailbox contains unread mail. Programs that clean up temporary directories (tmpreaper or tmpwatch, for example) will fail without atime. There are also hierarchical storage systems that, like the Unix system administrator of old, use atime to determine when to move files to slower storage. So atime needs to stick around, lest users run into a different kind of unpleasant surprise.

For now, the only recourse for users who run into (or are worried about) this problem is to explicitly mount their filesystems with the "noatime" option. Further ahead, it might be possible to make some tweaks to btrfs to mitigate the problem; Boaz Harrosh suggested disabling atime updates when the free space falls below a certain threshold or moving the atime data into a separate data structure. But nobody appears to be working on such solutions now. So it may be that, as usage of btrfs grows, users will occasionally be surprised that reading a file can consume space in their filesystems.

Comments (50 posted)

Temporary files: RAM or disk?

Temporary files on Linux have traditionally been written to /tmp, at least those that don't need to persist across boots. Several Linux distributions are now planning to mount /tmp as a RAM-based tmpfs by default, which should generally be an improvement in a wide variety of scenarios—but not all. Debian has been planning to make that switch for the upcoming "wheezy" (7.0) release, but as a discussion on debian-devel shows, not all are happy with that decision.

Mounting /tmp on tmpfs puts all of the temporary files in RAM. That will reduce the amount of disk I/O that needs to be done, as the filesystem never actually touches the disk unless there is memory pressure. In that case, the tmpfs memory could get swapped out like other pages in the system, but in many cases a temporary file will be created without needing any disk I/O. The installer (or system administrator) can specify a maximum size for the filesystem as part of the mount options, but the memory is only actually used if files are written to it.

The latest incarnation of the dispute over putting /tmp into RAM (vs. having it as a disk directory in / or its own partition) was started by a posting from "Serge" that claimed the change was "useless". His complaint stemmed from various applications that put large files into /tmp (long videos, ISO images, unpacked archives, sort temporary files, etc.), which can cause problems for systems with little memory or with too small of a tmpfs. He argued that putting /tmp on tmpfs by default was the wrong choice for new users, and that savvy users could make the switch (or know enough to choose it at install time).

It is clear that there is a difference of opinion among participants in the thread about how /tmp should be used. Several people thought that /var/tmp should be used for large temporary files, with /tmp reserved for small files. Others pointed out that the file hierarchy standard (FHS) just says that /tmp is for files that do not need to persist across reboots, while /var/tmp is for those that do. But using /var/tmp for non-persistent large files has a drawback: those files do not get cleaned up on boot, unlike /tmp files—at least on Debian.

Many thread posters have been running their systems with a RAM-based /tmp for a long time with few or no issues, while others are clearly running into problems in that configuration. Large videos downloaded via the Flash plugin seem to be a particular problem, but there are others. It really comes down to a question of what /tmp is for.

Many in the long thread—something of a Debian tradition—believe that writing large files to /tmp is the wrong thing for an application to do. But no real alternative location that preserves the "wipe on reboot" semantics for /tmp has been offered. Workarounds like the TMPDIR environment variable can be used, but whatever directory that points to may just fill up with garbage over time.

Running out of /tmp space is a problem regardless of what kind of filesystem lies under it, but in the default Debian installation a disk-based /tmp will essentially have the whole disk available, as it only creates a single / partition. On the other hand, a RAM-based /tmp will almost certainly be much smaller, which can lead to applications filling up the filesystem much more easily. Even if those applications are "wrong" to do so, they evidently exist, so forcing users to confront that problem at times could be sub-optimal. More advanced users can make their own choice.

There were numerous invocations of Solaris in the thread as well, because it has used a RAM-based temporary filesystem for many years, seemingly without many problems. Russ Allbery said that he started in the "Solaris has been doing this for years, what's the problem?" camp, but after reading some of the objections in the thread had basically changed his mind. It comes down to a question of functionality vs. speed. For a default setting, working should be preferred over speed optimization.

As Joey Hess pointed out, until there are clear numbers about the performance improvement that comes from a RAM-based /tmp, making it the default is a premature optimization. There were examples offered where tmpfs made an enormous performance difference, but the question is not whether the feature is useful; it is, instead, whether it should be the default.

This is not the first time the issue has come up. Roger Leigh posted pointers to other threads and bug reports (some of which have their own long threads, of course). He is the developer that added the tmpfs-based /tmp for wheezy, but he mostly stayed clear of the discussion this time. It does not look like he is inclined to remove the default, though, so there have been suggestions that the issue be referred to the technical committee.

It is interesting to note that Fedora plans to move to a RAM-based /tmp for Fedora 18, and has already enabled it for Rawhide. The Fedora feature page notes that Solaris has been doing so since 1994 and that Arch currently defaults to a tmpfs-based /tmp. It also mentions that Debian and Ubuntu both plan to move in that direction.

At least for the near future—and probably beyond—RAM sizes will generally be far less than those of disks, particularly for machines targeted at non-technical users. While those users might benefit from the performance improvements that come from keeping /tmp in RAM, there is a risk that their applications will chew up their RAM with huge temporary files, leading to swap storms or broken applications.

The Solaris example may not be all that compelling as it is not really a consumer-oriented desktop system. The other Linux distributions may offer a better test case and Debian may well get the benefit of seeing what happens with them before wheezy is completely frozen. On the other hand, though, it is not really clear that Debian targets (or attracts) many novice users, so why make the vast majority of Debian users suffer the penalty of disk-based /tmp when they don't really need to? They can certainly switch to that, but the default might serve the majority quite well. At this point, one suspects that RAM-based /tmp will soon be the norm and that applications will get "fixed", but only time will tell.

Comments (119 posted)

Distribution quotes of the week

-- Seth Vidal

-- Wookey

Comments (3 posted)

Fedora 17 released

The Fedora 17 release is out. "Frankly, we believe this is the beefiest release ever -- chock full of condiments, more commonly known as Features, to customize your experience to your tastes. We take pride in our toppings, and in our fine ingredients; Fedora 17 includes both over- and under-the-bun improvements that show off the power and flexibility of the advancing state of free (range) software." Details can be found in the Fedora 17 feature list.

Full Story (comments: 24)

Fedora 17 ARM Beta Release

A Fedora 17 beta for ARM is now available. There are a number of images provided for various targets ("QEMU, Trimslice, Beagleboard XM and iMX based hardware platforms.") "We invite you to take part in making Fedora 17 for ARM a solid release by downloading, testing, and providing your valuable feedback. Please join us on the IRC in #fedora-arm on Freenode or send feedback and comments to the ARM mailing list."

Full Story (comments: 3)

Fedora 15 end of life

Now that Fedora 17 is out, Fedora 15 is approaching its end-of-life. There will be no more updates for Fedora 15 after June 26.

Full Story (comments: none)

bits from the NM process: advocacy, no more AM reports, AMs needed

Enrico Zini reports on some changes in the Debian New Maintainer process. "AM (Application Manager) reports were a bit of a burden of bureaucracy, and it has been the case that applications got stuck for considerable time waiting for the AM to have the time to write the report. Let us fix that: forget about writing the AM reports!"

Full Story (comments: none)

Cooperative Bug Isolation for Fedora 17

The Cooperative Bug Isolation Project (CBI) is now available for Fedora 17. The CBI is a research effort designed to find out what went wrong when your application crashes. "We currently offer instrumented versions of Evolution, The GIMP, GNOME Panel, Gnumeric, Liferea, Nautilus, Pidgin, and Rhythmbox."

Full Story (comments: none)

Distribution newsletters

• Debian Project News (May 28)
• DistroWatch Weekly, Issue 458 (May 28)
• Maemo Weekly News (May 28)
• Ubuntu Weekly Newsletter (May 27)

Comments (none posted)

Precision and purpose: Ubuntu 12.04 and the Unity HUD reviewed (ars technica)

Ryan Paul takes a look at Unity and the Heads-Up Dispaly (HUD) in the 12.04 release. "The HUD is powered by the same mechanism that supports Unity’s global menu system. In order to make an application’s menus available to Unity so that it can display it in the panel, the contents of the menu are relayed over the D-Bus interprocess communication system. The HUD taps into that data, essentially offering an alternate presentation of the menu. In addition to being extremely useful, the HUD is a compelling illustration of how Unity’s architectural strengths can be put to use in novel ways."

Comments (1 posted)

What's new in Fedora 17 (The H)

The H has a nice summary of the new features in Fedora 17. "Of the many changes made, the two that stand out are software rendering for GNOME Shell and the sandbox function for isolating applications. These, and many other changes, are likely to find their way into other distributions soon. Time will tell whether that will also be the case for the much-discussed filesystem reorganisation."

Comments (8 posted)

SUSE Turns 20, Ascends to the Cloud (Linux.com)

Linux.com has an interview with SUSE's Vice-President of Engineering Ralf Flaxa. "Linux.com: What’s the significance of 20 years in business? Ralf Flaxa: The significance comes in our longevity, which even surprises us to a degree. In fact, our first reaction was “Hey, 20 years!” There have been many other distributions that have come and gone – we were one of the first and we’re still here. We view the staying power as very significant."

Comments (2 posted)

Clustering, development, and galactic conquest at PGCon 2012

Every year, the PostgreSQL community has a developer meeting in Ottawa called PGCon. This year's PGCon was the largest yet, with 210 contributors and users attending. As usual, though, the most interesting developments happened outside the regular conference sessions. This year's events included new replication and clustering projects, changes to the PostgreSQL development process, and a battle for galactic conquest.

PostgresXC 1.0 beta

There were two major announcements made at the one-day Postgres Clustering Summit before the main conference. The PostgresXC project announced its 1.0 beta release, and 2ndQuadrant announced its plans for a new replication method for PostgreSQL. The Clustering Summit, a meeting of developers working on replication and database clustering for PostgreSQL, was sponsored by NTT Open Source of Japan.

PostgresXC is a project to build a write-scalable cluster for PostgreSQL. A PostgresXC cluster is intended to be a co-located group of four to twenty servers, and is designed to be able to handle larger numbers of database connections and concurrent write queries than a single server. The goals of the project are to build something which satisfies a lot of the same use cases as Oracle Real Application Clusters (RAC), Oracle's database clustering technology. PostgresXC implements a shared-nothing architecture instead of RAC's shared-storage architecture in order to survive a larger variety of hardware failure situations.

As of the 1.0 beta release, PostgresXC supports most PostgreSQL syntax and functionality for a small cluster of database servers so you can theoretically deploy most PostgreSQL-based applications on PostgresXC. Notable limitations with 1.0 beta are the lack of triggers and savepoints, as well as the inability to add new nodes without rebuilding the cluster. These issues will be addressed in 1.1 and later versions.

One of the primary differences between regular PostgreSQL and PostgresXC is the notion of distributed tables. Each table you create can be distributed across the cluster based on round-robin, a hash, or a column value, instead of being replicated to all cluster nodes. This allows the database cluster to support more data than any single node can.

The PostgresXC project primarily consists of developers from NTT Open Source and EnterpriseDB, and it is licensed under The PostgreSQL license. Once PostgresXC is more stable, expect rapid adoption due to the needs of users for bigger, more fault-tolerant database servers.

New "logical" replication project

The second replication announcement was 2ndQuadrant's plan to create a new replication technology for PostgreSQL, which they call "Bi-Directional Replication", mostly to distinguish it from other forms of replication. They have secured sponsorships from companies like McAfee, have begun work on the patches, and plan to get the feature done for PostgreSQL 9.3 in 2013. 2ndQuadrant is a PostgreSQL consulting company headquartered in the United Kingdom, lead by committer Simon Riggs.

At an evening pizza session, they explained how the new replication will work. PostgreSQL's streaming binary replication, introduced in version 9.0, allows a replication server to receive copies of data pages over a normal PostgreSQL database connection (usually port 5432). This connection is called the "walsender" and its associated listener process on the replica is called the "walreceiver" (WAL stands for Write-Ahead Log). This works well for whole-database replication, but does not permit users to replicate only specific parts of the database, or to run any kind of write queries on the replicas. As a result, other types of replication are also needed.

The new feature will allow clients to choose to receive either replicated query statements or changed rows over the same walsender connection, so that listeners can filter or modify the replication stream. More importantly, it would allow a replica to create database changes as well as receiving them, provided that conflicts were handled by database or middleware code, thus implementing one form of multi-master replication for PostgreSQL.

This type of replication configuration is already possible with replication tools such as Slony-I, Londiste, and Bucardo. However, all of those tools suffer from administration and performance overhead which many users find unacceptable because they are implemented in Postgres user space instead of in the Postgres backend. Bi-Directional Replication is expected to be more efficient, and thus useful to a wider array of users. It may even become the underlying data transport mechanism for future versions of the older replication systems.

Bi-Directional Replication is also intended to solve a different use-case than PostgresXC. While PostgresXC is designed to provide scalability and consistency for database writes within a single data center, the new replication is intended to support asynchronous, geographically distributed replication, at the expense of consistency. Such "eventual consistency" systems have become popular among web startups recently, and PostgreSQL users are no exception.

CommitFests and the PostgreSQL development process

One of the other meetings at PGCon was the annual Developer Meeting, at which 20 of the largest code contributors to PostgreSQL meet in person and discuss current and future development of the database. At the Developer Meeting three years ago, the PostgreSQL project adopted the CommitFest model for managing patch review and annual development. At this one, the group reviewed how the CommitFest process was going, and decided to make some minor changes to try to improve it.

The central idea of the CommitFest model is to completely clear out the queue of pending patches every two months, in order to prevent catastrophic levels of unreviewed patch buildup. CommitFests have allowed the project to accept a large number of patch submissions and make sure that everything submitted got a serious review. At the developer meeting, contributor Noah Misch said that the guarantee of code review was one of the things which attracted him to the PostgreSQL project in the first place.

Unfortunately, over the last three releases, the last of the four CommitFests have been getting longer: one month for 9.0, two months for 9.1, and three for 9.2. Contributing to this problem is that the number of patch reviewers has not been growing as fast as the number of submissions, leading to a risk of reviewer burn-out.

The main reason the final CommitFest does not finish on schedule has been large, complex, not-quite-ready patches which need extensive review by core committers. In order to address this, PostgreSQL will add a planning period after the third, or penultimate, CommitFest in order to separate pending patches into large or small, and ready for committers or not quite ready. Then, in the third week of the final CommitFest, the team will be able to do a final triage of what's ready for release and what's not, and conclude the CommitFest after the expected 30 days.

Another cause of contributor pain has been submitting new features without an agreed specification, often causing those patches to need complete recoding. In order to add a more formal process around specification review, the project will be adding a "design specification" section to the CommitFests and encouraging people to submit specifications for review. The developers also adopted a "one patch, one review" policy, which means that patch contributors are obligated to review at least one patch contributed by someone else.

Conquering the galaxy with PostgreSQL

As a new event this year, PGCon included a Schemaverse tournament. Schemaverse is a text-mode galactic conquest wargame, played entirely in a PostgreSQL database. Users issue commands to build and move spaceships, engage in battles, and conquer and mine planets, all in the form of SQL queries. You win the game either by conquering the most planets, or by successfully hacking the database.

Schemaverse was invented by Josh McDougall as a fun data security exercise for the DEFCON 2011 security conference. Notably, no player at DEFCON, PGCon, nor in the open online version has yet managed to win by hacking the PostgreSQL database. The game also demonstrates how PostgreSQL allows sophisticated application functionality to be built either inside or outside the database.

The PGCon tournament, which ran for the entirety of the conference, was won by Chris Browne of the domain registrar Afilias. Chris spent weeks leading up to PGCon honing automated "artificial intelligence" scripts to control his fleet of spaceships, and rapidly seized over 500 planets. He won several prizes including a championship sweatshirt.

Other conference highlights

Peter van Hardenberg, database lead at platform-as-a-service company Heroku, delivered the keynote for PGCon. His talk was about Heroku's users, the cloud, NoSQL, and the future of PostgreSQL. First he discussed the large number of PostgreSQL databases (1.6 million) belonging to Heroku's hundreds of thousands of users, who are mostly small-team agile-development web application developers. Van Hardenberg appealed to the PostgreSQL developers to keep Heroku's users in mind when working on PostgreSQL and new features, particularly emphasizing the need for usability improvements.

In case studies at the conference NASA scientists talked about using Postgres for data collection on the Aura spacecraft for ozone monitoring. 2ndQuadrant staff presented migrating top website Fotolog from MySQL to PostgreSQL. A contributor to open source library management system Evergreen demonstrated their application and explored its data architecture. And Mozilla staff talked about the Socorro crash collection system.

Other talks focused on many of the features and internals of PostgreSQL 9.2, currently in beta. This included Range Types, the new replication protocol, and the internals of how many of the performance improvements were implemented. As they did last year, PostgreSQL's Russian development team introduced another new indexing idea, this time for similarity searching.

PGCon 2012 was a busy, intense week for PostgreSQL developers. Users can expect a lot more great things coming out of the project because of it. If your job centers around PostgreSQL, you should consider attending next year. Later this year, you can attend Postgres Open, a business and user-oriented conference in Chicago, or PostgreSQL Europe in Prague.

Comments (12 posted)

Quote of the week

— Aaron Seigo

Comments (3 posted)

GCC Explorer - an interactive take on compilation

Matt Godbolt announces GCC explorer, a web-based tool for exploring how code tweaks change the machine code emitted by the compiler. "Particularly with some of the newer features of C++11 — lambdas, move constructors, threading primitives etc — it’s nice to be able to see how your elegant code becomes beautiful (and maybe even fairly optimal) machine code." The GCC explorer code is on github for those who want to set up their own instance.

Comments (88 posted)

LibreOffice 3.5.4 released

The LibreOffice 3.5.4 release is out; this is a minor update but it may be of interest to those interested in better performance: "LibreOffice 3.5.4 is the fastest version of the best free office suite ever, with up to 100% performance gains when opening large files (depending on operating system, hardware configuration and file contents)."

Full Story (comments: 14)

RPM 4.10 released

Panu Matilainen announces the release of RPM 4.10.0. Most of the changes targeted robustness and correctness, but a few new features crept in as well, including support for parsing the tilde (~) operator in package version numbers.

Comments (35 posted)

GNU SASL 1.8.0 released

Version 1.8.0 of the GNU Simple Authentication and Security Layer (SASL) is out. New features include SAML20 (RFC 6595) and OPENID20 (RFC 6616) support, along with the addition of a number of SMTP server examples.

Full Story (comments: none)

systemd 183 released

Systemd 183 is out; this is a major release with a lot of new stuff; use on production machines is not recommended. "Note that we skipped 139 releases here in order to set the new version to something that is greater than both udev's and systemd's most recent version number." Changes include the incorporation of udev, the renaming of various configuration files and commands, a change to the LGPLv2.1+ license, better suspend and hibernate support, offline system update support, hardware watchdog support, /tmp as tmpfs by default, and lots more.

Full Story (comments: 28)

Development newsletters from the last week

• Caml Weekly News (May 29)
• GCC 4.7.1 status report (May 30)
• Perl Weekly (May 28)
• PostgreSQL Weekly News (May 27)
• Ruby Weekly (May 31)
• Tahoe-LAFS Weekly News (May 27)

Comments (none posted)

The Design of LLVM (Dr. Dobb's)

Dr. Dobb's has a moderately detailed overview of the design of LLVM by Chris Lattner. "In particular, LLVM IR is both well specified and the only interface to the optimizer. This property means that all you need to know to write a front end for LLVM is what LLVM IR is, how it works, and the invariants it expects. Since LLVM IR has a first-class textual form, it is both possible and reasonable to build a front end that outputs LLVM IR as text, then uses UNIX pipes to send it through the optimizer sequence and code generator of your choice. It might be surprising, but this is actually a pretty novel property to LLVM and one of the major reasons for its success in a broad range of different applications. Even the widely successful and relatively well-architected GCC compiler does not have this property: its GIMPLE mid-level representation is not a self-contained representation."

Comments (13 posted)

SFC expands compliance efforts to Samba and the Linux kernel

The Software Freedom Conservancy has announced that the Samba project (which, like BusyBox, is another of SFC's member projects) will be engaging in license compliance efforts with the organization. In addition, it announced a new "GPL Compliance Project for Linux Developers" that is working with seven Linux kernel copyright holders to work on compliance for their copyrights in the kernel. "Matthew Garrett, an accomplished Linux kernel developer, was the first to put forward his copyrights as part of the GPL Compliance Project for Linux Developers, and was quickly joined by six other individuals. In a statement today, Matthew noted: 'For some time, many Linux kernel copyright holders have offered our moral support to the BusyBox enforcement efforts through Conservancy, but didn't have the ability to formalize that support. I'm glad to put my copyrights forward for this new initiative, and welcome any Linux kernel copyright holders who wish to join us to reach out to Conservancy via <compliance@sfconservancy.org>.'"

Comments (4 posted)

The Linux Foundation Announces New Tool for Tracking Free and Open Source Software Components

The Linux Foundation has announced the release of The Linux Foundation FOSS Bar Code Tracker. "Released as an open source project under the MIT license, the new software tool aims to simplify the way open source components are tracked and reported by using an auto-generated, custom QR code for each product. The QR code contains important information on the Free and Open Source Software (FOSS) stack contained in a product, such as component names, version numbers, license information and links to download the source code, among other details."

Comments (5 posted)

Judge Alsup Rules: Oracle's Java APIs are Not Copyrightable (Groklaw)

Groklaw has the news that the judge in Oracle v. Google has ruled that the Java APIs are not copyrightable. "The overall name tree, of course, has creative elements but it is also a precise command structure — a utilitarian and functional set of symbols, each to carry out a pre-assigned function. This command structure is a system or method of operation under Section 102(b) of the Copyright Act and, therefore, cannot be copyrighted. Duplication of the command structure is necessary for interoperability." That pretty much makes the case a total loss for Oracle, but, of course, appeals are possible.

Comments (17 posted)

Lib-Ray: Non-DRM Open-Standards HD Video Format

Terry Hancock launched a kickstart campaign to fund his work on Lib-Ray, a non-DRM open-standards HD Video Format. Unfortunately we did not hear about it in time for last week's edition and the campaign closes June 3. Still, it sounds like an interesting project and other contributions may still be possible. "Unlike Blu-Ray, Lib-Ray releases do not support DRM, encryption, or region-coding options, and are intended for worldwide release. Thus the standard is designed with a highly-adaptable localization scheme, providing many more subtitles than are typically available on Blu-Ray or DVD regional releases."

Full Story (comments: 2)

Linux Tycoon [and more] going Open Source via donations

Bryan Lunduke, author of the game "Linux Tycoon" and the visual programming suite "Illumination Software Creator" is moving his software to the GNU Public License and is looking for donations to fund this work. "During this process I am making regular (multiple times per day) updates and trying to keep everything documented and transparent. In this way I hope to provide a solid case-study on moving a proprietary software business to Open Source -- In the hope that others can follow and do the same."

Full Story (comments: none)

Google launches Chromebook, Chromebox and gets it right (GigaOm)

GigaOm reviews the new Chromebook from Samsung, along with the associated desktop "Chromebox" device. "The ChromeOS and the devices based on the OS have reached a point in maturity where they can be used as an 'optional' or second computer. It is also benefiting from the fact that most of us have become used to living and working inside the browser."

Comments (9 posted)

Moonlight sent into twilight (The H)

The H looks into the death of Moonlight, a free software implementation of Microsoft's Silverlight platform. "Microsoft has been playing down Silverlight over the last year and a half, after previously promoting it as strategically important. In Windows 8, Silverlight is taking a back seat to HTML5 in the browser and for applications. Without Microsoft's weight behind Silverlight, the future for Moonlight has looked less and less promising."

Comments (57 posted)

This Cadillac Is Powered by Linux (Wired)

Wired is impressed with the Linux-powered in-vehicle infotainment (IVI) system in the most recent Cadillac XTS. "While the XTS’ spate of processors and controllers isn’t running the open source offspring of Linus Torvalds, the game-changing infotainment intender known as the Cadillac User Experience (CUE) is. [...] Buried deep within the dash is a three-core ARM 11 processor, powering two displays: one eight-inch capacitive touch screen — the first non-resistive display to come to a production car — and a second, 12.3-inch fully configurable instrument cluster mounted behind the steering wheel. Two of those cores adapt on the fly to handle voice commands powered by the same Nuance technology used by many automakers, along with Apple’s personal assistant, Siri. But with CUE, everything is processed on board."

Comments (10 posted)

Hadoop: The Definitive Guide, 3rd Edition--New from O'Reilly Media

O'Reilly Media has released "Hadoop: The Definitive Guide, 3rd Edition" by Tom White.

Full Story (comments: none)

2nd Call For Papers, 19th Annual Tcl/Tk Conference 2012

The Tcl/Tk Conference will take place November 12-16, 2012 in Chicago, Illinois. The call for papers will be open until August 27.

Full Story (comments: none)

LUGOD presentation: "Wikimedia and Wikipedia"

The Linux Users' Group of Davis (LUGOD) will have a presentation on "Wikimedia and Wikipedia" by Trevor Parscal and Roan Kattouw from the Wikimedia Foundation at their June 18 meeting in Davis, California.

Full Story (comments: none)

PyCon Australia 2012 and Google Australia announce gender diversity grants

PyCon Australia and Google Australia have announced that they are joining forces to offer gender diversity delegate grants to women who wish to attend PyCon Australia in 2012. "These grants will cover up to $AUD500 of travel, accommodation and registration costs for women living outside of the Southern Tasmania region to attend this year's conference." Applications for the gender diversity delegates grants will close on June 15. PyCon AU takes place August 18-19, 2012 in Hobart, Tasmania.

Full Story (comments: none)

PyCon DE 2012 - Registration Open

PyCon DE will take place October 29 - November 3, 2012 in Leipzig, Germany. Early bird registration is open until the end of June. The call for talks and tutorials is still open as well.

Full Story (comments: none)

Linux Color Management Hackfest 2012

There will be a Linux Color Management Hackfest November 16-19, 2012 in Brno, Czech Republic. Some funds are available to sponsor a limited number of people. Applications must be submitted by September 30.

Full Story (comments: none)

Events: June 7, 2012 to August 6, 2012

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
June 6 June 8	LinuxCon Japan	Yokohama, Japan
June 6 June 10	Taiwan Mini DebConf 2012	Hualien, Taiwan
June 7 June 10	Linux Vacation / Eastern Europe 2012	Grodno, Belarus
June 8 June 10	SouthEast LinuxFest	Charlotte, NC, USA
June 9 June 10	GNOME.Asia	Hong Kong, China
June 11 June 16	Programming Language Design and Implementation	Beijing, China
June 11 June 15	YAPC North America	Madison, Wisconsin, USA
June 12	UCMS '12: 2012 USENIX Configuration Management Workshop: Virtualization, the Cloud, and Scale	Boston, USA
June 12	WiAC '12: 2012 USENIX Women in Advanced Computing Summit	Boston, USA
June 12	USENIX Cyberlaw '12: 2012 USENIX Workshop on Hot Topics in Cyberlaw	Boston, USA
June 12 June 13	HotCloud '12: 4th USENIX Workshop on Hot Topics in Cloud Computing	Boston, USA
June 13 June 15	2012 USENIX Annual Technical Conference	Boston, MA, USA
June 13	WebApps '12: 3rd USENIX Conference on Web Application Development	Boston, USA
June 13 June 14	HotStorage '12: 4th USENIX Workshop on Hot Topics in Storage and File Systems	Boston, MA, USA
June 14 June 17	FUDCon LATAM 2012 Margarita	Margarita, Venezuela
June 14 June 15	TaPP '12: 4th USENIX Workshop on the Theory and Practice of Provenance	Boston, MA, USA
June 15	NSDR '12: 6th USENIX/ACM Workshop on Networked Systems for Developing Regions	Boston, MA, USA
June 15 June 16	Nordic Ruby	Stockholm, Sweden
June 15 June 16	Devaamo summit	Tampere, Finland
June 16	Debrpm Linux Packaging Workshop in the Netherlands	The Hague, Netherlands
June 19 June 21	Solutions Linux Open Source	Paris, France
June 20 June 21	Open Source Summit (NASA, State Dept, VA)	College Park, MD, USA
June 26 June 29	Open Source Bridge: The conference for open source citizens	Portland, Oregon, USA
June 26 July 2	GNOME & Mono Festival of Love 2012	Boston, MA, USA
June 30 July 6	Akademy (KDE conference) 2012	Tallinn, Estonia
June 30 July 1	Quack And Hack 2012	Paoli, PA, USA
July 1 July 7	DebConf 2012	Managua, Nicaragua
July 2 July 8	EuroPython 2012	Florence, Italy
July 5	London Lua user group	London, UK
July 6 July 8	3. Braunschweiger Atari & Amiga Meeting	Braunschweig, Germany
July 7 July 12	Libre Software Meeting / Rencontres Mondiales du Logiciel Libre	Geneva, Switzerland
July 7 July 8	10th European Tcl/Tk User Meeting	Munich, Germany
July 8 July 14	DebConf12	Managua, Nicaragua
July 9 July 11	GNU Tools Cauldron 2012	Prague, Czech Republic
July 10 July 15	Wikimania	Washington, DC, USA
July 10 July 11	AdaCamp Washington, DC	Washington, DC, USA
July 11	PuppetCamp Geneva @RMLL/LSM	Geneva, Switzerland
July 11 July 13	Linux Symposium	Ottawa, Canada
July 14 July 15	Community Leadership Summit 2012	Portland, OR, USA
July 16 July 20	OSCON	Portland, OR, USA
July 26 July 29	GNOME Users And Developers European Conference	A Coruña, Spain
August 3 August 4	Texas Linux Fest	San Antonio, TX, USA

If your event does not appear here, please tell us about it.