LWN Weekly Edition
Leading itemsSecurity
Kernel development
Distributions
Development
Announcements
->Multipage format
This page
Previous weekFollowing week
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for June 1, 2012Sizing up the ColorHug open source colorimeterHaving covered the open source graphics software community for several years, I was immediately intrigued when I learned about the ColorHug project. ColorHug is a spare-time effort by Red Hat's Richard Hughes to build an open source monitor calibration device. I am still resigned to the fact that most people's eyes will glaze over whenever you utter the phrase "color management," but with ColorHug, at least interested parties have a simple and affordable option — and as with any good open source project, there are hints at more things still to come. Color me goodFor the benefit of the uninitiated, the ColorHug is a tristimulus colorimeter. That means it can measure the amount of RGB light hitting its sensor; by displaying a carefully-chosen set of color swatches on the screen, it can characterize the display's output and generate an ICC profile. You can then load the ICC profile into the OS color management system, and enjoy balanced output. The ColorHug sports a 64-pixel light sensor, and like other colorimeters uses filters to read the red, green, and blue values separately. Unlike many other colorimeters, the ColorHug also converts the readings into a standardized XYZ color value in the firmware, and can correct the output to account for the different illumination characteristics (e.g., varying backlights and primary colors) found in different display types. Unlike essentially every other colorimeter, the ColorHug is supported only under Linux — it ships with a Fedora-based live CD to perform the calibration, although the resulting ICC profile can be used by any operating system. That aspect of the project receives little attention, but it is significant. There are other colorimeters supported by open source software projects like Argyll (which handles underlying calibration tasks), but there is no official Linux support. On top of that, the existing Windows and OS X client software is usually restrictive, with limits on the number of copies that can be installed, requiring email registration, and so on. Hughes made a point of designing the ColorHug to be fast and accurate hardware at a low price point (currently £60), but the software support is equally significant. It even gets free firmware updates, which could make for some interesting developments down the road. How it works![[ColorHug device]](/images/2012/05-colorhug-device-sm.jpg)
Physically the ColorHug is a small black puck about the size of a matchbox, with a sensor aperture on one side and a mini-USB port on the end. It comes with a mini-to-fullsize USB cable and the aforementioned live CD. Hughes is also the author and maintainer of GNOME Color Manager (g-c-m) and colord, so that software is used by the live CD to detect attached displays and colorimeters, as well as to step the user through the process of creating a display profile. G-c-m should automatically recognize your displays and list them in its main window; to start creating a profile you simply select one and click the "Create Profile" button. The application will ask you to name your new profile (although it maintains a time-sorted list of the resulting profiles in the window, too, so the name is primarily for your benefit) and select the accuracy at which to analyze the monitor. The approximate measurement times listed — up to 20 minutes for the most thorough — are very conservative when it comes to the ColorHug; I cannot think of any reason to create a low-accuracy profile other than curiosity. I tested the ColorHug against a Pantone Huey (a common proprietary colorimeter), and creating an accurate profile took well under 10 minutes for each device, with a slight edge going to the ColorHug. Whatever accuracy you choose, when you click the "Start" button the application launches into a sequence of color patches displayed in the center of the screen. You must have the ColorHug centered on the display area and keep it still for the duration of the process. That proved to be the trickiest part of the formula for me; the USB cable is on the thick side, and the ColorHug weighs so little that the cable pulls it out of position if you don't take care to wedge it in place with a nearby object. Even then, profiling a laptop is easier because you can maneuver the display to lie flat. Back when CRTs were the norm, proprietary colorimeters often used suction cups to stick to the front of the monitor. That is clearly a bad idea for LCDs, but some form of clip to hang the device in place against a vertical surface might be a nice addition in the future.
After a few minutes of silent color-swatch juggling, g-c-m announces that it is finished, and you can instantly apply the freshly baked ICC profile to your display. This is the point where you might notice trouble. Early versions of the software asked you to select the white point of the monitor during the first step, and the wording encouraged you to select the daylight-balanced D65, but that resulted in profiles skewed sharply into the red part of the spectrum. The correct choice was to use the display's "Native" white point, and updated versions of the software and live CD now choose that setting automatically. The other tricky bit is that it might be hard to find the actual ICC profile files if you are running the live CD. They are located in ~/.local/share/icc/, so you need to copy them to another storage location in order to load them into your normal desktop environment and reap the benefits. However, you still may find the newly-adjusted profile looks different than you expect. The truth is that most LCDs are factory set to a very cool-blue white point because that appears brighter to the human eye. But brighter is not accurate; I immediately noticed that the Fedora live CD's aquatic background image had a lot more color variation when I applied the new ICC profile; without it the entire ocean scene was deep blue. Your eyes will adjust to the change. More importantly, if you profile multiple displays, your eyes will not have to adjust when you switch between them. You can see "before" and "after" photos above ("before" is on the left). Of course, how much of a difference you see depends on your own display as well, but hopefully it is at least clear that the grays are much bluer in the "before" shot. Using the live CD is actually pretty important; Hughes updates the client code and utilities frequently and his updated ISOs are the only officially guaranteed path to making all of the pieces work together. Your distribution might not package the most recent version of Argyll or other dependencies for several months. Auxiliary bitsDisplay profiling is the ColorHug's primary purpose, but the project has other interesting infrastructure built in as well. For instance, it is firmware-upgradeable: the live CD comes with a GUI firmware updater that checks the project web site for a new release, and allows you to download and flash it. The first time I tried that, it did not work. But the ColorHug is also designed with a separate firmware bootloader to prevent accidentally bricking the device. The ColorHug site explains how to recover from a bad flash with command line utilities included on the live CD, and I am happy to say that they worked. You can also use the ColorHug to calculate a CCMX color transformation matrix, if you have access to a separate spectrophotometer. A spectrophotometer is a different class of calibration device from a colorimeter; rather than using a light sensor to measure RGB values, it measures the amount of light output by a display across the full visual spectrum, in tiny increments. They are pretty expensive in practice, but if you can wrestle one away from a friend, you can use it to create a CCMX matrix that will be shared with the rest of the community. These matrices are most valuable for non-standard display types; most LCD screens work fine with the standard sRGB matrix shipping in the ColorHug. But mobile phones, TVs, and other devices do differ, and the project is interested in collecting as many good samples at it can, which will then be included in future firmware updates. You can grab and install new CCMXes with the colorhug-ccmx tool included on the CD. On the ColorHug mailing list, there was a thread about using other colorimeters to generate CCMX data. The process relies on Argyll, and author Graeme Gill confirmed that the Argyll tools can create a matrix from two colorimeters. I asked Hughes what the relative value of such a colorimeter-to-colorimeter matrix is, and he likened it to trying to adjust your speedometer by driving alongside another car: not totally useless, but definitely not reliable enough to recommend. Finally, the G-c-m build on the live CD includes tools for examining ICC profiles themselves. In the version of the CD I tested, they qualify as interesting but not rigorous. For example, you can show a 3D rendering of each profile as a solid space, but you cannot overlay to profiles to compare them. I hope that that will improve in the future, the other major Linux color management system, Oyranos, has a very nice profile inspection tool called ICC Examin, so there is hope. Speaking of Oyranos, just because colord is used by the live CD, that does not mean that the ColorHug's ICC profiles are usable only with colord on a GNOME system. The profiles are bound to the hardware, whichever software is used to load them. Oyranos developer Kai-Uwe Behrmann is also working on a set of enhancements to ICC file metadata via the OpenICC project which should make interoperability better in the future, but you can simply install the profiles on an Oyranos system and take advantage of them now. You can also submit them to the taxi service, which collects user-generated ICC profiles in an effort to build a more reliable set of real-world options for users who cannot generate their own profiles. You can even use the profiles in Mac OS X and Windows if you multi-boot. The futureAutomatically installing the profiles created by the live CD is a possible enhancement that has come up on the ColorHug mailing list, although for now it is not high-priority. More interesting is the possibility of extending the hardware's usefulness — such as to CRT displays and projectors. Supporting CRTs requires different color-measurement timing, due to that hardware's refresh rate. Projectors are another beast altogether. There are proprietary colorimeters intended for use with projectors, and technically the ColorHug hardware can be used with them as well — but it is tricky. There is an aperture disk in front of the ColorHug light sensor; removing it allows the sensor to see a larger sample area, which would be useful for projectors, but the details of placing and orienting the device have yet to be worked out. Similarly, removing the aperture disk would also make the device useful for profiling the ambient light in a room, but here again the details have yet to be worked out. Hughes and Gill have discussed the issues on the mailing list — including whether or not a light diffuser would be required, but it is possible that future versions of the software will support these or other new functions. Looking even further out, Hughes has floated the idea of building an open source spectrophotometer as a sequel or companion piece to the ColorHug. There was interest in the idea on the mailing list, but most thought such a device would be useful only if it was designed to help profile reflective light — in other words, printers. Not all spectrophotometers do so; the hardware challenge is significantly steeper, because in addition to making precise measurements at specific wavelengths, a reflective light spectrometer also needs to provide a precise illumination source, which is a tricky prospect for any project. Hughes indicated that such a device was a 2013-or-later project, and it would cost at least four or five times as much as the ColorHug. Of course, the ColorHug proves that there is at least some market for open source color-measurement hardware, which few people would have predicted before the project launched. In fact, according to Hughes' talk at Libre Graphics Meeting 2012, he can barely keep up with demand for the devices. The tale of designing, manufacturing, and iterating the devices is interesting for anyone looking for real-world information on open hardware projects; hopefully slides or a recording of the session will be available soon. As an open hardware device, you can download the gEDA schematics and the firmware for the ColorHug from the same repository that hosts the client software. But for everyone who is merely interested in getting good results from a monitor, the ColorHug is a good buy on its own. It would probably be a good buy even if it was closed source: it is fast, inexpensive, and accurate. But the fact that you can tweak and modify it only adds to the value.
Relicensing and rebasing LibreOfficeProjects the size of LibreOffice often tend to get a little unwieldy; the size of the code is such that even seemingly trivial tasks like removing dead code can take a long time. Considering the sheer size of the project and the fact that its copyright ownership is distributed, it would be natural to doubt the sanity of anybody proposing to simultaneously move 1.5 years worth of work to a new base and adopt a new license. But that is just what LibreOffice has in mind.LibreOffice is based on the original OpenOffice.org project; it inherited its current LGPLv3 license from there. The project is clearly happy with copyleft licensing, but it occasionally shows signs of wanting a bit more flexibility than LGPLv3 sometimes provides. LibreOffice has, since the beginning, accepted all changes under a dual license, mixing LGPLv3 with version 2 of the Mozilla Public License. But the LibreOffice project does not own the original OpenOffice.org code, so it must distribute its work under LGPLv3, essentially dropping the MPLv2 license from the changes that have been made since the project began. Something interesting happened a while after LibreOffice launched, though: Oracle donated the code to the Apache Software Foundation (ASF) which, after some work, has released it under the Apache License. That is a permissive license, which is not something LibreOffice is interested in. But the Apache License is compatible with MPLv2; a derived product containing code under both licenses can be distributed as an MPLv2-licensed whole. Thus, LibreOffice has concluded:
With the relicensing of the original OpenOffice.org code-base to
Apache License 2.0 by Oracle, we're now able to incrementally
rebase our own code on top of that to offer a choice of licensing
that does not only include LGPLv3 but also any of GPLv3.0+,
LGPLv3.0+, and the AGPLv3.0+ that the MPLv2+ allows. This will also
allow us to incorporate any useful improvements that are made
available under that license from time to time.
In other words, LibreOffice intends to toss out its original OpenOffice.org base and move its changes over to the newly released, Apache licensed version; the result will be a LibreOffice that can be distributed under the MPL, which the project sees as being worthwhile:
As we compete with our own code, licensed under an unhelpfully
permissive license, the MPL provides some advantages around
attracting commercial vendors, distribution in both Apple and
Microsoft app-stores, and as our Android and iPhone ports advance
in tablets and mobile devices.
Getting there will be an interesting process, though. The code given to the ASF is substantially the same as the code LibreOffice started with, but, as some Apache contributors have taken pains to point out, that code is not available under the Apache License. Contrary to the text quoted above, Oracle did not relicense the code; that code was, instead, given to the ASF, which was given the freedom to release it under a new license. The only code that is actually available under the Apache License is that found in the Apache OpenOffice 3.4 release. Anything that has not been officially released by Apache has not truly been relicensed. Indeed, it has been claimed that code found in the Apache repositories—and, thus, currently distributed by Apache—is not necessarily available under the Apache License. So LibreOffice will have to start with the official release, which has had a lot of components carved out of it. Much of that code has been replaced with permissively-licensed code, but not all of it. There are also several chunks of code that Apache OpenOffice may relicense and integrate into a future release, but that has not happened yet. LibreOffice developers have requested help in clarifying the license status of that code, but answering those queries has not been a high priority for Apache OpenOffice, especially while the latter was working flat-out to make its own first release. So the relicensing of LibreOffice is not going to be a quick task. The project must start with, and limit itself to, code that is known to be available under the Apache license. Then it will be necessary to rebase the large number of changes made over the lifetime of the LibreOffice project onto the new release; the project thinks that will be relatively easy, because Apache OpenOffice has, thus far, not made a lot of changes:
It is worth pointing out that only around 6% of the files in the
Apache OpenOffice incubator project have any code change beyond
changing the license headers. A rather substantial proportion of
the those changes are highly localised, or are inclusions of code
not generated by the Apache project itself.
There remains the task of filling in all of the parts that Apache OpenOffice simply removed from the code base or has not yet gotten around to releasing under the Apache License. How easy that will be is not entirely clear; it may well be that, even after rebasing the project, LibreOffice may not be able to make a fully MPL-licensed release for some time. There are also little details like getting the license headers right, but they seem minor by comparison. In summary, the LibreOffice project appears to be setting itself up for a fair amount of work. License changes are hard, especially when one does not necessarily have the cooperation of the copyright holders; it is only possible this time around through the surprising combination of the Apache relicensing and the LibreOffice project's prescient-seeming decision to require LGPL/MPL dual licensing on all contributions. The reward from all of this work, the project hopes, will be more flexible licensing that helps it to compete with the project upon which it is rebasing. All told, the interesting interplay between these two projects seems destined to continue for the foreseeable future.
SFC expands license compliance effortsTo a large extent, BusyBox has been the "poster child" for GPL enforcement, at least in the US. That may now be changing with the announcement that the Software Freedom Conservancy (SFC) has signed up multiple Linux kernel and Samba developers whose copyrights can be used in license compliance efforts. That expands the scope of license enforcement activities while also removing the need to use the controversial GPLv2 death penalty threat to ensure compliance with the kernel's license—SFC should be able to go directly at kernel enforcement now. Copyright holders step upThere are three parts to the announcement. First off, the Samba project, which is an SFC member project, has moved its existing compliance efforts over to SFC. Secondly, a new "GPL Compliance Project for Linux Developers" has been created for Linux kernel copyright holders to allow their contributions to be used in enforcing the GPL on that code. So far, seven copyright holders have joined the project. Lastly, several other SFC member projects (Evergreen, Inkscape, Mercurial, Sugar Labs, and Wine) have requested that the organization handle any compliance issues for their code. The kernel copyright holders that have joined the project are—somewhat curiously—not listed, other than Matthew Garrett, who was the first. In an email exchange with SFC executive director Bradley M. Kuhn, he said that he asked Garrett to "officially speak on behalf of this program", but that the others are welcome to speak up if they wish. Essentially, Kuhn is shielding those developers from having to handle questions if they don't want to, which is part of the mission of SFC: "we take care of boring work for Free Software projects so the developers can focus on the developing the code". Furthermore, Kuhn recognizes that GPL compliance activities are sometimes controversial in the community. So, he is happy to deflect any criticism in his direction:
I'm aware this issue can be
divisive for some (I don't really understand why myself, but I see it's
there), so I don't want developers to feel they all have to stand up and
take the influx of FUD on this issue. That's my job to do on their
behalf.
Similarly, Kuhn asked Jeremy Allison to be the contact person for the Samba copyright holders, who were also unnamed. There were nine of those before the announcement, but Kuhn said that another stepped forward after it went out. Part of the idea behind the announcement is to do just that: attract more copyright holders, particularly kernel copyright holders, to the compliance efforts. Kuhn also expects that some of the copyright holders may make themselves known in "comment threads, blogs and the like". Copyright coverageOne question that arises when only a subset of the copyrights in a body of code are being used is whether those copyrights are sufficient to be used for compliance purposes. For Samba, there is little question in Kuhn's mind that Allison's contributions would be sufficient by themselves: "Jeremy's copyrights *alone* are so extensively spread across the many Samba codebases for multiple decades that it's probably impossible to have a working Samba binary without including very substantial portions of Jeremy's copyright works". But he is happy to have other Samba developers on board "to show their solidarity and so that they could give input into the process of the compliance activity". On the kernel side, Garrett himself noted that his copyrights might not be useful for enforcement activities: "Most of the past work I've done is in bits of the kernel that are rarely present in infringing devices, and most of my recent work is owned by my employer rather than me." But that was an off-the-cuff blog comment that Kuhn said "painted a substantially bleaker outlook than the reality" once SFC started investigating Garrett's copyrights. Beyond that, Kuhn said, the addition of the other copyright holders has further strengthened SFC's hand:
Conservancy is confident that those developers
who signed up for compliance activity on their Linux copyrights hold
enough copyright that the code can't, for example, be trivially written
out of Linux by someone who wants to avoid working with Conservancy on a
compliance matter related to Linux.
Kuhn said that SFC is now going through the laborious process of registering the copyrights of all of those involved. The basic methodology is "going through git logs, finding commits by a particular developer, verifying their copyright claim to that commit, and then preparing the source code in a form that the copyright office can grok", he said. That will take some time, but, once it's done, those registrations will be searchable by those interested at the US copyright office. But seven copyright holders is only a tiny fraction of the thousands of contributors to the Linux kernel over the years. To understand the scope of the copyrights that can be enforced, we would need to know just who has signed up with SFC. Until the copyright registration process is completed—or a lawsuit filed—we won't really have a good feel for that it seems. Compliance, not litigationWhen asked about any litigation on the horizon, both Kuhn and Allison were quick to point out that SFC only uses lawsuits as a last resort. The policy has always been to file suit only "when a company simply refuses to respond to repeated requests from Conservancy about compliance problems", Kuhn said, while Allison called lawsuits "a 'LISTEN TO ME !!!!' action" for those who are non-responsive. In fact, Kuhn said, "the average is less than one lawsuit every three years", though SFC has been accused of being "overly litigious" by some. By way of contrast, SFC deals with more than a hundred minor compliance issues (e.g. some kind of compliance question) each year. In addition, situations that "required complex negotiation" number in the twenties each year, he said. There is also a certain amount of educational work that SFC does, including Kuhn speaking publicly about the issue at various conferences "which (hopefully) reaches a wide audience and (again, hopefully) raises the level of understanding" about compliance for companies beyond those SFC interacts with directly. The announcement essentially puts companies on warning that there are "cops on the beat", Allison said:
They are low-key cops, who would rather give you a
stern talking to and get you to fix your driving on the quiet
than drag you publicly into court, but it does mean that if
you're using the code in these projects you do have to at least
*think* about compliance. You can't just ignore it, there are
consequences.
PlansSFC will be consulting with the copyright holders to determine the compliance steps that are to be taken. The organization is not looking to go its own way, but rather to work with the stakeholders to find a path that is acceptable to all. As the announcement put it:
Conservancy's new effort will be run in a collaborative manner with the
project developers. All copyright holders involved will have the
opportunity to give input and guidance on Conservancy's strategy in dealing
with compliance issues. Thus, all Conservancy's compliance matters will
have full support of relevant copyright holders.
That attitude may be helpful in bringing in other developers that might be leery of just appointing SFC as their copyright agent. By making it clear that they will be consulted and are able to help define the strategy and tactics, SFC is clearly trying to alleviate fears in the hopes of bringing in more copyright holders. The addition of kernel and Samba copyrights (and, likely to a lesser degree, those of the other SFC member projects) will clearly expand the number and kinds of license compliance offenders the organization can target. But Kuhn does not see SFC spending any more time on compliance than it has in the past. It is third on the list of SFC activities as reported in its 2010 Form 990 [PDF], and Kuhn does not see that changing as SFC "doesn't want license compliance to be our sole or even primary focus". That said, there are a number of things that need to be done. Kuhn said that SFC will be giving the non-responsive companies in its queue "another go-around" in hopes that the announcement will spur them to start engaging. That queue, which numbers around 250 violations, will also need to be re-prioritized based on product age and availability, he said. Compliance is an important issue for our communities. It is not just that some are completely ignoring the—pretty minimal in the grand scheme—requirements that we put on our freely available code, but also that violators are getting an advantage on their compliant competitors. As Allison put it: "The people who do think about compliance are subsidising their competitors who don't. I think that's unfair." With this announcement, SFC now has more weapons in its arsenal so that it can reduce that problem, while still working within the community it serves. Compliance may be controversial in some circles, but choosing a license that places restrictions on distribution—as the GPL does—doesn't really make sense unless it is enforced. Those who would rather not see SFC (or others) push compliance on companies might be best served by more permissive licenses.
Security TACK: TLS key pinning for everyoneMoxie Marlinspike and Trevor Perrin have created a TLS enhancement called Trust Assertions for Certificate Keys (TACK) that is designed to protect users against SSL certificate fraud with minimal overhead or infrastructure. The TACK proposal has been submitted to the IETF, as has a competing key-pinning mechanism created by Google's Chrome team. Although the two systems share broad similarities, their implementations are quite different. The troubleTACK is designed to combat flaws with the certificate authority (CA) trust chain model used by applications to check the validity of a remote site's TLS public key. The entire CA chain can be undermined by one bad CA, but there are other difficulties, too. First, any given hostname (e.g., www.example.net) can map to several different servers, with several certificate and thus several different CA chains deployed at once (e.g., in different jurisdictions) — which makes it harder to verify a connection. Second, the CA chain itself may change at any time, due to CA actions that are outside the control of the www.example.net site administrators. There have been plenty of previous approaches to circumventing CA problems (including Marlinspike's own Convergence). "Certificate pinning" is a client-side technique in which the browser remembers (or "pins") a known-good SSL certificate across multiple browser sessions, and alerts the user if the certificate suddenly seems to change. The TACK proposal argues that certificate pinning is flawed because it binds the trust not to the server's TLS key but to the certificate, which includes the key plus the CA's signature and various other metadata. Thus, by extension, pinning the entire certificate ties trust of the server's TLS key to trust of the CA. Google Chrome added a key-pinning feature (which tracks known-good TLS keys, even though it is often erroneously referred to as "certificate pinning") with Chrome 13 in mid-2011. But the TACK project says that this, too, has its flaws. First, Chrome's key-pinning uses a whitelist approach, with the browser designating specific CAs as trustworthy. But as long as the CA chain includes a whitelisted CA anywhere in the chain, the whole chain is trusted. Second, if a server issues a new TLS key, it does not roll out to users until a new version of Chrome is released. TACKIn TACK, the identity-verification mechanism is also key-pinning, but rather than pinning a CA key, the browser pins a special "TACK" data structure created and controlled independently by the web site's administrators. The TACK includes a cryptographic signature of the site's TLS key, plus a public encryption key that the browser can use to verify the signature. Obviously the TACK public key and the TLS public key need to be different. In addition to this basic identity information, the TACK also includes an expiration date and a "generation" counter that gets incremented every time the site administrator creates a new TACK (so that old TACKs can be revoked if necessary). The final piece of the puzzle is that TACK is an extension to TLS itself, rather than an extension to HTTP as in Google's pinning proposal. In practice, when the client application initiates a TLS connection, it requests a TACK from the server as part of the handshake step. When the server sends a TACK back in return, the client checks the generation information against its current cache for the site (if it has one), checks the expiration date, and validates the signature. If everything checks out, the TACK is "pinned" locally for future reference. The client can still choose to verify the CA chain or use other means to cross-check the identity of the server. When the client pins a TACK, it is supposed to assign a cache lifespan to it equal to the amount of time between when it first saw the TACK and the present — up to a maximum of 30 days. In other words, a TACK that was first encountered seven days ago gets pinned for another seven days, while a TACK from one year ago gets the 30-day maximum. In the maximum-lifespan case, however, the TACK will only remain in the local cache if the user has returned to the site at least once every 30 days; otherwise it will have expired on the 31st day and a new pin will be created. The big drawback of the TACK mechanism is that, like other client-side key-pinning methods, it does not protect the user against a man-in-the-middle (MITM) attack during the very first connection attempt to the remote server. It relies on the client having known-good information pinned locally. On the other hand, the TACK generation, cache lifespan, and expiration date features are intended to mitigate the damage that a MITM attack can inflict. The worst-case scenario is a bad pin that lingers for 30 days, which is half the time the rogue google.com certificate issued by DigiNotar lingered in 2011 — and it could have lasted much longer had it not been discovered. Another advantage of TACK's design is that in the worst-case scenario, the server administrator can single-handedly push out an update, rather waiting on the CA or another party. But TACK's developers also note that there is no reason that TACKs must only be retrieved "live" by the browser. Search engines could index well-known TACKs and browser-makers could pre-load lists of them into their applications. The TACK team has posted example TACK code for Apache and OpenSSL, plus a Python tool for making TACK requests. Google, pins, and the futureGoogle submitted its own key-pinning proposal to the IETF, based on an evolution of its original feature in Chrome. The Google proposal is similar in many ways to TACK. For starters, unlike the built-in pinning currently found in Chrome, it allows each client application to check and cache its own pins rather than depend on a vendor-supplied list. It also pins individual site keys rather than CA keys, so it is independent of the CA chain-of-trust. Google's pin format includes a max-age field, which takes the place both of TACK's expiration date and its maximum cache-lifespan criteria. Google's proposal does not track generations, however. To recover from a lost or stolen pin-signing key, it instead mandates that server administrators keep a "Backup Pin" — a second, un-deployed signing key that the administrators must store offline. The key fingerprint of the Backup Pin is published by the server, though, so that clients will recognize it if it needs to be deployed. Presumably when the Backup Pin is deployed, the administrators will then need to create a new, backup backup Pin. There are other details that differ between the proposals (such as what signature algorithms are required), but the most fundamental difference is that Google's pinning proposal is an extension to HTTP: servers send their pinnable key signatures in a new HTTP header, rather than as part of the TLS handshake. TACK's proponents argue that because the CA weakness is fundamentally a TLS problem, it should be fixed in TLS itself, and that building an HTTP-based solution leaves other important protocols (e.g., IMAP and POP3) unprotected. HTTP proponents argue that an HTTP solution is easier to deploy due to the difficulty of setting up a TLS stack, and that HTTP protects the most attack targets. As of today, TACK is being discussed by the IETF's TLS Working Group. It is a new enough proposal that it is likely to see revisions before it advances any further in the IETF standards process. Google's key pinning proposal was accepted as an Internet-Draft by the Web Security Working Group in December. Whichever makes it to broad browser and server acceptance first, it is fair to say that users will start hearing more about "pinning" in one form or another — but they still should not expect the CA system to disappear any time soon.
Brief items Security quotes of the week
This has been a public service announcement made necessary by some damn' fool European Commission directive that confused a goal (securing web users' privacy) with a technology (cookies). Film at eleven.
-- Charlie
Stross
Let's say you chose NOT to accept any cookies from bbc.com (most people are going to tend toward a binary decision -- all or none -- not try to micromanage their cookies). The result of blocking all BBC cookies will be that (apparently) you'll be forced to see this banner over and over and over ... and over again. How do you stop it? By accepting BBC cookies of course!
-- Lauren Weinstein
When I helped to develop the open standards that computers use to
communicate with one another across the Net, I hoped for but could not
predict how it would blossom and how much human ingenuity it would
unleash. What secret sauce powered its success? The Net prospered precisely
because governments — for the most part — allowed the Internet to grow
organically, with civil society, academia, private sector and voluntary
standards bodies collaborating on development, operation and governance.
-- Vint
Cerf worries about the future of the internet
Even though humans produce distributions with pitifully few bits of
security, I think passwords will always be with us. As one component in a
system with many layers, passwords can be valuable as a low-cost
authentication mechanism which nearly all people can do with no special
equipment. The important thing is to stop considering them the first and
last step in authentication.
-- Joseph Bonneau
These days I'd argue that multi-user is such a corner case that it's
not worth optimizing for it as far as defaults are concerned. If
you're trying to run a secure multi-user system, you need to be an
expert system administrator, keep up with all security patches, and
even then, good luck to you. (The reality is that these days, no
matter what OS you're talking about, shell == root. And that's
probably even true on the most unusably locked down SELinux system.)
-- Ted Ts'o
SSL fix flags forged certificates before they're accepted by browsers (Ars Technica)Over at Ars Technica, Dan Goodin writes about Trust Assertions for Certificate Keys (TACK), a proposed extension to SSL/TLS designed to discover fake certificates before they are accepted. "The opt-in system works by allowing SSL sites to sign valid SSL certificates, the domain name, and an expiration date with a TACK key. Once an end user has visited the site a few times using a TACK-compatible browser, a 'pin' for that site is activated on the user's computer. If the end user later encounters a forged certificate for that same site—as was the case when DigiNotar was breached—the browser will reject the session and return a warning to the user." One of TACK's co-creators is Moxie Marlinspike, who proposed the Convergence alternative certificate-management framework in 2011.
Android Malware Genome Project launched (The H)The H covers the debut of the Android Malware Genome Project by researchers from North Carolina State University. The team "has already collected more than 1,200 samples of Android malware, including GingerMaster and DroidKungFu, and has organised them into various malware families. [Xuxian] Jiang told Dark Reading that 'the purpose is to engage the research community to better our understanding of mobile threats and develop effective solutions against them.'" Access to the data set, however, is restricted.
Implementing UEFI Secure Boot in FedoraOn his blog, Matthew Garrett writes about plans for supporting UEFI secure boot in Fedora 18. In it he covers signing the first-stage bootloader with a Microsoft key, while signing GRUB 2, the kernel, modules, etc. with a Fedora key. It is a compromise to try to avoid problems for users who want to boot Linux on Windows 8 hardware. "The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key."
New vulnerabilities chromium: multiple vulnerabilities
cobbler: remote code execution
dokuwiki: cross-site scripting/request forgery
kernel: denial of service
kernel: privilege escalation
mailman: information disclosure
net-snmp: denial of service
pidgin: multiple vulnerabilities
python: insecure file creation
python-tornado: HTTP header injection
request-tracker3.8: multiple vulnerabilities
strongswan: authentication bypass
wireshark: denial of service
xinetd: service disclosure flaw
Page editor: Jake Edge Kernel development Brief items Kernel release statusThe 3.5 merge window remains open as of this writing, so there is no current development kernel. The flow of patches into the mainline continues; see the separate article below for a summary of what has been merged thus far.Stable updates: there have been no stable updates released in the last week. The 3.0.33, 3.2.19, 3.3.8, and 3.4.1 updates are all in the review process as of this writing; they could all be released at any time.
Quotes of the week
Lumpy reclaim had a purpose but in the mind of some, it was to kick
the system so hard it trashed. For others the purpose was to
complicate vmscan.c. Over time it was giving softer shoes and a
nicer attitude but memory compaction needs to step up and replace
it so this patch sends lumpy reclaim to the farm.
— Mel
Gorman
Jiri [Kosina] is now also marked as the maintainer of floppy.c, I
shall be publically branding his forehead with red hot iron at the
next opportune moment.
— Jens Axboe
As a kernel rights holder I question the legality of Matthew's
proposal, and it would be amusingly unfortunate if the Software
Conservancy ended up beginning some of its Linux enforcement
against Fedora.
— Alan Cox
klibc 2.0 releasedVersion 2.0 of the minimal C library klibc has been released. "A bit delayed due to kernel.org breakin, but development is kicking in again. The 2.0 branch saw boot time tests and deployments in Debian, so we are quite certain it should work for the most out of you, if not please let us know." The biggest change appears to be proper support for buffered I/O in the stdio functions.
Kernel development news 3.5 merge window part 2The 3.5 merge window continued in full force after last week's summary, with another 4,000 non-merge changesets pulled into the mainline since then; there have now been over 8,600 changes merged for 3.5, and the merge window is not done yet. The most significant user-visible changes are:
Changes visible to kernel developers include:
As of this writing, the 3.5 merge window has a few more days left to run. The final article in this series will come out once the merge window has closed.
Uprobes in 3.5Uprobes is a kernel patch with a long story and many contentious discussions behind it. This code has its roots in utrace, a user-space tracing and debugging API that was first covered here in early 2007. Utrace ran into various types of opposition (only partly related to its own origin in SystemTap) and has never been merged, but a piece of it lives on in the form of uprobes, which is charged with the placement of probes into user-space code. After several mailing-list rounds of its own, uprobes was finally merged for the 3.5 kernel development cycle. Just how this facility will be used remains to be seen, however.At the core of uprobes is this function:
#include <linux/uprobes.h>
int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc);
The inode structure represents an executable file; the probe is to be placed at offset bytes from the beginning. The uprobe_consumer structure tells the kernel what is to be done when a process encounters the probe; it looks like:
struct uprobe_consumer {
int (*handler) (struct uprobe_consumer *self, struct pt_regs *regs);
bool (*filter) (struct uprobe_consumer *self, struct task_struct *task);
struct uprobe_consumer *next;
};
The filter() function is optional; if it exists, it determines whether handler() is called for each specific hit on the probe. The handler returns an int, but the return value is ignored in the current code. Since probes are associated with files, they affect all processes that run code from those files. A special copy is made of the page to contain the probe; in that copy, the instruction at the specified offset is copied and replaced by a breakpoint. When the breakpoint is hit by a running process, filter() will be called if present, and handler() will be run unless the filter said otherwise. Then the displaced instruction is executed (using the "execute out of line" mechanism described in this article) and control returns to the instruction following the breakpoint. Uprobes thus implements a mechanism by which a kernel function can be invoked whenever a process executes a specific instruction location. One could imagine a number of things that said kernel function could do; there has been talk, for example, of using uprobes (and, perhaps someday, something derived from utrace) as a replacement for the much-maligned ptrace() system call. Tools like GDB could place breakpoints with uprobes; it might even be possible to load simple filters for conditional breakpoints into the kernel, speeding their execution considerably. Uprobes could also someday be a component of a Dtrace-like dynamic tracing functionality. For now, though, the interfaces for that kind of feature have not been added to the kernel; none have even been proposed. What the current implementation does have is integration with the perf events subsystem. New dynamic "events" can be added to any file location via an interface similar to that used for dynamic kernel tracepoints. In particular, there is a new file called uprobe_events in the tracing directory (/sys/kernel/debug/tracing/ on most systems) that is used to add and remove events. As an example, a line like:
echo 'p:bashme /bin/bash:0x4245c0' > /sys/kernel/debug/tracing/uprobe_events
would place a new event (called "bashme") at location 0x4245c0 in
the bash executable. The event would then appear with all other
events in /sys/kernel/debug/tracing/events, in the
uprobes subdirectory. Like other events, it is not actually
turned on until its enabled attribute is set. See Documentation/trace/uprobetracer.txt for
details on the interface at this level.
Placing uprobes is, by default, a privileged operation requiring the CAP_SYS_ADMIN capability. One can remove the privilege requirement by setting the perf_paranoid sysctl knob to -1, but doing so will allow the placement of dynamic tracepoints anywhere in the system, in kernel or user space. Thus, one need not be overly paranoid to leave perf_paranoid at its default setting. The perf tool has been enhanced to make working with dynamic user-space tracepoints easy. One can, for example, set a tracepoint at the entry to the C library's malloc() implementation with:
perf probe -x /lib64/libc.so.6 malloc
That tracepoint can then be treated like any other event understood by perf. See the explanatory text from Ingo Molnar's pull request for examples of what can be done. Most kernel patches are conceived, implemented, reviewed, and merged into the mainline over a fairly short period of time. But some of them seem to languish for years without making much progress. Uprobes was such a patch set. It must have been frustrating for the developers to keep revising and posting this code, only to see it shot down over and over again. But the kernel community can be supportive of developers who show both persistence and a willingness to listen to criticism. The result, in this case, is a user-space probing mechanism that has been simplified, made more robust, and integrated into the existing events infrastructure. Hopefully it was worth the wait.
Atime and btrfs: a bad combination?Unix and Unix-like systems have traditionally recorded the time of last access for each file in the system. This practice has fallen partially out of favor over the last decade for a simple reason: writing the last-accessed time ("atime") takes up a lot of I/O bandwidth when lots of files are being read; see this article from 2007, for example. The worst of the atime-related problems have long since been mitigated by moving to the "relatime" mount option by default; relatime only updates atime a maximum of once per day for unchanging files. But now it seems that atime recording can be especially problematic with the btrfs filesystem, and relatime may not help much.One of the core design features of btrfs is its copy-on-write nature. Blocks on disk are never modified in place; instead, when it becomes necessary to commit a change, the affected block is copied and rewritten into a new location. Copy-on-write applies to metadata as well as data; if a file's metadata (such as its last-accessed time) is changed, the block containing that metadata will be copied to a new spot. So, on btrfs, an operation that reads a lot of files (creating a tar archive, say, or a recursive grep) can, through atime updates, cause the copying and rewriting of a lot of metadata blocks. Needless to say, performance is not improved this way, but that is not where the big problem comes in. As Alexander Block pointed out, the real problem has to do with the interaction between atime, copy-on-write, and snapshots. Btrfs provides a fast snapshotting feature that can create a copy of the state of the filesystem at a specific time. When a snapshot is created, it shares all data and metadata with the "trunk" filesystem. Should a file be changed, the resulting copy-on-write operation separates the trunk from the snapshot, keeping both versions of the data available. So snapshots can be thought of as being nearly free as long as the filesystem remains relatively quiet. Snapshots will share data and metadata, so they do not require a lot of additional space. Atime updates change the situation, though. If somebody takes a snapshot of a filesystem, then performs a recursive grep on that filesystem, the last-access time of every file touched may be updated. That, in turn, can cause copy-on-write operations on each file's inode structure, with the result that many or all of the inodes in the filesystem may be duplicated. That can increase the space consumption of the filesystem considerably; Alexander posted an example where a recursive grep caused 2.2GB of free space to disappear. That is a surprising result for what is meant to be a read-only operation. Once upon a time, when disk capacities were measured in megabytes, it was said that the only standard feature of Unix systems was the message of the day telling users to clean up their files. Atime was often used by harried system administrators trying to recover some disk space; they would scan for infrequently-accessed files and, depending on how desperate the situation was and how powerful their users were, either send lists of unused files to users or simply back those files up to tape and delete them. It is somewhat ironic that a feature meant (among other things) to help keep disk space free has now, on next-generation filesystems, become part of the problem. It's worth noting that the relatime option (which only updates atime once per day unless the file has been modified since the last atime update) is of little help here. It only takes one atime update to force an inode to be rewritten and unshared with any snapshots. So the fact that such updates are limited to one per day offers little in the way of consolation. Users are also unlikely to be consoled by one other aspect of the problem pointed out by Alexander: since reading data can consume space in the filesystem, read operations might fail with "no space available" errors on an overflowing filesystem. That may make it difficult or impossible to fix the problem by copying data out of a full filesystem. By the time that happens, a typical user could be forgiven for thinking that, perhaps, they don't need last-accessed time tracking at all. Along those lines, Alexander suggested that it might be a good idea to default to "noatime" (which turns off atime recording entirely) for btrfs mounts, even if that means that btrfs would then behave differently than other filesystems. That idea was quickly shot down for a simple reason: there are still applications that actually need the atime information to function correctly. The classic example is the mutt email client which, in the absence of atime, cannot tell whether a mailbox contains unread mail. Programs that clean up temporary directories (tmpreaper or tmpwatch, for example) will fail without atime. There are also hierarchical storage systems that, like the Unix system administrator of old, use atime to determine when to move files to slower storage. So atime needs to stick around, lest users run into a different kind of unpleasant surprise. For now, the only recourse for users who run into (or are worried about) this problem is to explicitly mount their filesystems with the "noatime" option. Further ahead, it might be possible to make some tweaks to btrfs to mitigate the problem; Boaz Harrosh suggested disabling atime updates when the free space falls below a certain threshold or moving the atime data into a separate data structure. But nobody appears to be working on such solutions now. So it may be that, as usage of btrfs grows, users will occasionally be surprised that reading a file can consume space in their filesystems.
Patches and updates Core kernel code
Development tools
Device drivers
Documentation
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Miscellaneous
Page editor: Jonathan Corbet Distributions Temporary files: RAM or disk?Temporary files on Linux have traditionally been written to /tmp, at least those that don't need to persist across boots. Several Linux distributions are now planning to mount /tmp as a RAM-based tmpfs by default, which should generally be an improvement in a wide variety of scenarios—but not all. Debian has been planning to make that switch for the upcoming "wheezy" (7.0) release, but as a discussion on debian-devel shows, not all are happy with that decision. Mounting /tmp on tmpfs puts all of the temporary files in RAM. That will reduce the amount of disk I/O that needs to be done, as the filesystem never actually touches the disk unless there is memory pressure. In that case, the tmpfs memory could get swapped out like other pages in the system, but in many cases a temporary file will be created without needing any disk I/O. The installer (or system administrator) can specify a maximum size for the filesystem as part of the mount options, but the memory is only actually used if files are written to it. The latest incarnation of the dispute over putting /tmp into RAM (vs. having it as a disk directory in / or its own partition) was started by a posting from "Serge" that claimed the change was "useless". His complaint stemmed from various applications that put large files into /tmp (long videos, ISO images, unpacked archives, sort temporary files, etc.), which can cause problems for systems with little memory or with too small of a tmpfs. He argued that putting /tmp on tmpfs by default was the wrong choice for new users, and that savvy users could make the switch (or know enough to choose it at install time). It is clear that there is a difference of opinion among participants in the thread about how /tmp should be used. Several people thought that /var/tmp should be used for large temporary files, with /tmp reserved for small files. Others pointed out that the file hierarchy standard (FHS) just says that /tmp is for files that do not need to persist across reboots, while /var/tmp is for those that do. But using /var/tmp for non-persistent large files has a drawback: those files do not get cleaned up on boot, unlike /tmp files—at least on Debian. Many thread posters have been running their systems with a RAM-based /tmp for a long time with few or no issues, while others are clearly running into problems in that configuration. Large videos downloaded via the Flash plugin seem to be a particular problem, but there are others. It really comes down to a question of what /tmp is for. Many in the long thread—something of a Debian tradition—believe that writing large files to /tmp is the wrong thing for an application to do. But no real alternative location that preserves the "wipe on reboot" semantics for /tmp has been offered. Workarounds like the TMPDIR environment variable can be used, but whatever directory that points to may just fill up with garbage over time. Running out of /tmp space is a problem regardless of what kind of filesystem lies under it, but in the default Debian installation a disk-based /tmp will essentially have the whole disk available, as it only creates a single / partition. On the other hand, a RAM-based /tmp will almost certainly be much smaller, which can lead to applications filling up the filesystem much more easily. Even if those applications are "wrong" to do so, they evidently exist, so forcing users to confront that problem at times could be sub-optimal. More advanced users can make their own choice. There were numerous invocations of Solaris in the thread as well, because it has used a RAM-based temporary filesystem for many years, seemingly without many problems. Russ Allbery said that he started in the "Solaris has been doing this for years, what's the problem?" camp, but after reading some of the objections in the thread had basically changed his mind. It comes down to a question of functionality vs. speed. For a default setting, working should be preferred over speed optimization. As Joey Hess pointed out, until there are clear numbers about the performance improvement that comes from a RAM-based /tmp, making it the default is a premature optimization. There were examples offered where tmpfs made an enormous performance difference, but the question is not whether the feature is useful; it is, instead, whether it should be the default. This is not the first time the issue has come up. Roger Leigh posted pointers to other threads and bug reports (some of which have their own long threads, of course). He is the developer that added the tmpfs-based /tmp for wheezy, but he mostly stayed clear of the discussion this time. It does not look like he is inclined to remove the default, though, so there have been suggestions that the issue be referred to the technical committee. It is interesting to note that Fedora plans to move to a RAM-based /tmp for Fedora 18, and has already enabled it for Rawhide. The Fedora feature page notes that Solaris has been doing so since 1994 and that Arch currently defaults to a tmpfs-based /tmp. It also mentions that Debian and Ubuntu both plan to move in that direction. At least for the near future—and probably beyond—RAM sizes will generally be far less than those of disks, particularly for machines targeted at non-technical users. While those users might benefit from the performance improvements that come from keeping /tmp in RAM, there is a risk that their applications will chew up their RAM with huge temporary files, leading to swap storms or broken applications. The Solaris example may not be all that compelling as it is not really a consumer-oriented desktop system. The other Linux distributions may offer a better test case and Debian may well get the benefit of seeing what happens with them before wheezy is completely frozen. On the other hand, though, it is not really clear that Debian targets (or attracts) many novice users, so why make the vast majority of Debian users suffer the penalty of disk-based /tmp when they don't really need to? They can certainly switch to that, but the default might serve the majority quite well. At this point, one suspects that RAM-based /tmp will soon be the norm and that applications will get "fixed", but only time will tell.
Brief items Distribution quotes of the week
If you're interested in working on it I can imagine a number of places
to shove this data and I know what's necessary from the rpmdb/yum
history/repos to do it.
-- Seth Vidal
How can 'opening a big .ppt I was just sent', be a corner case? That's
the sort of thing 'normal people' do on a daily basis.
-- Wookey
And if we have any pretence of Debian being useful outside the people on this list we really should have defaults which mean it 'just works' (if your disk isn't full already).
Fedora 17 releasedThe Fedora 17 release is out. "Frankly, we believe this is the beefiest release ever -- chock full of condiments, more commonly known as Features, to customize your experience to your tastes. We take pride in our toppings, and in our fine ingredients; Fedora 17 includes both over- and under-the-bun improvements that show off the power and flexibility of the advancing state of free (range) software." Details can be found in the Fedora 17 feature list.
Fedora 17 ARM Beta ReleaseA Fedora 17 beta for ARM is now available. There are a number of images provided for various targets ("QEMU, Trimslice, Beagleboard XM and iMX based hardware platforms.") "We invite you to take part in making Fedora 17 for ARM a solid release by downloading, testing, and providing your valuable feedback. Please join us on the IRC in #fedora-arm on Freenode or send feedback and comments to the ARM mailing list."
Fedora 15 end of lifeNow that Fedora 17 is out, Fedora 15 is approaching its end-of-life. There will be no more updates for Fedora 15 after June 26.
Distribution News Debian GNU/Linux bits from the NM process: advocacy, no more AM reports, AMs neededEnrico Zini reports on some changes in the Debian New Maintainer process. "AM (Application Manager) reports were a bit of a burden of bureaucracy, and it has been the case that applications got stuck for considerable time waiting for the AM to have the time to write the report. Let us fix that: forget about writing the AM reports!"
Fedora Cooperative Bug Isolation for Fedora 17The Cooperative Bug Isolation Project (CBI) is now available for Fedora 17. The CBI is a research effort designed to find out what went wrong when your application crashes. "We currently offer instrumented versions of Evolution, The GIMP, GNOME Panel, Gnumeric, Liferea, Nautilus, Pidgin, and Rhythmbox."
Newsletters and articles of interest Distribution newsletters
Precision and purpose: Ubuntu 12.04 and the Unity HUD reviewed (ars technica)Ryan Paul takes a look at Unity and the Heads-Up Dispaly (HUD) in the 12.04 release. "The HUD is powered by the same mechanism that supports Unity’s global menu system. In order to make an application’s menus available to Unity so that it can display it in the panel, the contents of the menu are relayed over the D-Bus interprocess communication system. The HUD taps into that data, essentially offering an alternate presentation of the menu. In addition to being extremely useful, the HUD is a compelling illustration of how Unity’s architectural strengths can be put to use in novel ways."
What's new in Fedora 17 (The H)The H has a nice summary of the new features in Fedora 17. "Of the many changes made, the two that stand out are software rendering for GNOME Shell and the sandbox function for isolating applications. These, and many other changes, are likely to find their way into other distributions soon. Time will tell whether that will also be the case for the much-discussed filesystem reorganisation."
SUSE Turns 20, Ascends to the Cloud (Linux.com)Linux.com has an interview with SUSE's Vice-President of Engineering Ralf Flaxa. "Linux.com: What’s the significance of 20 years in business? Ralf Flaxa: The significance comes in our longevity, which even surprises us to a degree. In fact, our first reaction was “Hey, 20 years!” There have been many other distributions that have come and gone – we were one of the first and we’re still here. We view the staying power as very significant."
Page editor: Rebecca Sobol Development Clustering, development, and galactic conquest at PGCon 2012Every year, the PostgreSQL community has a developer meeting in Ottawa called PGCon. This year's PGCon was the largest yet, with 210 contributors and users attending. As usual, though, the most interesting developments happened outside the regular conference sessions. This year's events included new replication and clustering projects, changes to the PostgreSQL development process, and a battle for galactic conquest. PostgresXC 1.0 betaThere were two major announcements made at the one-day Postgres Clustering Summit before the main conference. The PostgresXC project announced its 1.0 beta release, and 2ndQuadrant announced its plans for a new replication method for PostgreSQL. The Clustering Summit, a meeting of developers working on replication and database clustering for PostgreSQL, was sponsored by NTT Open Source of Japan. PostgresXC is a project to build a write-scalable cluster for PostgreSQL. A PostgresXC cluster is intended to be a co-located group of four to twenty servers, and is designed to be able to handle larger numbers of database connections and concurrent write queries than a single server. The goals of the project are to build something which satisfies a lot of the same use cases as Oracle Real Application Clusters (RAC), Oracle's database clustering technology. PostgresXC implements a shared-nothing architecture instead of RAC's shared-storage architecture in order to survive a larger variety of hardware failure situations. As of the 1.0 beta release, PostgresXC supports most PostgreSQL syntax and functionality for a small cluster of database servers so you can theoretically deploy most PostgreSQL-based applications on PostgresXC. Notable limitations with 1.0 beta are the lack of triggers and savepoints, as well as the inability to add new nodes without rebuilding the cluster. These issues will be addressed in 1.1 and later versions. One of the primary differences between regular PostgreSQL and PostgresXC is the notion of distributed tables. Each table you create can be distributed across the cluster based on round-robin, a hash, or a column value, instead of being replicated to all cluster nodes. This allows the database cluster to support more data than any single node can. The PostgresXC project primarily consists of developers from NTT Open Source and EnterpriseDB, and it is licensed under The PostgreSQL license. Once PostgresXC is more stable, expect rapid adoption due to the needs of users for bigger, more fault-tolerant database servers. New "logical" replication projectThe second replication announcement was 2ndQuadrant's plan to create a new replication technology for PostgreSQL, which they call "Bi-Directional Replication", mostly to distinguish it from other forms of replication. They have secured sponsorships from companies like McAfee, have begun work on the patches, and plan to get the feature done for PostgreSQL 9.3 in 2013. 2ndQuadrant is a PostgreSQL consulting company headquartered in the United Kingdom, lead by committer Simon Riggs. At an evening pizza session, they explained how the new replication will work. PostgreSQL's streaming binary replication, introduced in version 9.0, allows a replication server to receive copies of data pages over a normal PostgreSQL database connection (usually port 5432). This connection is called the "walsender" and its associated listener process on the replica is called the "walreceiver" (WAL stands for Write-Ahead Log). This works well for whole-database replication, but does not permit users to replicate only specific parts of the database, or to run any kind of write queries on the replicas. As a result, other types of replication are also needed. The new feature will allow clients to choose to receive either replicated query statements or changed rows over the same walsender connection, so that listeners can filter or modify the replication stream. More importantly, it would allow a replica to create database changes as well as receiving them, provided that conflicts were handled by database or middleware code, thus implementing one form of multi-master replication for PostgreSQL. This type of replication configuration is already possible with replication tools such as Slony-I, Londiste, and Bucardo. However, all of those tools suffer from administration and performance overhead which many users find unacceptable because they are implemented in Postgres user space instead of in the Postgres backend. Bi-Directional Replication is expected to be more efficient, and thus useful to a wider array of users. It may even become the underlying data transport mechanism for future versions of the older replication systems. Bi-Directional Replication is also intended to solve a different use-case than PostgresXC. While PostgresXC is designed to provide scalability and consistency for database writes within a single data center, the new replication is intended to support asynchronous, geographically distributed replication, at the expense of consistency. Such "eventual consistency" systems have become popular among web startups recently, and PostgreSQL users are no exception. CommitFests and the PostgreSQL development processOne of the other meetings at PGCon was the annual Developer Meeting, at which 20 of the largest code contributors to PostgreSQL meet in person and discuss current and future development of the database. At the Developer Meeting three years ago, the PostgreSQL project adopted the CommitFest model for managing patch review and annual development. At this one, the group reviewed how the CommitFest process was going, and decided to make some minor changes to try to improve it. The central idea of the CommitFest model is to completely clear out the queue of pending patches every two months, in order to prevent catastrophic levels of unreviewed patch buildup. CommitFests have allowed the project to accept a large number of patch submissions and make sure that everything submitted got a serious review. At the developer meeting, contributor Noah Misch said that the guarantee of code review was one of the things which attracted him to the PostgreSQL project in the first place. Unfortunately, over the last three releases, the last of the four CommitFests have been getting longer: one month for 9.0, two months for 9.1, and three for 9.2. Contributing to this problem is that the number of patch reviewers has not been growing as fast as the number of submissions, leading to a risk of reviewer burn-out. The main reason the final CommitFest does not finish on schedule has been large, complex, not-quite-ready patches which need extensive review by core committers. In order to address this, PostgreSQL will add a planning period after the third, or penultimate, CommitFest in order to separate pending patches into large or small, and ready for committers or not quite ready. Then, in the third week of the final CommitFest, the team will be able to do a final triage of what's ready for release and what's not, and conclude the CommitFest after the expected 30 days. Another cause of contributor pain has been submitting new features without an agreed specification, often causing those patches to need complete recoding. In order to add a more formal process around specification review, the project will be adding a "design specification" section to the CommitFests and encouraging people to submit specifications for review. The developers also adopted a "one patch, one review" policy, which means that patch contributors are obligated to review at least one patch contributed by someone else. Conquering the galaxy with PostgreSQLAs a new event this year, PGCon included a Schemaverse tournament. Schemaverse is a text-mode galactic conquest wargame, played entirely in a PostgreSQL database. Users issue commands to build and move spaceships, engage in battles, and conquer and mine planets, all in the form of SQL queries. You win the game either by conquering the most planets, or by successfully hacking the database. Schemaverse was invented by Josh McDougall as a fun data security exercise for the DEFCON 2011 security conference. Notably, no player at DEFCON, PGCon, nor in the open online version has yet managed to win by hacking the PostgreSQL database. The game also demonstrates how PostgreSQL allows sophisticated application functionality to be built either inside or outside the database. The PGCon tournament, which ran for the entirety of the conference, was won by Chris Browne of the domain registrar Afilias. Chris spent weeks leading up to PGCon honing automated "artificial intelligence" scripts to control his fleet of spaceships, and rapidly seized over 500 planets. He won several prizes including a championship sweatshirt. Other conference highlightsPeter van Hardenberg, database lead at platform-as-a-service company Heroku, delivered the keynote for PGCon. His talk was about Heroku's users, the cloud, NoSQL, and the future of PostgreSQL. First he discussed the large number of PostgreSQL databases (1.6 million) belonging to Heroku's hundreds of thousands of users, who are mostly small-team agile-development web application developers. Van Hardenberg appealed to the PostgreSQL developers to keep Heroku's users in mind when working on PostgreSQL and new features, particularly emphasizing the need for usability improvements. In case studies at the conference NASA scientists talked about using Postgres for data collection on the Aura spacecraft for ozone monitoring. 2ndQuadrant staff presented migrating top website Fotolog from MySQL to PostgreSQL. A contributor to open source library management system Evergreen demonstrated their application and explored its data architecture. And Mozilla staff talked about the Socorro crash collection system. Other talks focused on many of the features and internals of PostgreSQL 9.2, currently in beta. This included Range Types, the new replication protocol, and the internals of how many of the performance improvements were implemented. As they did last year, PostgreSQL's Russian development team introduced another new indexing idea, this time for similarity searching. PGCon 2012 was a busy, intense week for PostgreSQL developers. Users can expect a lot more great things coming out of the project because of it. If your job centers around PostgreSQL, you should consider attending next year. Later this year, you can attend Postgres Open, a business and user-oriented conference in Chicago, or PostgreSQL Europe in Prague.
Brief items Quote of the week
Keep in mind that Free software can be sold, and on devices such as
these getting software that is tested and well packaged is a
service much like bottling water. Yes, water can be had at no cost,
but the convenience and safety of bottled water makes it something
you can sell. So while we are shipping a ton of great Free software
applications by default as part of Plasma Active (and therefore the
Vivaldi tablet) so that it is useful out of the box, we are also
going to be encouraging developers of Free software to think about
putting a small price tag on their applications in the catalog.
— Aaron
Seigo
GCC Explorer - an interactive take on compilationMatt Godbolt announces GCC explorer, a web-based tool for exploring how code tweaks change the machine code emitted by the compiler. "Particularly with some of the newer features of C++11 — lambdas, move constructors, threading primitives etc — it’s nice to be able to see how your elegant code becomes beautiful (and maybe even fairly optimal) machine code." The GCC explorer code is on github for those who want to set up their own instance.
LibreOffice 3.5.4 releasedThe LibreOffice 3.5.4 release is out; this is a minor update but it may be of interest to those interested in better performance: "LibreOffice 3.5.4 is the fastest version of the best free office suite ever, with up to 100% performance gains when opening large files (depending on operating system, hardware configuration and file contents)."
RPM 4.10 releasedPanu Matilainen announces the release of RPM 4.10.0. Most of the changes targeted robustness and correctness, but a few new features crept in as well, including support for parsing the tilde (~) operator in package version numbers.
GNU SASL 1.8.0 releasedVersion 1.8.0 of the GNU Simple Authentication and Security Layer (SASL) is out. New features include SAML20 (RFC 6595) and OPENID20 (RFC 6616) support, along with the addition of a number of SMTP server examples.
systemd 183 releasedSystemd 183 is out; this is a major release with a lot of new stuff; use on production machines is not recommended. "Note that we skipped 139 releases here in order to set the new version to something that is greater than both udev's and systemd's most recent version number." Changes include the incorporation of udev, the renaming of various configuration files and commands, a change to the LGPLv2.1+ license, better suspend and hibernate support, offline system update support, hardware watchdog support, /tmp as tmpfs by default, and lots more.
Newsletters and articles Development newsletters from the last week
The Design of LLVM (Dr. Dobb's)Dr. Dobb's has a moderately detailed overview of the design of LLVM by Chris Lattner. "In particular, LLVM IR is both well specified and the only interface to the optimizer. This property means that all you need to know to write a front end for LLVM is what LLVM IR is, how it works, and the invariants it expects. Since LLVM IR has a first-class textual form, it is both possible and reasonable to build a front end that outputs LLVM IR as text, then uses UNIX pipes to send it through the optimizer sequence and code generator of your choice. It might be surprising, but this is actually a pretty novel property to LLVM and one of the major reasons for its success in a broad range of different applications. Even the widely successful and relatively well-architected GCC compiler does not have this property: its GIMPLE mid-level representation is not a self-contained representation."
Page editor: Jonathan Corbet Announcements Brief items SFC expands compliance efforts to Samba and the Linux kernelThe Software Freedom Conservancy has announced that the Samba project (which, like BusyBox, is another of SFC's member projects) will be engaging in license compliance efforts with the organization. In addition, it announced a new "GPL Compliance Project for Linux Developers" that is working with seven Linux kernel copyright holders to work on compliance for their copyrights in the kernel. "Matthew Garrett, an accomplished Linux kernel developer, was the first to put forward his copyrights as part of the GPL Compliance Project for Linux Developers, and was quickly joined by six other individuals. In a statement today, Matthew noted: 'For some time, many Linux kernel copyright holders have offered our moral support to the BusyBox enforcement efforts through Conservancy, but didn't have the ability to formalize that support. I'm glad to put my copyrights forward for this new initiative, and welcome any Linux kernel copyright holders who wish to join us to reach out to Conservancy via <compliance@sfconservancy.org>.'"
The Linux Foundation Announces New Tool for Tracking Free and Open Source Software ComponentsThe Linux Foundation has announced the release of The Linux Foundation FOSS Bar Code Tracker. "Released as an open source project under the MIT license, the new software tool aims to simplify the way open source components are tracked and reported by using an auto-generated, custom QR code for each product. The QR code contains important information on the Free and Open Source Software (FOSS) stack contained in a product, such as component names, version numbers, license information and links to download the source code, among other details."
Judge Alsup Rules: Oracle's Java APIs are Not Copyrightable (Groklaw)Groklaw has the news that the judge in Oracle v. Google has ruled that the Java APIs are not copyrightable. "The overall name tree, of course, has creative elements but it is also a precise command structure — a utilitarian and functional set of symbols, each to carry out a pre-assigned function. This command structure is a system or method of operation under Section 102(b) of the Copyright Act and, therefore, cannot be copyrighted. Duplication of the command structure is necessary for interoperability." That pretty much makes the case a total loss for Oracle, but, of course, appeals are possible.
Lib-Ray: Non-DRM Open-Standards HD Video FormatTerry Hancock launched a kickstart campaign to fund his work on Lib-Ray, a non-DRM open-standards HD Video Format. Unfortunately we did not hear about it in time for last week's edition and the campaign closes June 3. Still, it sounds like an interesting project and other contributions may still be possible. "Unlike Blu-Ray, Lib-Ray releases do not support DRM, encryption, or region-coding options, and are intended for worldwide release. Thus the standard is designed with a highly-adaptable localization scheme, providing many more subtitles than are typically available on Blu-Ray or DVD regional releases."
Linux Tycoon [and more] going Open Source via donationsBryan Lunduke, author of the game "Linux Tycoon" and the visual programming suite "Illumination Software Creator" is moving his software to the GNU Public License and is looking for donations to fund this work. "During this process I am making regular (multiple times per day) updates and trying to keep everything documented and transparent. In this way I hope to provide a solid case-study on moving a proprietary software business to Open Source -- In the hope that others can follow and do the same."
Articles of interest Google launches Chromebook, Chromebox and gets it right (GigaOm)GigaOm reviews the new Chromebook from Samsung, along with the associated desktop "Chromebox" device. "The ChromeOS and the devices based on the OS have reached a point in maturity where they can be used as an 'optional' or second computer. It is also benefiting from the fact that most of us have become used to living and working inside the browser."
Moonlight sent into twilight (The H)The H looks into the death of Moonlight, a free software implementation of Microsoft's Silverlight platform. "Microsoft has been playing down Silverlight over the last year and a half, after previously promoting it as strategically important. In Windows 8, Silverlight is taking a back seat to HTML5 in the browser and for applications. Without Microsoft's weight behind Silverlight, the future for Moonlight has looked less and less promising."
This Cadillac Is Powered by Linux (Wired)Wired is impressed with the Linux-powered in-vehicle infotainment (IVI) system in the most recent Cadillac XTS. "While the XTS’ spate of processors and controllers isn’t running the open source offspring of Linus Torvalds, the game-changing infotainment intender known as the Cadillac User Experience (CUE) is. [...] Buried deep within the dash is a three-core ARM 11 processor, powering two displays: one eight-inch capacitive touch screen — the first non-resistive display to come to a production car — and a second, 12.3-inch fully configurable instrument cluster mounted behind the steering wheel. Two of those cores adapt on the fly to handle voice commands powered by the same Nuance technology used by many automakers, along with Apple’s personal assistant, Siri. But with CUE, everything is processed on board."
New Books Hadoop: The Definitive Guide, 3rd Edition--New from O'Reilly MediaO'Reilly Media has released "Hadoop: The Definitive Guide, 3rd Edition" by Tom White.
Calls for Presentations 2nd Call For Papers, 19th Annual Tcl/Tk Conference 2012The Tcl/Tk Conference will take place November 12-16, 2012 in Chicago, Illinois. The call for papers will be open until August 27.
Upcoming Events LUGOD presentation: "Wikimedia and Wikipedia"The Linux Users' Group of Davis (LUGOD) will have a presentation on "Wikimedia and Wikipedia" by Trevor Parscal and Roan Kattouw from the Wikimedia Foundation at their June 18 meeting in Davis, California.
PyCon Australia 2012 and Google Australia announce gender diversity grantsPyCon Australia and Google Australia have announced that they are joining forces to offer gender diversity delegate grants to women who wish to attend PyCon Australia in 2012. "These grants will cover up to $AUD500 of travel, accommodation and registration costs for women living outside of the Southern Tasmania region to attend this year's conference." Applications for the gender diversity delegates grants will close on June 15. PyCon AU takes place August 18-19, 2012 in Hobart, Tasmania.
PyCon DE 2012 - Registration OpenPyCon DE will take place October 29 - November 3, 2012 in Leipzig, Germany. Early bird registration is open until the end of June. The call for talks and tutorials is still open as well.
Linux Color Management Hackfest 2012There will be a Linux Color Management Hackfest November 16-19, 2012 in Brno, Czech Republic. Some funds are available to sponsor a limited number of people. Applications must be submitted by September 30.
Events: June 7, 2012 to August 6, 2012The following event listing is taken from the LWN.net Calendar.
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Monitor before profiling]](/images/2012/05-colorhug-before-sm.jpg)
![[Monitor after profiling]](/images/2012/05-colorhug-after-sm.jpg)