LWN.net Weekly Edition for March 22, 2012

Fedora mulls ARM as a primary architecture

The ARM architecture is growing in popularity and is expected to expand its reach beyond the mobile and "small embedded" device space that it currently occupies. Over the next few years, we are likely to see ARM servers and, potentially, desktops. Fedora has had at least some ARM support for the last few years, but always as a secondary architecture (SA), which meant that the support lagged that of the two primary architectures (32 and 64-bit x86) of the distribution. Recently, though, there has been discussion of "elevating" ARM to a primary architecture (PA), but, so far, there is lots of resistance to a move like that.

The subject came up at a meeting of the Fedora engineering steering committee (FESCo) on March 19. Adding ARM as a primary architecture for Fedora 18 was a late addition to the agenda, which annoyed some, but the discussion was largely to "start the ball rolling and collect feedback from everyone", as Kevin Fenzi put it. There will be many other opportunities to discuss the idea, he said. The meeting log bears that out as the only vote taken (or even proposed) was to ask for input from various teams (QA, release engineering, kernel, and infrastructure) about the impact of a change like that.

The difference between primary and secondary architectures for Fedora is rather large. Releases cannot be made without all of the packages building and working for each primary architecture, whereas secondary architecture packages can languish. In fact, the current release of Fedora for ARM is based on Fedora 14—though there are alphas of Fedora 15 and 17—which is past its end-of-life on x86.

The meeting discussion focused mostly on the motivation for making ARM a PA and why the project's goals couldn't be met, at least for now, by remaining as an SA. Much of the motivation, it would seem, is for Fedora to get out ahead of the curve on ARM support. Making ARM a "first class citizen" would increase its visibility and put the full weight of the Fedora community behind the effort. Therein, it seems, lies part of the problem.

One could argue that Fedora has already fallen behind the curve with respect to ARM given what Ubuntu and Debian are doing to support the architecture. There is a lot going on with Linux on ARM, and Fedora may well find itself becoming less relevant if support for ARM does not improve. But the question seems to be whether that support needs to improve as an SA before even considering whether it can be a new PA.

Based on the discussion in the meeting, Matthew Garrett posted an RFC draft of the requirements to promote an architecture to a PA. So far, there has been no architecture that has transitioned from an SA to a PA, so some kind of ground rules need to be established. Garrett lists seven potential criteria, prefaced by:

Much of the response to that posting concerns the amount of time it (currently) takes to build ARM packages (vs. the time for x86 and other architectures). Jakub Jelinek noted that GCC builds for 64-bit x86 are on the order of two hours, while building for ARM takes much longer. He followed that up with actual numbers from GCC 4.7 builds for Fedora 17, which ranged from one-and-a-half hours for x86_64 to more than 26 hours for armv5tel (and more than 24 for armv7hl). Brendan Conoboy pointed out that plans for newer "enterprise hardware" would cut those ARM build times in half, but that still leaves a substantial gap.

Slow builds are not just an annoyance, as there are some impacts for the distribution if package building takes "too long". Josh Boyer lists two. If a package builds for the x86 family, but then fails to build for ARM, the x86 build will have to be resubmitted after the ARM problem is fixed. In addition, when trying to do an update (for a security issue for example), it has to wait for the slowest build to finish before the update can go out. Adam Williamson also notes another problem that could arise in the release verification process:

Build speed is a technical issue that can presumably be overcome (eventually) with faster hardware. Other possibilities like cross-compiling on faster x86_64 servers or parallelizing the Koji build system (perhaps using something like distcc) seem to have been ruled out by Fedora release engineering or the Fedora ARM team. While some remain unconvinced, Conoboy is adamant that cross-compilation is not a good solution:

But some question the wisdom of even having criteria for promoting SAs to PAs, whether it makes sense for Fedora to even consider ARM for a PA, or both. Kevin Kofler is definitely in the last category as he believes that the current list of PAs "should be set in stone unless a MAJOR change in hardware landscape happens". Some would argue that the change is already happening. But he is concerned that additional PAs put a burden on all of the package maintainers, so that it should require an extraordinary event (like "x86 gets discontinued by the hardware manufacturers and everyone uses ARM instead") before any change like that is even considered. He continues:

Kofler harps on the same points throughout the thread, belittling the ARM market share (at least in the market segments that he thinks should be targeted) and finding the build times for ARM packages to be untenable. He considers the large existing base of ARM devices to be unsuitable for installing Fedora, at least at this point. But, as Richard W.M. Jones points out, that is changing rapidly:

Others were also skeptical of the current ARM hardware being a good target for Fedora, but Williamson points out that getting Fedora ARM running does more than just target those devices. The ARM project is looking toward the future, both on servers and mobile devices. Getting the distribution running on one is a big step toward having it available for the other.

But the speed of the build system is just one symptom of the problems that another PA will bring. One of the bigger questions, which remains largely unanswered as yet, is what making ARM a PA would do for Fedora as a distribution. It's reasonably clear why it would help the Fedora ARM project to have ARM as a primary, but the advantages to the distribution, at least at this point, are less clear. As Garrett put it:

Peter Jones Robinson outlined many of the advantages that the Fedora ARM team sees in another fedora-devel thread. Essentially, it would spread the load of responsibility throughout the Fedora community. That is, of course, the underlying concern of many posters in the threads. But Jones sees it this way:

There is, of course, nothing stopping the ARM team from achieving most of its goals while staying as secondary architecture. It will be more difficult and likely require more volunteers, but the Fedora project as a whole needs to be convinced of the advantages of taking on the "burden" of ARM as a primary. So far, the ARM project doesn't seem to have made a convincing case for that, but, given the importance of the architecture going forward, one might guess that the situation may change in the next year or two. In the meantime, setting some goal posts for any secondary architecture that wants to be promoted seems like a good first step.

Comments (60 posted)

Mozilla reconsiders H.264

Mozilla raised eyebrows in mid-March when a patch materialized that would allow Gecko to fall back on operating system or hardware media decoders for multimedia content — in particular for patent- and royalty-encumbered codecs like H.264, which are not supported natively in Gecko. The project had fought hard to promote the adoption of unencumbered alternatives (such as Ogg Theora or Google's WebM), so many on the mozilla.dev.platform discussion group saw enabling any support for H.264 as a violation of principle. Mozilla's Chief Technology Officer Brendan Eich argued, however, that the decision is an improvement over the existing Flash-fallback method, that Mozilla has more important fights to focus on, and that the blame for WebM's snail-paced adoption lies squarely at the feet of Google.

History of the WebM, part 1

Eich posted his take on the situation in a round-up at his own blog (which was then syndicated at the official Mozilla Hacks blog), which started with a history lesson on H.264, WebM, and the HTML5 <video> element. As far back as 2007, Eich and Mozilla had argued for the standardization of the <video> and <audio> elements to include "unencumbered" baseline codecs — at the time, Ogg Vorbis for audio and Ogg Theora for video. Eich argued that the term "unencumbered" most accurately describes the state of the codecs needed to ensure that the open web remains open; the issue is not about open source (for there are open source implementations of encumbered codecs), nor is it about patents (for standards bodies can and will accepted a patented codec if the patent holders agree to license it under royalty-free terms).

In 2007's <video> element fight, the main opponent of Theora was H.264, which was being pushed by a royalty-collecting consortium of companies. The protracted battle ultimately resulted in a non-decision, with the default codec language being removed from the draft standard. But the situation appeared to take a sudden shift in favor of unencumbered codecs in 2009, when Google purchased codec-maker On2 and released the WebM codec under unencumbered terms.

WebM was far newer than Theora and offered quality roughly on par with H.264; Theora's relative performance was a big reason Google did not advocate it as the default codec.. Google subsequently announced its intention to transcode YouTube videos to WebM, and in 2010 Adobe announced that it would support WebM in its Flash products (meaning not just the browser plug-ins, but the content-creation tools and media-delivery servers as well). In January 2011, Google went a step further, and publicly announced that it would drop support for H.264 from its Chrome browser.

But that change never happened. 14 months later, Chrome still supports H.264, and Eich and other Mozilla employees report that Google has remained silent about the decision when asked. Adobe didn't implement the WebM support that it promised either.

Meanwhile, Eich said, H.264 adoption has continued to spread, which has hampered Firefox's growth. For starters, Google's oft-cited promise to trancode YouTube's content to WebM is not all it is cracked up to be: to date Google has only transcoded half of the site's videos, and more importantly, YouTube only delivers WebM content to desktops, and only for those videos that serve no ads (which Eich said makes up a shrinking portion of the total). No other major sites have rejected H.264 in favor of WebM delivery either, while the consumer electronics industry builds more and more H.264 encoding support into video cameras.

On the desktop, Firefox falls back on the Flash plug-in's H.264 support, but on mobile devices, there is no such option. Mozilla believes that mobile browsers are clearly the battlefield deserving the most attention, and on top of that, the project's Boot to Gecko (B2G) effort stands no chance of making it into device-makers' products without H.264 support, Eich said.

Patches and a new API

The combination of Google not pushing WebM and content creators adopting H.264 puts Mozilla in an untenable position, Eich said. Yet the majority of phone and tablet designs ship with a pre-authorized (meaning the royalty fees have been paid) H.264 decoder in silicon, which is what led developer Andreas Gal to propose letting Gecko hand H.264 decoding (and perhaps other encumbered formats like MP3 and AAC) down to the OS or hardware level. Although other option have been discussed, Eich endorses Gal's solution. The upshot is that Firefox and B2G users will be able to see H.264 content on platforms that support it, Mozilla will not have to pay royalty fees (or pass them on to users), and the project can turn its attention to fighting for unencumbered codecs in the next round of standardization battles.

Those battles are not far off, Eich said, starting with the WebRTC real-time chat standard — which is already in-progress and looks poised to recommend unencumbered codecs. There will be other fights, he said, and other generations of video streaming codecs. Continuing to ignore H.264, particularly on mobile devices, would ultimately risk making Mozilla irrelevant further down the line, if not nonexistent altogether:

Gal's patch is attached to bug 714408. In essence, it creates a new API (which Gal dubbed the Media Player API, or MPAPI) for use by OS or hardware decoders. On desktop OSes like Linux, MPAPI would likely be tied in to a media framework like GStreamer (which can use encumbered codecs). It is also possible that the Mozilla Flash plug-in could continue to serve as the H.264 decoding chain on desktop systems, which would require work to expose MPAPI to the Gecko's main plug-in API, NPAPI. Of course, Adobe has also said that it will stop developing its NPAPI Flash plug-in for Linux, which means Flash-fallback will not remain a solution for free OSes in the long term.

As of now, the patch itself is the main source of information on MPAPI, which is still very much a work in progress. MPAPI would hand audio and video frames back to the main Gecko rendering toolchain, however, meaning the content would be fully integrated with the page's JavaScript, HTML, and CSS like any other <video> or <audio> element. The vast majority of the energy expended about the move has not been with its technical points, but with whether or not Mozilla's decision itself is wise, foolish, too pessimistic, or long overdue.

Reaction

The mozilla.dev.platform discussion thread about Gal's proposal is rife with critics arguing that Mozilla is making the wrong move, and with defenders inside and outside the project. The criticisms fall into three basic camps: those that think the H.264 situation is not as bad as described, those that feel Mozilla has not tried hard enough to advocate WebM (or that it should try Just One More Time), and those that object to enabling any form of H.264 playback on principle alone.

Critics who say that WebM has not lost to H.264 tend to point to Google's YouTube transcoding effort (which Eich countered in his blog post), or hold out hope that Google will indeed drop H.264 support from Chrome (often based on the number of WebM "supporters" listed). On the latter point, Eich argues that Chrome's H.264 support is moot, because the desktop browser would simply fall back on the Flash plug-in like Firefox does today. In fact, he said, Chrome's heavily optimized Flash plug-in amounts to a "practically custom" Flash offering "best-of-breed fallback."

Christopher Blizzard added that ultimately, the content-delivery sites simply are not interested in WebM:

Justin Lebar typified the position that Mozilla is giving up too quickly, calling it "going down without a fight," and saying that he would:

Side-stepping the fact that Lebar's public deadline idea paradoxically threatens to increase support for H.264 if it is not abandoned, Mozilla's Robert O'Callahan calls it "grossly unfair" to suggest that Mozilla has not fought hard enough or long enough against H.264 adoption:

Likewise, Asa Dotzler confirmed that Mozilla has spent months trying to get a response from Google on the H.264 support question, only to be met with silence. O'Callahan also observed that there is already mobile hardware capable of decoding WebM video, but that Google does not enable it on Android devices. In the absence of support from the format's owners, Mozilla says, it alone cannot move WebM forward.

The objection on principle is trickier. Some in the discussion thread expressed personal hurt that Mozilla was not standing its ground against H.264 playback support, but more were concerned that relenting would make it harder for the organization to lobby against encumbered formats in the future. Eich argued that anyone who ignores the fact that Firefox users watch H.264 video via the Flash plug-in is "hiding behind Mother Adobe's skirts" and is not taking a "realistic view of the entire fallback logic chain, and of Firefox's current acute dependency on Flash," which is not different in kind from falling back on an OS decoder. Gal concurred, noting that the MPAPI proposal is only "using existing accelerated decoders that already are licensed and available on the system."

Regarding Mozilla's ability to advocate for open and unencumbered formats in the future, Doztler said that the project has backed down on other lost battles in past, such as the document.all DOM feature, but has maintained it credibility. Mozilla can influence the web, and from time to time kill a bad idea, he said in another message, but "sometimes the Web decides to go where we don't want it;" not supporting it only costs the project developers and users.

Ultimately, though, Dotzler argues that even in light of H.264's popularity, Mozilla's WebM advocacy does not constitute a wasted effort:

A continuing story

It is worth remembering that however one feels about the H.264 lobby and its royalty-collecting schemes, the presence of dedicated video decoding chips is hardly an isolated situation. There are currently a great many components in our computers which are covered by patents, and many chips for which we do not have source code. Yet enabling software access to those components is not perceived as a violation of principle. Furthermore, it is hard to argue that the Flash plug-in that Firefox currently falls back on is turf worth fighting for — it has a history of bugs and security holes unrelated to the video decoders it ships.

The debate over enabling H.264 playback via MPAPI shows little sign of calming down. Eich, Dotzler, Gal, and the other project members continue to argue that enabling the format is a purely pragmatic move, and that it would be a better use of Mozilla's energy to combat encumbered codecs on the still-in-development format battles.

They got a boost on March 18 when Mozilla chief Mitchell Baker posted her own blog entry in support of the proposed change. Baker said that "giving our users a great experience" is both one of the project's key values, and a demanding goal that drives realistic product development.

The comment thread on Baker's blog follows much the same pattern as the discussion group. There are supporters, commiserators, and vocal critics. But wherever H.264 itself ends up on future versions of Firefox and B2G, one thing is for sure: H.264-vs-WebM is not the last codec fight the software world will see. As several in the thread pointed out, progress on H.265 is already well underway, and the players involved are similar — there can be little doubt that the battle will be similar, too.

Comments (25 posted)

The N9: what MeeGo could have been

There is value in whining at times. At a recent conference, your editor complained that he had been unable to get a sense for what MeeGo is really like since nobody had ever sent him an N9 handset. Some time thereafter, a shiny blue N9 showed up on the doorstep courtesy of the kind folks at Nokia. What follows are various impressions from playing with this new toy; your editor, normally an Android user, has found a lot to both like and dislike in this seemingly doomed smartphone platform.

The N9 is an attractive device, only slightly larger than a Nexus One. The spouse, upon handling it, complained about the rather sharp corners - but proved reluctant to hand the device back anyway. The corners do stand out in an age when everything is supposed to be rounded, and they can dig into the palm slightly, but it's all a matter of taste. The handset's specifications are reasonably standard for this vintage of device; there is a 1GHz processor, 1GB of RAM, and 16GB of storage. In a welcome change from previous Nokia devices, the N9 uses a standard micro-USB connector instead of something special Nokia made up for that specific handset. The camera is quite nice; there is also a front-facing camera, though the built-in Skype client is unable to use it. By all appearances, the handset is sealed forevermore; replacing the battery does not appear to be an option.

Android users will have likely gotten used to that environment's home screen which can be populated (especially with CyanogenMod builds) with a wide variety of application launchers, contact shortcuts, active widgets, and more. The N9 MeeGo experience is somewhat different, in that there are three specialized home screens with limited potential for customization. The first of these is the familiar matrix of icons providing access to applications on the phone. Users can rearrange the icons (including putting them into subfolders), but there is no way to put anything other than application launchers on this screen.

It is also possible to remove applications via this screen. Dishearteningly, one quickly learns that, as with many Android builds, some applications have been rendered immortal and unremovable. Your editor has little use for Facebook or Twitter applications, but they cannot be made to go away. The best that can be done is to move them to a folder where, at least, they can be kept out of sight.

The second "home" screen (accessible via a left or right swipe across the screen) shows the running applications in a 2x2 grid. Their current screens are visible, and specific applications can be killed if desired. As one might expect, tapping on an application's screen brings it back to the foreground. The third screen is a notification area; messages, weather information, and the latest urgent Twitter spam will show up here.

Annoyingly, none of the home screens rotate when the phone is held in the landscape orientation. Applications handle rotation without trouble, but the home screens appear to be special.

The applications shipped with the phone are generally attractive and nice to use - though sometimes they seem to get into dead end screens where a "back" button would be nice to have. There is a mapping and navigation application that works nicely and comes with suitably annoying voices in a wide range of languages. The camera application is feature-rich and responsive. There is a central account manager that organizes access credentials; interestingly, it can hook into Google, but not for contact information. Getting access to contacts will be one of the first things a former Android user will want to do; fortunately it is possible by telling the phone that Google is an Exchange server. WiFi tethering is built into the phone but "forbidden" for US users; fortunately, one can install the "SpotOn" application to get around that bit of obnoxiousness.

On the other hand, the web browser makes one wish for the Android equivalent. Android's browser has a "fit page to screen" option that does a nice job of rendering the interesting part of a web page in an optimally readable form; the MeeGo browser, instead, just mashes the entire page, unreadably, onto the screen, requiring zoom-in gestures and side-to-side scrolling for almost every page that has not been specifically designed for small screens. That Android feature, arguably, is on its own responsible for the fact that nobody at LWN has found the time to make a more mobile-friendly version of the site; the N9 has made it clear that not everybody has as good an experience.

The MeeGo on-screen keyboard, while being entirely functional, is also not as nice as the Android equivalent. There appears to be no built-in spelling correction or word prediction, making typing a longer and more error-prone process. That is one of the bigger shortcomings of this system. Typing on keyboard-less handsets is a painful enough procedure even with a top-quality on-screen keyboard; this is not the place for a second-rate solution. (Correction: there is a simple prediction mechanism that only seems to appear some of the time; it is better than nothing, but doesn't change the main point of this paragraph).

There is, naturally, an applications store full of things to add on to an N9. A number of important programs are there, and, inevitably, the handset comes with Angry Birds already installed. The range of available applications falls far short of that found in the Android store, though. That is far from surprising; given that MeeGo was a lame-duck platform from the beginning, there will be little motivation for developers to put any time into supporting it.

Inside the device

The MeeGo system is a far more Linux-like environment than Android provides. A terminal application comes preinstalled on the device; it works well enough for what it is, but the truth of the matter is that trying to do command-line work with an on-screen keyboard is always going to be painful. Fortunately, there's an easier way. If one puts the device into developer mode (a simple menu tweak) and plugs it into a computer's USB port, the device offers to connect in "SDK mode." In that mode, it presents as a network interface; there is even a built-in DHCP server so the computer side of the connection gets configured automatically. After that, it's just a matter of using SSH to obtain a shell on the handset. Unlike Android handsets, the N9 has Busybox on it from the start, so the shell is actually reasonably usable.

For the most part, the phone environment feels like Linux. There is, however, no functioning su command; one is, instead, supposed to use devel-su. The result is a shell that claims to be root, but all it takes is a find command run from the top of the filesystem to see that root is not all-powerful on this system. There are certain things that one still cannot access or change. This behavior is the result of the MeeGo security framework in action. Through a combination of trusted computing techniques and mandatory access control, Nokia keeps the device locked down at a certain level. It wouldn't do, after all, to let those pesky users have direct access to the media files that they think they bought on their handset.

Of course, keeping the users away is not the only motivation for the security framework; it is also intended to prevent applications from acting against the users' interests. Applications are installed with "resource tokens" describing the actions they are allowed to carry out; they include the ability to query location information, access the camera, make calls, etc. Superficially it looks a lot like the Android permissions mechanism, but the implementation appears to be wired more deeply in at the kernel level.

Notably, the application installer does not expose resource tokens to the user, so there is no way to know what types of access a given application will have - a major difference from Android. One suspects that most Android users never look at the list of requested permissions, but a subset of us tend to examine them closely indeed. The inability to know what access has been granted to an application seems like a major shortcoming. That will be doubly true anywhere outside of a strict walled-garden application repository; on this system, applications from outside Nokia's store, if they can be installed at all, can only have a restricted set of permissions. But, restricted or not, the user should have the chance to review the permissions requested by an application.

What if you want to bypass the mandatory access control and truly have full access to the device? The answer would appear to be a tool called INCEPTION. It allows the installation of applications with full privilege; one can also disable the security framework altogether. Your editor has not had the time to play with this tool, but it appears to be the ticket for those who are eager to void their warranties and reach for full control of the handset.

Perhaps a true measure of the freedom of a piece of hardware is the existence of independent operating system distributions for it. In the Android world, there is CyanogenMod along with a long list of less well-known, often more dubious, "mods." For the N9 the alternatives on offer are somewhat more restricted, but those who are truly adventurous can give NemoN9 a try. Nemo is the current incarnation of the "Mer" project; it is trying to continue the development of the MeeGo framework as an independent effort. Unfortunately, activity in this project seems to have slowed considerably, though it is still producing regular releases and its use in the upcoming Vivaldi tablet may spur development in the future. What releases Nemo has made have not found their way over to NemoN9, though, which was last updated in November, 2011.

The end of the line

Your editor has often said in the past that MeeGo could become a credible challenger to Android and a strong force in the mobile world in general. After some hands-on experience with a MeeGo device, that impression has not changed. MeeGo provides a polished and pleasant user experience. It falls short of current Android releases in some ways, but it is much nicer to use than the early Android-based devices were. With a bit of work, MeeGo could have been a truly competitive - and more community-friendly - alternative. The fact that things did not turn out that way is a sad comment on the state of the market and the management of certain companies.

The good news is that the developers who worked on this system are out there; many of them are still employed at Nokia. MeeGo may even see some further development for devices other than handsets. But the sad fact is that Nokia has placed its bets on a proprietary operating system with uncertain prospects in the mobile market. If that bet does not work out as hoped, Nokia may yet rediscover the high-quality, free-software alternative at its disposal. Then, perhaps, we'll see a new attempt to put MeeGo-based handsets on the market. For now, though, the N9 has all the look of a solid, sleek and polished platform with no future. In truth, it deserved better than that.

Comments (85 posted)

Page editor: Jonathan Corbet

Security

Shadow hardening

The shadow password mechanism was introduced to provide increased security over the early Unix practice of stored salted-and-hashed passwords in /etc/passwd — but that fact by no means makes it perfect. The Openwall security-hardened Linux distribution (better known as Owl) offers its own alternative called tcb, built around per-user directories and Blowfish encryption. Recently developer Paweł Hajdan unveiled a similar project of his own, which utilizes several of tcb's improvements, but supplies them in what Hajdan hopes is an easier-to-manage design.

Originally, /etc/passwd contained a hashed version of each account's password. The file itself was readable by any user, which enabled features like looking up usernames by their UIDs, and other tricks unrelated to the passwords themselves. The trouble was that attackers could calculate hashes offline then compare their results against /etc/passwd looking for matches. Shadow shuts down that attack vector by separating the hashed password information into a separate file that is readable only by trusted processes. In a sense, both tcb and hardened-shadow take that same approach: separating unrelated information further to reduce the potency of attacks.

Taking care of passwords

Openwall's tcb mechanism stores salted-and-hashed passwords in a directory of its own, /etc/tcb/, beneath which there is a separate directory for each user. Each user's directory is owned by that user account, and contains their own private shadow file (e.g., /etc/tcb/linus/shadow) also owned by the user.

Both the per-user directory and its contents are in the auth group with the SGID bit set for the directory, which provides for additional password policy enforcement. Namely, the system can grant a process read-only access to password hashes and the password-changing tools use each user's private directory as scratch space for the temporary and lock files required during the process. Openwall maintains that this set-up eliminates the need for SUID-root binaries altogether.

Apart from the system password-changing utilities, most application programs access shadow password hashes through either Pluggable Authentication Modules (PAM) or the Name Service Switch (NSS) facility. Tcb provides a custom PAM module named pam_tcb and a corresponding NSS module named libnss_tcb that supersede their /etc/shadow-based alternatives, such as the pam_unix module. As a result, migrating from a shadow system to tcb should be completely transparent to other applications. Openwall provides its own patched suite of shadow utilities (including login, useradd, chpasswd, newgrp, and so on) patched to support /etc/shadow in addition to /etc/tcb/*/shadow (configurable via a switch in /etc/login.defs), as well as utilities to convert from shadow to tcb and vice-versa.

But Owl's tcb scheme also involves a change to Glibc, so that the crypt(3) function uses the Blowfish cipher. The existing Glibc crypt() offers MD5, SHA-256, and SHA-512 encryption methods. The tcb version patches in Blowfish support that is "mostly compatible" with OpenBSD's bcrypt. The principle advantage is that Blowfish can be configured to use as many set-up rounds as desired when generating the hash; thus administrators can increase the number of rounds over time to make attacks more and more computationally expensive as processor power increases.

Hardened-shadow

Hajdan calls his package hardened-shadow, which he says is inspired directly by tcb. In a March 14 post to the owl-dev list, Hajdan asks for feedback from people interested in the project as well as help performing a security audit of the code.

Hardened-shadow is a reimplementation of the shadow utilities based on the original upstream sources, but supporting a somewhat tcb-like layout. The system uses /etc/hardened-shadow/, with a separate directory for each user account. But while tcb only stores a single shadow file for each account, hardened-shadow adds two more: a shell file and an aging file. The shell file allows chsh to modify the user account's default shell without having access to the shadow file, and aging holds the password expiration fields, which would eliminate the need for locking if administrators expired passwords before changing them, according to Hajdan.

In addition, hardened-shadow does not require a patched version of Glibc crypt(), and it supports more PAM options than does pam_tcb. Fans of the Blowfish algorithm might balk at the omission of their favorite cipher, but Hajdan has added a PAM parameter to pam_hardened_shadow that allows one to pass the hash algorithm as an argument. In the standard crypt() scheme, the hash algorithm used is designated by a token surrounded by $ characters — MD5 uses $1$, SHA-256 uses $5$, and so on. Hajdan believes making the PAM module indifferent to the hash algorithm makes the code more useful, so that Owl users can take advantage of Blowfish, while others can use their own cipher — and future-proofs it against new exploits that target particular ciphers.

On the down side, Hajdan admits that his code is currently incompatible with SELinux and with NIS. More importantly, of course, it is brand-new and virtually untested, so he advises against deploying it on any production systems.

Openwall's Alexander Peslyak (aka Solar Designer) offered several reasonable suggestions as feedback to Hajdan's list message. Among them were switching to a common open source license in place of Hajdan's current, custom license text, and building hardened-shadow around pam_tcb rather than implementing yet another PAM module with virtually the same goals. Hajdan's reply was that he felt pam_tcb's reliance on a patched Glibc made for too many packages, and that pam_tcb duplicated too many functions already found elsewhere in the tcb library.

Nevertheless, Peslyak said that Openwall would consider migrating to hardened-shadow's version of the shadow utilities (although not the PAM and NSS modules). The reason he gave was that Openwall currently maintains its patch set against a suite of tools that come from a blend of two different sources: the canonical shadow utilities, and a PAM-based implementation called SimplePAMApps. SimplePAMApps, however, is no longer being actively maintained. Owl and a few other distributions curate their own packages of it, but in the long run, synchronizing with an actively developed tool set is probably less work.

The shadowy future

Back in 2010, Hajdan submitted patches to the shadow utilities to enable optional support for tcb, a change that appears to have landed in the 4.1.5 release from February 2012. Nevertheless, tcb is still an essentially Owl-only tool. There does not seem to be much interest in packaging it among the large, "mainstream" distributions; in 2006 the idea was floated on the fedora-devel list where it was met with skepticism.

Some of the criticism from 2006 is questionable (for example, the concern that users could fill up the /etc filesystem by dumping files into their /tcb/ directories — the group permissions should prevent that), but others, such as requiring new methods for auditing password changes, may be more valid. Hardened-shadow does not alleviate all of those concerns, but by virtue of not requiring a patched Glibc, it at least stands a better chance of being packaged by other distributions.

Any hope for widespread deployment, however, will probably still need to address the SELinux incompatibility in the PAM module — and it will certainly need to wait for a proper security audit and more sets of eyes scrutinizing the code. Shadowed passwords were introduced to split up user account information in a manner that made security exploits harder. As both of these projects show, further division holds the possibility of protection against other attacks. Neither tcb nor hardened-shadow is a perfect solution, of course, but the existing shadow system is far from flawless, too, which is part of what makes a fresh re-examination of it such an interesting exercise.

Comments (34 posted)

Brief items

Security quotes of the week

-- Forbes on security research firms selling exploits to governments

-- Matthew Green

-- Bruce Schneier

Comments (8 posted)

CyanogenMod to disable root by default

The CyanogenMod project has announced that access to the root account will be disabled by default on CM9 installs. "Shipping root enabled by default to 1,000,000+ devices was a gaping hole. With these changes we believe we have reached a compromise that allows enthusiasts to keep using root if they so desire but also provide a good level of security to the majority of users."

Comments (8 posted)

New vulnerabilities

chromium, v8: multiple vulnerabilities

Package(s):	chromium, v8	CVE #(s):	CVE-2011-3031 CVE-2011-3032 CVE-2011-3033 CVE-2011-3034 CVE-2011-3035 CVE-2011-3036 CVE-2011-3037 CVE-2011-3038 CVE-2011-3039 CVE-2011-3040 CVE-2011-3041 CVE-2011-3042 CVE-2011-3043 CVE-2011-3044 CVE-2011-3046 CVE-2011-3047
Created:	March 16, 2012	Updated:	November 7, 2012
Description:	From the openSUSE advisory: Critical CVE-2011-3047: Errant plug-in load and GPU process memory corruption Critical CVE-2011-3046: UXSS and bad history navigation. High CVE-2011-3031: Use-after-free in v8 element wrapper. High CVE-2011-3032: Use-after-free in SVG value handling. High CVE-2011-3033: Buffer overflow in the Skia drawing library. High CVE-2011-3034: Use-after-free in SVG document handling. High CVE-2011-3035: Use-after-free in SVG use handling. High CVE-2011-3036: Bad cast in line box handling. High CVE-2011-3037: Bad casts in anonymous block splitting. High CVE-2011-3038: Use-after-free in multi-column handling. High CVE-2011-3039: Use-after-free in quote handling. High CVE-2011-3040: Out-of-bounds read in text handling. High CVE-2011-3041: Use-after-free in class attribute handling. High CVE-2011-3042: Use-after-free in table section handling. High CVE-2011-3043: Use-after-free in flexbox with floats. High CVE-2011-3044: Use-after-free with SVG animation elements.
Alerts:	openSUSE openSUSE-SU-2012:0374-1 2012-03-16 Gentoo 201203-19 2012-03-25 Ubuntu USN-1524-1 2012-08-08 Ubuntu USN-1617-1 2012-10-25 Mageia MGASA-2012-0324 2012-11-06

Comments (none posted)

gif2png: code execution

Package(s):	gif2png	CVE #(s):	CVE-2010-4695
Created:	March 16, 2012	Updated:	March 21, 2012
Description:	From the Gentoo advisory: The patch for CVE-2009-5018 causes gif2png to truncate GIF pathnames (CVE-2010-4695).
Alerts:	Gentoo 201203-15 2012-03-16

Comments (none posted)

gnash: heap-based buffer overflow

Package(s):	gnash	CVE #(s):	CVE-2012-1175
Created:	March 20, 2012	Updated:	March 27, 2012
Description:	From the Debian advisory: Tielei Wang from Georgia Tech Information Security Center discovered a vulnerability in GNU Gnash which is caused due to an integer overflow error and can be exploited to cause a heap-based buffer overflow by tricking a user into opening a specially crafted SWF file.
Alerts:	Debian DSA-2435-1 2012-03-20 Fedora FEDORA-2012-4070 2012-03-26 Fedora FEDORA-2012-4032 2012-03-26 openSUSE openSUSE-SU-2012:0415-1 2012-03-27 Gentoo 201207-08 2012-07-09

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2012-1146 CVE-2012-1179
Created:	March 19, 2012	Updated:	June 1, 2012
Description:	From the Red Hat bugzilla [1], [2]: 1) There is an issue when memcg unregisters events that were attached to the same eventfd: - On the first call mem_cgroup_usage_unregister_event() removes all events attached to a given eventfd, and if there were no events left, thresholds->primary would become NULL; - Since there were several events registered, cgroups core will call mem_cgroup_usage_unregister_event() again, but now kernel will oops, as the function doesn't expect that threshold->primary may be NULL. 2) In some cases it may happen that pmd_none_or_clear_bad() is called with the mmap_sem hold in read mode. In those cases the huge page faults can allocate hugepmds under pmd_none_or_clear_bad() and that can trigger a false positive from pmd_bad() that will not like to see a pmd materializing as trans huge. A privileged user in the KVM guest can use this flaw to crash the host. An unprivileged local user could use this flaw to crash the system.
Alerts:	Fedora FEDORA-2012-3712 2012-03-17 Ubuntu USN-1407-1 2012-03-27 Ubuntu USN-1406-1 2012-03-27 Ubuntu USN-1405-1 2012-03-27 Fedora FEDORA-2012-3715 2012-03-26 Ubuntu USN-1421-1 2012-04-12 Ubuntu USN-1422-1 2012-04-12 openSUSE openSUSE-SU-2012:0540-1 2012-04-20 SUSE SUSE-SU-2012:0554-1 2012-04-23 SUSE SUSE-SU-2012:0554-2 2012-04-26 Ubuntu USN-1431-1 2012-04-30 Ubuntu USN-1433-1 2012-04-30 Ubuntu USN-1440-1 2012-05-08 Ubuntu USN-1458-1 2012-05-31 Red Hat RHSA-2012:0743-01 2012-06-18 CentOS CESA-2012:0743 2012-06-19 Scientific Linux SL-kern-20120619 2012-06-19 Oracle ELSA-2012-2020 2012-06-21 Oracle ELSA-2012-0743 2012-06-21 Oracle ELSA-2012-2021 2012-06-23 Oracle ELSA-2012-2021 2012-06-23 Red Hat RHSA-2012:1042-01 2012-06-26 openSUSE openSUSE-SU-2012:0799-1 2012-06-28 Oracle ELSA-2012-2022 2012-07-02 Oracle ELSA-2012-2022 2012-07-02 Oracle ELSA-2012-0862 2012-07-02 openSUSE openSUSE-SU-2012:1439-1 2012-11-05 openSUSE openSUSE-SU-2013:0927-1 2013-06-10

Comments (none posted)

libapache2-mod-fcgid: excessive resource consumption

Package(s):	libapache2-mod-fcgid	CVE #(s):	CVE-2012-1181
Created:	March 20, 2012	Updated:	March 21, 2012
Description:	From the Debian advisory: It was discovered that the Apache FCGID module, a FastCGI implementation, did not properly enforce the FcgidMaxProcessesPerClass resource limit, rendering this control ineffective and potentially allowing a virtual host to consume excessive resources.
Alerts:	Debian DSA-2436-1 2012-03-19 Gentoo 201207-09 2012-07-09

Comments (none posted)

libpng10: code execution

Package(s):	libpng10	CVE #(s):	CVE-2011-3045
Created:	March 19, 2012	Updated:	April 2, 2012
Description:	From the Red Hat bugzilla: A type conversion flaw leading to an out-of-bounds heap buffer read was found in the way libpng, a library of functions for manipulation PNG image format files, performed expansion of certain iCCP, iTXt, and zTXt PNG image file chunks. A remote attacker could provide a specially-crafted Portable Network Graphics (PNG) image file, which once opened in an application, linked against libpng, could lead to denial of service or in some cases, execution of arbitrary code with permission of the user running such an application.
Alerts:	Fedora FEDORA-2012-3545 2012-03-19 Fedora FEDORA-2012-3536 2012-03-19 Red Hat RHSA-2012:0407-01 2012-03-20 CentOS CESA-2012:0407 2012-03-20 CentOS CESA-2012:0407 2012-03-21 Mandriva MDVSA-2012:033 2012-03-21 Oracle ELSA-2012-0407 2012-03-20 Oracle ELSA-2012-0407 2012-03-20 Ubuntu USN-1402-1 2012-03-22 Scientific Linux SL-libp-20120321 2012-03-21 Debian DSA-2439-1 2012-03-22 Fedora FEDORA-2012-3739 2012-03-24 openSUSE openSUSE-SU-2012:0432-1 2012-03-30 Fedora FEDORA-2012-3705 2012-03-31 openSUSE openSUSE-SU-2012:0466-1 2012-04-04 Gentoo 201206-15 2012-06-22 Slackware SSA:2012-206-01 2012-07-24

Comments (none posted)

minitube: insecure tmp file handling

Package(s):	minitube	CVE #(s):
Created:	March 16, 2012	Updated:	March 21, 2012
Description:	From the Gentoo advisory: Tomáš Pružina reported that Minitube does not handle temporary files securely. A local attacker could perform symlink attacks to overwrite arbitrary files with the privileges of the user running the application.
Alerts:	Gentoo 201203-18 2012-03-16

Comments (none posted)

nginx: information disclosure

Package(s):	nginx	CVE #(s):	CVE-2012-1180
Created:	March 20, 2012	Updated:	April 5, 2012
Description:	From the Debian advisory: Matthew Daley discovered a memory disclosure vulnerability in nginx. In previous versions of this web server, an attacker can receive the content of previously freed memory if an upstream server returned a specially crafted HTTP response, potentially exposing sensitive information.
Alerts:	Debian DSA-2434-1 2012-03-19 Gentoo 201203-22 2012-03-28 Mandriva MDVSA-2012:043 2012-03-29 Fedora FEDORA-2012-3991 2012-03-31 Fedora FEDORA-2012-4006 2012-03-31 openSUSE openSUSE-SU-2012:0469-1 2012-04-05

Comments (none posted)

openswan: denial of service

Package(s):	openswan	CVE #(s):	CVE-2011-2147
Created:	March 16, 2012	Updated:	March 21, 2012
Description:	From the Gentoo advisory: Improper permissions are used on /var/run/starter.pid and /var/lock/subsys/ipsec (CVE-2011-2147).
Alerts:	Gentoo 201203-13 2012-03-16

Comments (none posted)

pidgin: two denial of service vulnerabilities

Package(s):	pidgin	CVE #(s):	CVE-2011-4939 CVE-2012-1178
Created:	March 16, 2012	Updated:	March 26, 2012
Description:	From the Mandriva advisory: The pidgin_conv_chat_rename_user function in gtkconv.c in Pidgin before 2.10.2 allows remote attackers to cause a denial of service (NULL pointer dereference and application crash) by changing a nickname while in an XMPP chat room (CVE-2011-4939). The msn_oim_report_to_user function in oim.c in the MSN protocol plugin in libpurple in Pidgin before 2.10.2 allows remote servers to cause a denial of service (application crash) via an OIM message that lacks UTF-8 encoding (CVE-2012-1178).
Alerts:	Mandriva MDVSA-2012:029 2012-03-16 Fedora FEDORA-2012-4595 2012-03-24 Ubuntu USN-1500-1 2012-07-09

Comments (none posted)

pyfribidi: code execution

Package(s):	pyfribidi	CVE #(s):	CVE-2012-1176
Created:	March 21, 2012	Updated:	March 21, 2012
Description:	Pyfribidi suffers from a buffer overflow exploitable via a four-byte Unicode character.
Alerts:	Fedora FEDORA-2012-3549 2012-03-21 Fedora FEDORA-2012-3537 2012-03-21

Comments (none posted)

python-mwlib: denial of service

Package(s):	python-mwlib	CVE #(s):
Created:	March 19, 2012	Updated:	March 21, 2012
Description:	From the Fedora advisory: It was reported that mwlib suffered from a flaw that could allow a remote attacker to perform a denial of service attack on a mwlib installation by forcing it to parse a specially-crafted #iferror magic function. This issue has been resolved in version 0.13.5.
Alerts:	Fedora FEDORA-2012-2994 2012-03-17

Comments (none posted)

rubygem-actionpack: arbitrary HTML or webscript execution

Package(s):	rubygem-actionpack	CVE #(s):	CVE-2012-1098 CVE-2012-1099
Created:	March 19, 2012	Updated:	May 9, 2012
Description:	From the Red Hat bugzilla [1], [2]: 1) A cross-site scripting (XSS) flaw was found in the way the String class, used in Ruby on Rails, performed HTML escaping of SafeBuffer objects, when such objects were manipulated directly via '[]' method or other methods, also returning new instances of SafeBuffer object. By using these methods, such newly returned SafeBuffer instances would be inadvertently marked as HTML safe. If a Ruby on Rails application used SafeBuffer objects this way, a remote attacker could provide a specially-crafted input, which once processed by such SafeBuffer instance would pass the HTML escaping test without further filtering, possibly leading to arbitrary HTML or webscript execution. 2) A cross-site scripting (XSS) flaw was found in the way 'select' helper method of the Ruby on Rails performed HTML escaping of 'select' HTML tag options, when the tags were created manually. In this case, the select tag values might end up unescaped. A remote-attacker could provide a specially-crafted input to Ruby on Rails application, using select tags this way, which potentially resulted into arbitrary HTML or webscript execution.
Alerts:	Fedora FEDORA-2012-3321 2012-03-17 Fedora FEDORA-2012-3355 2012-03-17 Fedora FEDORA-2012-3321 2012-03-17 Debian DSA-2466-1 2012-05-09

Comments (none posted)

systemd: removal of arbitrary system files

Package(s):	systemd	CVE #(s):	CVE-2012-1174
Created:	March 19, 2012	Updated:	March 26, 2012
Description:	From the Mandriva advisory: A TOCTOU race condition was found in the way the systemd-logind login manager performed removal of particular records related with user session upon user logout. A local attacker could use this flaw to conduct symbolic link attacks, potentially leading to removal of arbitrary system files.
Alerts:	Mandriva MDVSA-2012:030 2012-03-16 openSUSE openSUSE-SU-2012:0383-1 2012-03-19 Fedora FEDORA-2012-4018 2012-03-26 Fedora FEDORA-2012-4024 2012-03-26

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The 3.3 kernel was released on March 18, so there is no development kernel as of this writing. Some of the headline features in the 3.3 release include the byte queue limits infrastructure, Open vSwitch, the return of much of the Android code to the staging tree, C6X architecture support, large physical address extension support for the ARM architecture, and more. Much more information can be found on the KernelNewbies 3.3 page.

The 3.4 merge window is open; see the separate article below for a summary of what has been merged so far.

Stable updates: the 3.0.25 and 3.2.12 stable updates were released, with the usual pile of important fixes, on March 19. For users of older kernels, the 2.6.27.62 and 2.6.32.59 long term kernel releases, with a relatively small number of fixes, came out on March 17.

The 3.0.26 and 3.2.13 stable updates are in the review process as of this writing; they can be expected on or after March 23.

Comments (none posted)

Quotes of the week

-- Christoph Lameter

-- H. Peter Anvin

-- Adam Jackson

Comments (none posted)

Gregg: Linux Kernel Performance: Flame Graphs

Brendan Gregg demonstrates "flame graphs" as a tool for tracking down kernel performance problems. "The perf report tree (and the ncurses navigator) do an excellent job at presenting this information as text. However, with text there are limitations. The output often does not fit in one screen (you could say it doesn’t need to, if the bulk of the samples are identified on the first page). Also, identifying the hottest code paths requires reading the percentages. With the flame graph, all the data is on screen at once, and the hottest code-paths are immediately obvious as the widest functions." The flame graph code is hosted on Github.

Comments (3 posted)

Linsched for 3.3

At the 2011 Kernel Summit, Google developer Paul Turner described a scheduler testing framework which, he said, would be released soon. Naturally, things took longer than expected, but, on March 14, Paul released a version of Linsched for general use. Given the amount of interest in this tool, it's likely that it will find its way into the mainline in a relative hurry.

Linsched is a framework that can run the kernel scheduler with various simulated workloads and draw conclusions about the quality of the decisions made. It looks at overall CPU utilization, the number of migrations, and more. It is able to simulate a wide range of hardware topologies with different characteristics.

The original Linsched posting was quite intrusive; it inserted over 5,000 lines of code into the kernel behind "#ifdef LINSCHED" lines. A determined effort has reduced that number slightly - to all of 20 lines of code. The rest has been cleverly hidden in a special "linsched" architecture that provides just enough support to run the scheduler in user space. The actual simulation and measurement code lives in the tools directory.

Making changes to the scheduler is a notoriously difficult task; one can easily add regressions for specific workloads that go unnoticed until the changes go into production. With enough simulated topologies and workloads, a tool like Linsched should be able to remove a lot of that risk from scheduler development. And that should lead to better kernel releases overall.

Comments (2 posted)

Kernel development news

3.4 Merge window part 1

The release of the 3.3 kernel on March 18 has led inevitably to the opening of the merge window for the 3.4 development cycle. As of this writing, some 3,500 non-merge changesets have been pulled into the mainline; this cycle, in other words, has just begun.

Some of the user-visible features merged for 3.4 include:

• The perf utility understands a new --uid flag, which restricts data gathering to processes owned by the given user ID. It is also now possible to specify multiple processes or threads with the --pid and --tid options.

• The perf events subsystem can now sample "taken branch" events on hardware with the "last branch record" functionality.

• The "zcache" compressed caching system (still in staging) can now use the crypto API for access to compression algorithms.

• The "Yama" security module has been merged; for now it just implements some restrictions on how the ptrace() system call can be used, but others may follow. Yama is meant to be a place to collect various discretionary access control mechanisms intended to make a system more secure.

• The kernel now has read-only support for the qnx6fs filesystem used with the QNX operating system.

• New drivers include:
  • Crypto: Tegra AES crypto engines.

  • Miscellaneous: EnergyMicro EFM32 UART/USART ports, Maxim DS2781 battery monitors, Solarflare SFC9000-family hwmon controllers, Solarflare SFC9000-family SR-IOV controllers, TI TPS62360 and TPS65217 power regulators, Samsung S5M8767 regulators, Renesas RSPI controllers, SuperH HSPI controllers, CSR SiRFprimaII SPI controllers, Broadcom BCM63xx SPI controllers, and Freescale i.MX on-chip ANATOP LDO regulators.

  • Network: Xilinx 10/100/1000 AXI Ethernet controllers, PEAK PCAN-ExpressCard, PCAN-USB and PCAN-PC CAN controllers, NXP Semiconductor LPC32xx ARM SoC-based Ethernet controllers, and TI CPSW switches.

  • USB: Ozmo USB-over-WiFi controllers.

  • Staging transitions: the old telephony drivers have been moved into staging in anticipation of their eventual removal from the kernel altogether.

  The kernel now also contains an audio USB gadget driver compliant with USB audio class 2.0.

Also worth noting: the "ramster" transcendent memory functionality was briefly added to the staging tree before being removed; various other changes had caused it to be seriously broken. Ramster can be thought of as a way of sharing memory across machines; a system with spare pages can host data for another that is under memory pressure. See this article for more details and this article for an exposition of the vision behind Ramster. Adding this functionality requires carving a number of features out of the OCFS2 filesystem and making them globally available. One assumes these patches will return for 3.5.

Changes visible to kernel developers include:

• Jump labels have been rebranded again; after a false start they are now known as "static keys". Details can be found in the new Documentation/static-keys.txt file.

• The (now) unused get_driver() and put_driver() functions have been removed from the kernel.

• The debugfs filesystem understands the uid=, gid=, and mode= mount options, allowing the ownership and permissions for the filesystem to be set in /etc/fstab.

• The zsmalloc allocator has been added to the staging tree; the older "xvmalloc" allocator has been removed.

• The Android "alarm" driver has been added to the staging tree.

• The deferred driver probing mechanism has been merged.

• The list of power management stages continues to grow; the kernel has new callbacks called suspend_late(), resume_early(), freeze_late(), thaw_early(), poweroff_late(), and restore_early() for operations that must be performed at just the right time.

• The "IRQ domain" abstraction has been merged; IRQ domains make it easier to manage interrupts on systems with more than one interrupt controller. See Documentation/IRQ-domain.txt for more information.

• The long-unused second argument to kmap_atomic() has been removed. Thanks to some preprocessor trickery, calling kmap_atomic() with two arguments still works, but a deprecation warning will result.

• There is a new mechanism for the autoloading of drivers for specific x86 CPU features. Such drivers should declare a MODULE_DEVICE_TABLE with the x86cpu type; see the comments at the head of arch/x86/kernel/cpu/match.c for details.

The 3.4 merge window can be expected to continue until roughly April 2. There are a lot of subsystem trees yet to be pulled, so one can expect a large number of changes to go in between now and then.

Comments (4 posted)

The perils of pr_info()

In the beginning there was printk() - literally: the 0.01 kernel release included 44 printk() calls. Since then, printk() has picked up details like logging levels and a lot of new formatting operators; it has also expanded to tens of thousands of call sites throughout the kernel. Developers often reach for it as the first way to figure out what is going on inside a misbehaving subsystem. If some developers have their way, though, printk() calls will become an endangered species. But not everybody has signed on to that goal.

There are certainly plenty of ways in which printk() could be improved. It imposes no standardization on messages, either across a subsystem or over time. As a result, messages can be hard for programs (or people) to parse, and they can change in trivial but obnoxious ways from one kernel release to the next. The actual calls, starting with text like:

    printk(KERN_ERR ...

are relatively verbose; among other things, that often causes printk() statements to run afoul of the 80-column line width restriction. Messages printed with printk() may also lack important information needed to determine what the kernel is really trying to say.

Various attempts have been made to improve on printk() over the years. Arguably the most successful of those is the set of functions defined for device drivers:

    int dev_dbg(struct device *dev, const char *format, ...);
    int dev_info(struct device *dev, const char *format, ...);
    int dev_notice(struct device *dev, const char *format, ...);
    /* ... */
    int dev_emerg(struct device *dev, const char *format, ...);

These functions, by embedding the logging level in the name itself, are more concise than the printk() calls they replace. They also print the name of the relevant device in standard form, ensuring that it's always possible to associate a message with the device that generated it. Use of these functions is not universal in device drivers, but it is widespread and uncontroversial.

There is a rather lower level of consensus surrounding a different set of functions (macros, really) that look like this:

    int pr_info(const char *format, ...);
    /* ... */
    int pr_emerg(const char *format, ...);

These functions, too, encode the logging level in the function name, making things more concise. They also attempt to at least minimally standardize the format of logging by passing the format string through a macro called pr_fmt(). That leads to a line like this appearing in several hundred source files in the kernel:

    #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

Due to the way the macro works, this line must appear before the #include block that would otherwise be at the beginning of the file. Defining pr_fmt() in this way causes all strings printed from the file to have the module name prepended; many subsystems use a literal string rather than the module name, but the intent is the same.

The spread of pr_*() through the kernel is mainly the result of an ongoing campaign by Joe Perches - notable for having just merged a 100,000-line whitespace-only ISDN subsystem cleanup patch for 3.4 - who has converted thousands of printk() calls over the years. To some developers, these changes are a welcome cleaning-up of the code; to others, they represent pointless code churn. The discussion has been quiet for a while, but it recently came back when Joe tried to convert the ext4 filesystem; ext4 maintainer Ted Ts'o rejected the conversion, saying:

David Miller commented on this decision in a rather unsympathetic fashion:

Ted probably does not feel like a relic, and he is probably not trying to be sophisticated; he is almost certainly trying to maintain code he is responsible for in the best way he can. In his view, changing a bunch of code from one print function to another - possibly introducing a lot of patch conflicts on the way - does not help in that regard. Beyond that, he said, the standardization introduced by these functions is nowhere near enough to solve the structured logging problem, meaning that, someday, all those calls will have to be changed yet another time when a proper solution is available.

Proponents of the change argue that some structure is better than none, and that the new functions offer some useful flexibility when the time to add more structure comes. They claim that the overall size of the kernel is reduced (slightly) due to better sharing of strings. Messages printed with pr_debug() can be enabled and disabled with the dynamic debugging interface, while straight printk() calls cannot. And, perhaps most of all, they argue that consistency across the code base has value - though that argument was heard rather less when the pr_*() interface was new and relatively unused.

Needless to say, this is not the kind of discussion that comes to any sort of definitive conclusion. With regard to ext4, the conversion will probably not take place anytime soon; that is Ted's turf, and it is unlikely that anybody can summon arguments strong enough to convince Linus to override him. Elsewhere in the kernel, though, these conversions will certainly continue. As will, undoubtedly, the associated flame wars.

Comments (8 posted)

Toward better NUMA scheduling

A non-uniform memory access (NUMA) system is a computer divided into "nodes," where each node (which may contain multiple processors) has some memory which is local to the node. All system memory is visible to all nodes, but accesses to memory that is not local to the accessing node must go over an inter-node bus; as a result, non-local accesses are significantly slower. There is, thus, a real performance advantage to be gained by keeping processes and their memory on the same node.

The Linux kernel has had NUMA awareness for some time, in that it understands that moving a process from one node to another can be an expensive undertaking. There is also an interface (available via the mbind() system call) by which a process can request a specific allocation policy for its memory. Possibilities include requiring that all allocations happen within a specific set of nodes (MPOL_BIND), setting a looser "preferred" node (MPOL_PREFERRED), or asking that allocations be distributed across the system (MPOL_INTERLEAVE). It is also possible to use mbind() to request the active migration of pages from one node to another.

So NUMA is not a new concept for the kernel, but, as Peter Zijlstra noted in the introduction to a large NUMA patch set, things do not work as well as they could:

As might be expected, the patch set is dedicated to the creation of a kernel that does not "completely blow." To that end, it adds a number of significant changes to how memory management and scheduling are done in the kernel.

There are three major sub-parts to Peter's patch set. The first is a reworked patch set first posted by Lee Schermerhorn in 2010. These patches change the memory policy mechanism to make it easier for the kernel to fix things up after a process's memory has been allocated on distant nodes. "Page migration" is the process of moving a page from one node to another without the owning process(es) noticing the change. With Lee's patches, the kernel implements a variation called "lazy migration" that does not immediately relocate any pages. Instead, the target pages are simply unmapped from the process's page tables, meaning that the next access to any of them will generate a page fault. Actual migration is then done at page fault time. Lazy migration is a less expensive way of moving a large set of pages; only the pages that are actually used are moved, the work can be spread over time, and it will be done in the context of the faulting process.

The lazy migration mechanism is necessary for the rest of the patch set, but it has value on its own. So the feature is made available to user space with the MPOL_MF_LAZY flag; it is intended to be used with the MPOL_MF_MOVE flag, which would otherwise force the immediate migration of the affected pages. There is also a new MPOL_MF_NOOP flag allowing the calling process to request the migration of pages according to the current policy without changing (or even knowing) that policy.

With lazy migration, memory distributed across a system as the result of memory allocation and scheduling decisions can be slowly pulled back to the optimal node. But it is better to avoid making that kind of mess in the first place. So the second part of the patch set starts by adding the concept of a "home node" to a process. Each process (or "NUMA entity" - meaning groups containing a set of processes) is assigned a home node at fork() time. The scheduler will then try hard to avoid moving a process off its home node, but within bounds: a process will still be run on a non-home node if the alternative would be an unbalanced system. Memory allocations will, by default, be performed on the home node, even if the process is running elsewhere at the time.

These policies should minimize the scattering of memory across the system, but, with this kind of scheduling regime, it is inevitable that, eventually, one node will end up with too many processes and too little memory while others are underutilized. So, sometimes, it will be necessary to rebalance things. When the scheduler notices that long-running tasks are being forced away from their home nodes - or that they are having to allocate memory non-locally - it will consider migrating them to a new node. Migration is not a half-measure in this case; the scheduler will move both the process and its memory (using the lazy migration mechanism) to the target node. The move is expensive, but the process (and the system) should run much more efficiently once it's done. It only makes sense for processes that are going to be around for a while, though; the patch set tries to approximate that goal by only considering processes with at least one second of run time for migration.

The final piece is a pair of new system calls allowing processes to be put into "NUMA groups" that will share the same home node. If one of them is migrated, the entire group will be migrated. The first system call is:

    int numa_tbind(int tid, int ng_id, unsigned long flags);

This system call will bind the thread identified by tid to the NUMA group identified by ng_id; the flags argument is currently unused and must be zero. If ng_id is passed as MS_ID_GET, the system call will, instead, simply return the current NUMA group ID for the given thread. A value of MS_ID_NEW, instead, creates a new NUMA group, binds the thread to that group, and returns the new ID.

The second new system call is:

    int numa_mbind(void *addr, unsigned long len, int ng_id, unsigned long flags);

This call will set up a memory policy for the region of len bytes starting at addr and bind it to the NUMA group identified by ng_id. If necessary, lazy migration will be used to move the memory over to the node where the given NUMA group is based. Once again, flags is unused and must be zero. Once the memory is bound to the NUMA group, it will stay with the processes in that group; if the processes are moved, the memory will move with them.

Peter provided some benchmark results from a two-node system. Without the NUMA balancing patches, over time, the benchmark ended up with just as many remote memory accesses as local accesses - allocated memory was spread across the system. With the NUMA balancer, 86% of the memory accesses were local, leading to a significant speedup. As Peter put it: "These numbers also show that while there's a marked improvement, there's still some gain to be had. The current numa balancer is still somewhat fickle." A certain amount of fickleness is perhaps to be expected for such an involved patch set, given how young it is. Given some time, reviews, and testing, it should evolve into a solid scheduler component, giving Linux far better NUMA performance than it has ever had in the past.

Comments (27 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.3 release . (March 19, 2012)

• Greg KH: Linux 3.2.12 . (March 19, 2012)

• Greg KH: Linux 3.0.25 . (March 19, 2012)

• Steven Rostedt: 3.0.25-rt43 . (March 21, 2012)

• Steven Rostedt: 3.0.24-rt42 . (March 21, 2012)

• Willy Tarreau: Linux 2.6.32.59 . (March 17, 2012)

• Willy Tarreau: Linux 2.6.27.62 . (March 17, 2012)

Core kernel code

• Thomas Gleixner: seqlock consolidation . (March 15, 2012)

• Mike Turquette: common clk framework . (March 16, 2012)

• Peter Zijlstra: sched/numa . (March 16, 2012)

• Andrea Arcangeli: AutoNUMA alpha6 . (March 16, 2012)

• Tejun Heo: cgroup: cftype based file interface . (March 18, 2012)

• Lai Jiangshan: rcu: implement call_srcu() . (March 19, 2012)

• Hans de Goede: poll: add poll_requested_events() and poll_does_not_wait() functions . (March 20, 2012)

• Frederic Weisbecker: Nohz cpusets v2 (adaptive tickless kernel) . (March 21, 2012)

Development tools

• Paul Turner: LinSched for v3.3-rc7. (March 15, 2012)

• Hui Zhu: KGTP (Linux Kernel debugger and tracer) 20120319 release(include hotcode analyzer hotcode.py) . (March 19, 2012)

• Jiri Olsa: perf tool: Add new event group management . (March 20, 2012)

• Mathieu Desnoyers: LTTng 2.0 stable . (March 20, 2012)

Device drivers

• Carsten Emde: Provide workarounds to use DRM/KMS with broken graphics hardware . (March 15, 2012)

• jean.pihet@newoldbits.com: PM: Create the AVS class of drivers . (March 16, 2012)

• Aneesh V: Add TI EMIF SDRAM controller driver . (March 16, 2012)

• Seth Forshee: Add driver for Apple gmux device . (March 16, 2012)

• Rob Clark: RFC: dma-buf: userspace mmap support . (March 16, 2012)

• Sylwester Nawrocki: New camera controls . (March 16, 2012)

• Laurent Pinchart: soc-camera: Add support for configurable line stride . (March 21, 2012)

• Tomoya MORINAGA: sound/soc/codecs: add LAPIS Semiconductor ML26124 . (March 19, 2012)

• Ying-Chun Liu (PaulLiu): rtc: add support for Freescale SNVS RTC . (March 19, 2012)

• Diego Elio Pettenò: gpio: add support for ITE IT87xx Super I/O GPIO. . (March 20, 2012)

• jean.pihet@newoldbits.com: PM: Create the AVS class of drivers . (March 20, 2012)

• Mugunthan V N: Adding new TI Common Platform ethernet SWitch driver . (March 20, 2012)

• Martin K. Petersen: Write same support v4 . (March 20, 2012)

• Nicolas Ferre: of: Add generic device tree DMA helpers . (March 20, 2012)

• Jon Brenner: TAOS tsl2x7x . (March 20, 2012)

• Roland Stigge: USB: gadget driver for LPC32xx . (March 20, 2012)

• Venu Byravarasu: regulator: Add TPS65090 regulator driver . (March 21, 2012)

Documentation

• Christoph Hellwig: XFS status update for February 2012 . (March 18, 2012)

Filesystems and block I/O

• Shaohua Li: Add TRIM support for raid linear/0/1/10 . (March 16, 2012)

• Evgeniy Polyakov: pohmelfs: call for inclusion . (March 16, 2012)

• John Stultz: [RFC] Volatile ranges (v4) . (March 16, 2012)

• Adrian McMenamin: VMUFAT filesystem [0/4] . (March 21, 2012)

Memory management

• Aneesh Kumar K.V: memcg: Add memcg extension to control HugeTLB allocation . (March 16, 2012)

• Lai Jiangshan: add kmalloc_align() . (March 20, 2012)

Architecture-specific

• Yinghai Lu: PCI, x86: pci root bus hotplug support . (March 19, 2012)

• Joerg Roedel: Introduce irq_remap_ops to x86 . (March 20, 2012)

• Kent Yoder: [PATCH 00/17] Platform Facilities Option and crypto accelerator driver . (March 21, 2012)

Security-related

• Will Drewry: seccomp_filter: syscall filtering using BPF . (March 15, 2012)

• John Johansen: AppArmor v2 - Add the ability to mediate mount . (March 15, 2012)

• Mimi Zohar: ima: appraisal extension . (March 21, 2012)

Virtualization and containers

• Raghavendra K T: Paravirtualized ticketlocks . (March 21, 2012)

Miscellaneous

• Lucas De Marchi: kmod 7 . (March 19, 2012)

• Theodore Ts'o: e2fsck logging improvements . (March 20, 2012)

• Kay Sievers: udev 182 . (March 20, 2012)

• Stephen Hemminger: iproute2 3.3.0 . (March 20, 2012)

Page editor: Jonathan Corbet

Distributions

Distributions looking at LLVM

The LLVM compiler infrastructure project and its Clang C front-end have been making strides over the last few years, to the point where some distributions are looking into using these tools more widely. We have already seen efforts to build the Linux kernel with Clang, but members of the Fedora and Debian communities (at least) are discussing going beyond that and building the entire distribution with an LLVM-based toolchain. While there are obvious benefits to trying that, it will likely be a ways off—if ever—before the benefits outweigh the costs of such a move.

The LLVM project started in 2000, but it was its adoption by Apple in 2005 that spurred much of its growth. The notoriously GPL-averse company has wanted to move away from using GCC for some time, and the BSD-ish licensed LLVM is the path that it has chosen. It is more than just a licensing issue, though, as LLVM has some technical advantages as well. But it is thought that GCC's move to GPLv3, with its more explicit patent provisions and anti-Tivo-ization language, has made the GCC to LLVM move that much more important to Apple. In any case, Xcode 4, the most recent version of the tool set shipped for building Mac OS X and iOS applications, now includes LLVM rather than GCC.

That change led "jonathan" to post a query to the fedora-devel mailing list about the status of using LLVM for building Linux packages. The query was a bit cryptic, but it spurred an interesting discussion about where LLVM is, and why (or why not) a distribution like Fedora might be interested in heading down that path. Matthew Garrett pointed out that LLVM is already used by the hfsplus-tools package (which provide utilities for the HFS+ filesystem) and the Mesa software rasterizer (llvmpipe), but beyond that:

In essence, Garrett is outlining the requirements for any new feature to be adopted into Fedora. Certainly performance is one of the advantages that is touted for LLVM; Apple claims that it compiles twice as fast as GCC while producing faster code. Undoubtedly, there are some programs where that is the case, but Adam Jackson has done some testing with the X server, and didn't find any huge performance increases for the LLVM-generated code:

(Jackson's footnote says that he should re-run the tests and file a GCC bug.)

In addition to Jackson's tests, Vladimir Makarov has done some benchmarking of GCC vs. LLVM (see the links in the lower left) that don't seem to show any major performance advantages for LLVM—in fact GCC looks like it more than holds it own. In addition, as he notes, the compilation speed comparison is often done with both compilers using the -O2 optimization level, but that it is fairer to compare GCC's -O1 with LLVM's -O2 in terms of generated code quality. When doing so, the 2x speed increase touted by Apple disappeared in his tests.

Debian developer Sylvestre Ledru has run an experiment rebuilding the Debian archive with LLVM. His focus was not on benchmarks, rather it was looking at how easily LLVM could handle the large diversity of C/C++ code in the archive. He was surprised to find relatively few problems: "I was expecting many issues and bugs caused by clang but I have been surprised to notice that most of the issues are either difference in C standard supported, difference of interpretation or corner cases."

Of the 15,600+ packages built, nearly 1400 failed and Ledru documented the kinds of problems he found. Some of the failures were for things like warnings that Clang emitted that GCC didn't, resulting in compilation failure because of the -Werror flag (turn all warnings into errors). In addition, he pointed out that the problems building packages are being fixed rapidly. The Clang 2.9 release in September 2011 failed on 14.5% of packages, while the 3.0 release from January only failed on 8.8%.

Those warnings produced by Clang are actually part of what Ledru was after with his test. One of the aims of LLVM is to generate better warnings and diagnostic information, which will be useful even for packages and distributions that never intend to use LLVM for production. In the end, Ledru concluded:

Whether that happens remains to be seen, of course, and in any event, it's not likely to happen overnight. But LLVM is definitely improving, and some of the BSDs are working on switching permanently (FreeBSD, for example; OpenBSD still seems most interested in pcc). Not having a BSD-licensed compiler that can build the kernel and user space has always been somewhat controversial in BSD-land, so switching to LLVM, which has a non-copyleft license (the University of Illinois/NCSA Open Source License), would be a step in the right direction.

There are definitely still plenty of hurdles to clear, especially for a distribution like Debian that supports lots of different architectures. GCC has a multi-year head start on supporting various CPU architectures, so LLVM may not be available for all of Debian's needs. Jackson pointed out some other deficiencies with LLVM, at least from his perspective:

One thing that the rise of LLVM has done is to provide competition for GCC, which has certainly been helpful in pushing GCC in new and interesting directions. As Jackson put it:

While the Linux kernel has been mostly built using LLVM, it is still a work in progress. GCC is still required to build some parts and there are changes needed to LLVM before that can change. Jeff Garzik said that he has been working on it, but "LLVM still needs several obscure compiler changes before we can even boot a no-op kernel".

There is little question that experiments and tests with LLVM are a good thing. Various bugs, in packages and compilers, will be ferreted out and both GCC and LLVM will improve. Some are concerned that Apple will rule the project in ways that could be detrimental to other projects (a la CUPS, which was mentioned by several fedora-devel posters), but there is little evidence of that occurring. It is free software, so, if that happens, there will be no barriers to continuing development. A bigger problem could be patents, which are not addressed by the LLVM license—some day a contributing patent-holder could potentially start suing. But that isn't a problem that is confined to LLVM.

It will be interesting to see how the adoption of LLVM goes. It seems likely to start with FreeBSD, but Linux distributions may eventually try it out as well. There will need to be compelling reasons to do so, but with the progress the compiler suite has made, those reasons may not be all that long in coming. Until correctly working kernels can be built, distributions obviously can't switch over completely. But, a complete switch is not necessarily a requirement, and there is nothing stopping interested distributions from building some of their packages from LLVM today.

Comments (32 posted)

Brief items

Distribution quotes of the week

-- Lars Wirzenius

-- Martin Owens

-- Greg Kroah-Hartman

Comments (none posted)

Fedora rejects Journal-only logging, ponders ARM

The minutes from the March 19 FESCO meeting include a couple of interesting conclusions. One is that the plan to incorporate the Journal as the new only logging system in Fedora 18 has been rejected. There was a lot of discomfort about binary logging formats, loss of syslog functionality, and more. Promoting the ARM architecture to "primary" status was also discussed with interest at the meeting, though the main conclusion was "more study." There has been a bit of complaining on the lists that most people did not even know that ARM was going to be discussed at this meeting. Full meeting details can be found in the IRC log.

Full Story (comments: 17)

GNU Linux-libre 3.3-gnu is now available

The Linux-libre project aims to build a Linux kernel free from binary blobs and other non-free software. It's used in several Free Software Foundation approved distributions such as gNewSense and Blag. Version 3.3-gnu is available.

Full Story (comments: none)

openSUSE 12.2 Milestone 2 is out

openSUSE has released the second milestone for the 12.2 release. Some of the new features that are making their way into openSUSE 12.2 include Grub2 and Plymouth and GCC 4.7. Jos Poortvliet covers more openSUSE news on his blog.

Comments (none posted)

Distribution News

Debian GNU/Linux

Bits from Debian Med team

Andreas Tille shares a few bits from the Debian Med project. In these bits you'll find out more about bug squashing, the project's anniversary, notes from a recent sprint, mentoring, Debian Developers that came to Debian through Debian Med, future plans, and lessons learned.

Full Story (comments: none)

delegation for the Press Team

Alexander Reichle-Schmehl and Meike Reichle have been active members of the Debian press team for many years (thanks!), but they have decided to step down. Francesca Ciceri and Neil McGovern have been promoted from assistants to press officers.

Full Story (comments: none)

Report from DSA Team Sprint

The Debian System Administration team recently met in Oslo, Norway for a sprint. "The goals of the meeting were to develop a long term plan for Debian's infrastructure, to review the current set of machines we maintain and the services we provide, and to formulate some policies and procedures regarding account and group management." This report covers Hardware & Sponsorships, Hosting & Virtualization, Service Changes, and User and Group Management.

Full Story (comments: none)

Fedora

It is time for another episode of: NAME THAT FEDORA RELEASE

Fedora Project Leader Robyn Bergeron announces that the naming process for Fedora 18 is open until March 27. "Frankly, I'm expecting an a-bun-dance of suggestions, and not just from people who cannot take my punni-ness any longer. Rules and guidelines for suggestions, as well as the location in which you may make your name suggestions, are on this wiki page: http://fedoraproject.org/wiki/Name_suggestions_for_Fedora_18"

Full Story (comments: none)

Ubuntu family

Mythbuntu changing to LTS only releases

The Mythbuntu team has announced that starting with the 12.04 release they will be moving to LTS only releases. Mythbuntu provides MythTV packages for Ubuntu and an Ubuntu variant integrating those packages. "MythTV packages will still be provided in the Ubuntu and Mythbuntu repositories for every Ubuntu release (both LTS and non-LTS)."

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (March 20)
• DistroWatch Weekly, Issue 448 (March 19)
• Maemo Weekly News (March 19)
• Ubuntu Weekly Newsletter, Issue 257 (March 18)

Comments (none posted)

"Anonymous" Linux sparks concerns (The H)

The H is reporting that a Linux distribution, supposedly created by the "Anonymous" activist group, has been taken down by SourceForge due to security concerns. The distribution is based on Ubuntu 11.10 with additional privacy and security tools like Tor. "In a statement released late last night, SourceForge explained that it had taken the distribution off its servers as significant concerns were raised concerning the software bundle's authenticity and possible maliciousness. SourceForge stated that while it tends to consider projects to be amoral and thus even host software that could be considered controversial, it decided to take Anonymous-OS down as soon as it became clear that it might include malicious software and did not appear to be officially connected with the Anonymous movement. Almost as soon as the release of Anonymous-OS was announced on a new Tumblr page, the activist group stated via its Twitter account that Anonymous-OS is a fake and contains trojans."

Comments (14 posted)

Bodhi Linux, the Beautiful Configurable Lightweight Linux (Linux.com)

Here's a review of Bodhi Linux on the Linux.com site. "The fancy profile greets you with a dozen virtual desktops and a shower of penguins, some of who sadly meet their demise. Then you can customize any of the profiles any way you like with different themes, Gadgets, whatever you want, and quickly switch between them without logging out. So you could have a work and home profile, a single and multi-monitor profile, a travel profile."

Comments (8 posted)

Fedora 16 And GNOME Shell: Tested And Reviewed (Tom's Hardware)

For those with some spare time: Tom's Hardware has posted a 28-page review of Fedora 16. "We chose to cover Fedora because it's the first top-tier Linux distro to adopt GNOME 3 as its default desktop. Far from a mere upgrade to GNOME 2, GNOME 3 introduces the radical new GNOME Shell graphical user interface (GUI). This article is just as much about GNOME 3 and its controversial new GNOME Shell than it is about Fedora."

Comments (172 posted)

Page editor: Rebecca Sobol

Development

Perl 5.16 and beyond

Perl 5's yearly release rhythm is well established. Because major releases come out every single year, a major release no longer introduces a slew of new features. Instead, it consists of a smaller set of features and bug fixes. Upgrading to a new major version is easier than it's ever been. So why upgrade at all? What's changed in Perl 5 recently, and where is it going?

Syntax extensions outside the core

One of the most important changes has been a push toward making the Perl 5 core more extensible. This work started with the Perl 5.12.0 release, and has continued since then. Many of the internal APIs have been cleaned up and documented. It has also become much easier to write extensions that work like Perl builtins, and even to extend the syntax in entirely new ways. Extensions can modify the "optree" while Perl is compiling your code. The optree is the Perl interpreter's internal representation of a piece of code. It's what the interpreter executes when it runs your program. When extensions are compiled down to ops, they can be as fast as an implementation in the core would be.

The smart match feature that was added in 5.10.0 provides a good example. Smart match includes the "given" and "when" builtins, as well as the "~~" operator. In the code, smart match examples can look like:

    $value ~~ @array
    @array ~~ qr/foo/

The behavior of ~~ varies based on the type of both operands. The first example checks to see if $value is a member of @array. The second example checks to see if any member of @array matches the regular expression. The Perl 5 Porters eventually realized that this feature's behavior is much too complicated. Unfortunately, this realization happened after the feature shipped, so we can't simply remove or change it out from under existing users.

What we need is a way to change the behavior in future releases while still providing access to the old behavior. We could, of course, just do this in the Perl core's C code. If the "enable old behavior" flag is on, we use the old code path, otherwise we use the new path. While this is feasible, it's not desirable. It clutters the core, and the more features that go down this path, the messier the core gets.

Jesse Luehrs's smartmatch module lets us alter the behavior of the smart match feature in a lexical scope by loading a module. The smartmatch::engine::core module implements the original behavior as introduced in 5.10.0. If we include "use smartmatch 'core'" in our own code, the smart match operator and builtins use the old behavior. Meanwhile, another module can include "use smartmatch 'sane'" and get a different behavior.

Because this module takes advantage of hooks in the Perl interpreter for syntax extensions, it can actually insert itself into the compiled Perl optree. This means that the extension can be as fast as the core implementation without being part of the core C code. When the smartmatch behavior in the core is changed, this module can be shipped with that release. We can even arrange for code that includes "use v5.14" to use the old smart match behavior, while code that includes "use v5.18" gets the new behavior.

This extensibility also makes it easy to prototype new features as CPAN modules. Florian Ragwitz's List::Gather module on CPAN adds gather and take "builtins" which compile down to operations in the parsed program. This syntax provides a nice way to create an array from an iterating operator:

    use List::Gather;
    
    my @list = gather {
        while (<$fh>) {
            next    if /^\s*$/;
            next    if /^\s*#/;
            last    if /^(?:__END__|__DATA__)$/;
            take $_ if some_predicate($_);
        }
    
        take @defaults unless gathered;
    };

This syntax could easily be included in the core simply by bundling the List::Gather module in the core distribution. All we need to do is make use v5.18 load List::Gather behind the scenes. There's no need to alter the core at all.

What's new in Perl 5.16.0

Perl 5.16.0 doesn't have any huge features, but it does have a collection of small features and bug fixes that will make Perl 5 better. In 5.16.0, we now have a __SUB__ token. This token returns a reference to the current subroutine, letting us write cleaner recursive closures:

    use feature 'current_sub';

    my $factorial = sub {
        my $val = shift;
        if ( $val > 1 ) {
            return $val * __SUB__->( $val - 1 );
        }
        else {
            return $val;
        }
    };

    print $factorial->(5);

The 5.16.0 release also adds support for the Unicode 6.1 standard, in addition to several other Unicode improvements. We now have much better support for Unicode characters in symbol names (packages, methods, etc.). The new fc operator and \F escape implement the Unicode foldcase operation, which does proper case-folding for all languages.

As part of the previously mentioned push to make the interpreter internals more extensible, 5.16.0 documents a number of functions for manipulating "pads". A pad (or scratchpad) is the data structure that stores lexical variables for each subroutine. This API was already in use by some modules, like List::Gather, but documenting it means that module authors can rely on the API's stability.

There has also been a lot of work on the core documentation. Perl 5.16.0 ships with a new object-oriented programming tutorial, and the object-oriented reference documentation has been rewritten from scratch with expanded coverage.

In addition to these changes, Perl 5.16.0 has many other bug fixes, performance improvements, documentation improvements, and core module updates.

A detour through the Perl 5 ecosystem

Just talking about the core when talking about the state of Perl 5 misses much of what makes Perl Perl. Perl's greatest strength has always been the Comprehensive Perl Archive Network (CPAN) and the larger Perl community. This is a huge topic, so I'll only hit a few highlights.

If you haven't looked at Perl recently, you may not have heard of Moose. Moose borrows from Common Lisp Object System, Perl 6, and many other languages. Moose provides a clean declarative API for declaring classes and roles. This eliminates huge amounts of boilerplate that Perl's native object system requires, allowing developers to focus on what their class does, rather than how it does it. Moose also provides a self-hosted metamodel, which means that it can be extended by writing classes and roles using Moose itself. Moose has been adopted by many CPAN authors for their own modules, and there are dozens of Moose extensions available.

Perl 5 has a number of excellent web frameworks available. Catalyst, Dancer, and Mojolicious are all mature, well-supported, and widely used. All of these frameworks build on top of the PSGI spec and Plack tools. These are inspired by Python's WSGI and Ruby's Rack respectively. Any application or framework that implements the PSGI spec can easily be deployed using FastCGI, standalone servers like Starman, mod_perl, or even plain old CGI. This makes writing and deploying Perl web applications easier than it's ever been.

Perl also has a number of Object-relational mapping modules, but the most popular is DBIx::Class. It has been under development since 2005, and has seen wide adoption throughout the community, attracting a number of core contributors, as well as inspiring dozens of extensions.

The services built around the CPAN archive are also exciting. The MetaCPAN API provides a web API to a database of every distribution ever uploaded to the CPAN archive. This API is open and freely usable, so anyone can build tools on top of it. There is already a new CPAN search site, also called MetaCPAN, that uses this API.

The CPAN Testers service collects test reports on every distribution uploaded to CPAN. Clients test the distributions on many different platforms and Perl versions. The service received its twenty millionth test report on March 7, 2012, and is currently receiving nearly 1 million reports a month.

The community is also busy organizing events. There are YAPCs (Yet Another Perl Conferences) this summer in the US, Germany, Brazil, and Japan, as well as many smaller workshops scheduled or in the works.

Future Plans for Perl 5

What happens after Perl 5.16.0? What will be in 5.18.0 or 5.20.0? Perl 5 development is volunteer driven, and I cannot commit anyone else's time. That said, here are some ideas that have been floated for future releases.

The project I'm most excited about is work on a MOP for the Perl 5 core. A MOP is a Meta Object Protocol. This is what Moose provides, but Moose does this as an extension. The goal is to create an API that can be implemented by the Perl 5 core and extended by modules like Moose. Putting this in the core has the potential to make modules like Moose much faster. However, the MOP is not just for Moose. It will be flexible enough to support multiple object systems, and will be usable as a minimal Object Oriented system all on its own, without extension.

As a bonus, this work will also include a few new bits of syntax. In particular, classes and methods will finally be distinct from packages and subroutines, and there will also be core support for roles, named method parameters, and attribute declaration.

I already mentioned the work on making the core more extensible. That work is an ongoing effort that opens up the possibility of breaking backward compatibility in a saner way, as with the smartmatch module example. This in turn frees up core developers to fix old design mistakes and introduce new features that might otherwise break old code.

Unicode work is also progressing. The 5.18.0 release will (hopefully) include support for set operations on Unicode character classes in regular expressions, as well as Unicode-related performance improvements.

While it's hard to predict the specifics of the future, I'm excited to see the activity and effort going into Perl 5 core development these days. The new release schedule, along with a move to Git, seems to have attracted some new contributors. Looking at the activity in the core and the community, Perl 5 is on a healthy path toward the future.

Comments (10 posted)

Brief items

Quotes of the week

-- Greg Kroah-Hartman

-- Ingo Molnar

-- Guido van Rossum

-- Éric Araujo (thanks to Zbigniew Jędrzejewski-Szmek)

Comments (62 posted)

Cinnamon 1.4 released

Cinnamon is a project started by Linux Mint, aimed at providing a GNOME2 experience using GNOME3 underpinnings. The project has announced the release of Cinnamon 1.4. The download page shows versions for Linux Mint 12, Ubuntu 11.10, Fedora 16, openSUSE 12.1, Arch Linux, Gentoo, and Frugalware. You can also grab the source from github.

Comments (53 posted)

Crossroads I/O release 1.0.0

Crossroads I/O is a fork of the ZeroMQ messaging system. The Crossroads developers say:

The 1.0.0 release is the project's first; it includes a few API changes and a ZeroMQ compatibility mode.

Full Story (comments: none)

Git project seeks discussion on "push" change

The Git project is considering a change to how the "push" operation works and is looking for comments from developers who may be affected by it. "In the current setting (i.e. push.default=matching), 'git push' without argument will push all branches that exist locally and remotely with the same name. This is usually appropriate when a developer pushes to his own public repository, but may be confusing if not dangerous when using a shared repository. The proposal is to change the default to 'upstream', i.e. push only the current branch, and push it to the branch 'git pull' would pull from."

Full Story (comments: 47)

H.264 support coming to Firefox

Mitchell Baker and Brendan Eich have both posted articles explaining the decision by the Mozilla project to add support for the H.264 video codec in Firefox if the underlying platform provides it. "What I do know for certain is this: H.264 is absolutely required right now to compete on mobile. I do not believe that we can reject H.264 content in Firefox on Android or in B2G and survive the shift to mobile. Losing a battle is a bitter experience. I won’t sugar-coat this pill. But we must swallow it if we are to succeed in our mobile initiatives. Failure on mobile is too likely to consign Mozilla to decline and irrelevance."

Comments (32 posted)

kconfig-frontends 3.3.0

Kconfig-frontends is a new project seeking to package the kernel's configuration system. "kconfig-frontends is not meant to replace the in-tree Linux kconfig (at least in the foreseeable future!), but really targets third-party projects that want an easy path to following the evolution of kconfig, without the burden to manually synchronise occasionally." The project plans to release shortly after each major kernel release.

Full Story (comments: none)

KDevelop 4.3 released

Version 4.3 of the KDevelop development environment has been announced. New features include beginning C++11 support, better version control system integration, better integration with the projects.kde.org site, improved source formatting, and a number of performance improvements.

Comments (2 posted)

notmuch 0.12

Version 0.12 of the notmuch email indexing system has been released. New features include reply-to-sender, better tagging operations in the Emacs interface, the ability to save attachments, Python 3.2 support, and more.

Full Story (comments: none)

xpra 0.1.0 released

Xpra calls itself "screen for X"; "it allows you to run X programs, usually on a remote host, direct their display to your local machine, and then to disconnect from these programs and reconnect from the same or another machine, without losing any state." Version 0.1.0, the first major xpra release, is now available. There is a long list of new features, most of which are performance- or security-related.

Comments (none posted)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (March 20)
• LibreOffice development summary (March 15; likely to be the last one)
• Perl Weekly (March 19)
• PostgreSQL Weekly News (March 18)
• Ruby Weekly (March 15)

Comments (none posted)

Van Rossum: Python is not too slow (InfoWorld)

InfoWorld has a short interview with Guido van Rossum, the creator of Python. In it, he talks about Python 3, Unicode, the Global Interpreter Lock (GIL), and more. "At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant."

Comments (103 posted)

Page editor: Jonathan Corbet

Announcements

Articles of interest

Seigo: Spark becomes Vivaldi

On his blog, Aaron Seigo writes about a name change for the Spark tablet: "Today, however, I get to share with you an adjustment in our naming scheme. We ran into a problem with trademark for the name we had chosen, and this pushed us back to the drawing board. This may have been a blessing in disguise as we looked into some of the broader pictures around this topic. In the end, we decided to go with a musical theme that celebrates some of the brightest lights in human history when it comes to making, playing and living interesting lives. For our debut product, the tablet formerly known as Spark, we have re-christened it: Vivaldi"

Comments (29 posted)

Greg KH Readies for Collaboration Summit, Talks Raspberry Pi (Linux.com)

Over at Linux.com Jennifer Cloer talks with Greg Kroah-Hartman about the Linux Foundation Collaboration Summit in April, and other topics. When asked how moderating a panel of developers differs from interviewing Linus Torvalds, he replied: "Seriously, it's much the same, but instead of just one person answering questions, there are 3 different viewpoints being offered, which can result in the conversation leading places you never expect. An example of this would be the kernel panel that happened last year at LinuxCon Japan, where the developers on stage got into a big technical argument with the kernel developers in the audience, much to the amusement of the rest of the audience. If done well, it can show the range of ideas the the kernel developer community has, and how while we don't always agree with each other, we work together to create something that works well for everyone."

Comments (none posted)

New Books

Programming Your Home--New from Pragmatic Bookshelf

Pragmatic Bookshelf has released "Programming Your Home" by Mike Riley.

Full Story (comments: none)

jQuery: Novice to Ninja, 2nd Edition--New from SitePoint

SitePoint has released "jQuery: Novice to Ninja, 2nd Edition" by Earle Castledine and Craig Sharkie.

Full Story (comments: none)

Calls for Presentations

international Openmobility conference 2012 - Call for papers

The Openmobility conference takes place April 21, 2012 in Prague, Czech Republic. The call for papers deadline is March 28. "Speak about your work/hobby/community, whatever is open & mobile or Open Source Hardware. I am looking for people from Openmoko community, GTA04, SHR and Qt Moko developers. But it's not limited only to these projects, you could represent MeeGo/Mer/Nemo, webOS, Open Source forks of Android, Panda Board, Arduino, Raspberry Pi. I am sure, I have forgotten other great projects like OsmocomBB etc. So, please spread this message in various communities. :-)"

Comments (none posted)

Libre Software Meeting (LSM), Call for Papers

The 13th Libre Software Meeting will take place July 7-12, 2012, in Geneva, Switzerland. The call for papers deadline is March 31.

Full Story (comments: none)

Upcoming Events

Linux Plumbers conference

Early registration is open for the 2012 Linux Plumbers Conference (LPC). The conference will be held in San Diego, California August 29-31 (run concurrently with LinuxCon).

Comments (none posted)

Events: March 22, 2012 to May 21, 2012

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
March 23 March 24	Cascadia IT Conference (LOPSA regional conference)	Seattle, WA, USA
March 24 March 25	LibrePlanet 2012	Boston, MA, USA
March 26 March 29	EclipseCon 2012	Washington D.C., USA
March 26 April 1	Wireless Battle of the Mesh (V5)	Athens, Greece
March 28	PGDay Austin 2012	Austin, TX, USA
March 28 March 29	Palmetto Open Source Software Conference 2012	Columbia, South Carolina, USA
March 29	Program your own open source system-on-a-chip (OpenRISC)	London, UK
March 30	PGDay DC 2012	Sterling, VA, USA
April 2	PGDay NYC 2012	New York, NY, USA
April 3 April 5	LF Collaboration Summit	San Francisco, CA, USA
April 5 April 6	Android Open	San Francisco, CA, USA
April 10 April 12	Percona Live: MySQL Conference and Expo 2012	Santa Clara, CA, United States
April 12 April 13	European LLVM Conference	London, UK
April 12 April 15	Linux Audio Conference 2012	Stanford, CA, USA
April 12 April 19	SuperCollider Symposium	London, UK
April 13	Drizzle Day	Santa Clara, CA, USA
April 16 April 18	OpenStack "Folsom" Design Summit	San Francisco, CA, USA
April 17 April 19	Workshop on Real-time, Embedded and Enterprise-Scale Time-Critical Systems	Paris, France
April 19 April 20	OpenStack Conference	San Francisco, CA, USA
April 21	international Openmobility conference 2012	Prague, Czech Republic
April 23 April 25	Luster User Group	Austin, Tx, USA
April 25 April 28	Evergreen International Conference 2012	Indianapolis, Indiana
April 27 April 29	Penguicon	Dearborn, MI, USA
April 28	Linuxdays Graz 2012	Graz, Austria
April 28 April 29	LinuxFest Northwest 2012	Bellingham, WA, USA
May 2 May 5	Libre Graphics Meeting 2012	Vienna, Austria
May 3 May 5	Utah Open Source Conference	Orem, Utah, USA
May 7 May 9	Tizen Developer Conference	San Francisco, CA , USA
May 7 May 11	Ubuntu Developer Summit - Q	Oakland, CA, USA
May 8 May 11	samba eXPerience 2012	Göttingen, Germany
May 11 May 12	Professional IT Community Conference 2012	New Brunswick, NJ, USA
May 11 May 13	Debian BSP in York	York, UK
May 13 May 18	C++ Now!	Aspen, CO, USA
May 17 May 18	PostgreSQL Conference for Users and Developers	Ottawa, Canada

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol