LWN.net Weekly Edition for March 3, 2011

SCALE: Understanding Unity

Canonical's Ted Gould presented a session about the Ubuntu project's Unity interface on February 27 at SCALE 9x. It would be a tad inaccurate to refer to the session as a "talk," though — Gould spent a few minutes touring Unity via slideshow and describing the goals of the project, but he reserved close to 45 minutes of the allotted hour for audience questions. That turned out to be a wise move: the audience filled the available Q&A time with a steady stream of questions about Unity's design, features, place in the desktop stack, distinction from GNOME Shell, and deployment in Ubuntu. For a high-visibility project nearly two release cycles old, it seems there is a noticeable user-confusion issue still to be solved. Partly that may just be the lot of all interface redesigns — after all, GNOME Shell certainly shares the challenge, as did KDE 4.

Unity basics unmasked

Gould's session was entitled "Unity: Why Does It Matter?," and it started with an account of the goals of the project. First, he said, Unity began as an attempt to integrate design with "soft skills" like psychology, sociology, and art, more directly into the software development process. This is the user-centric design paradigm, which is not historically the way open source software is shaped. Most often, open source fills the values important to enthusiasts (technical superiority, ability to tinker, power, and freedom), but lags on other values important to the "everybody else" crowd, such as predictability, discoverability, and look-and-feel enjoyability.

The headline changes in Unity's design stem from regular usability tests performed every six weeks at Canonical's offices in London. The process involves selecting volunteers and asking them to perform real-world tasks that they regularly undertake on their own computers, then observing and interviewing them on the results. Gould gave an example of someone who responded that they spent a lot of time uploading personal photos to Facebook, and would be asked to bring in a camera or flash drive with their photos and do the same thing on the test machine. He added that it is important that the developers and designers be present at the test sessions to talk to and get feedback from the volunteers, rather than just watch video or read quantitative results.

The biggest change in Unity is the redefining of the way the top panel functions. Users expressed trepidation about clicking on icons and menus in the panel because over the years, Windows and Linux systems have devolved the space into a catch-all for unrelated functions: menus that reveal selections when clicked, buttons that perform actions, useless eye-candy, and so on. Unity implements a strict menu-functionality-only policy referred to as application indicators. Any application can place an indicator in the menu, but clicking on it cannot itself perform an action; it can only reveal a drop-down menu.

The second piece of the system is the left-hand side launcher panel, which like many dock interfaces, holds large launcher buttons for favorite applications as well as icons for running applications, and shortcut buttons for the file browser, application browser, trash, and workspace switcher. Here, too, Unity implements a strict display policy, grouping all windows from the same application into one icon, allowing drag-and-drop reordering only within the application launcher segment, and presenting an API developers can use to add right-click functions.

Unity also presents a searchable "quicklist" application browser rather than a text-driven menu. The visual presentation is almost identical to GNOME Shell's (including categories), but it uses Zeitgeist on the back-end to search through recently-used applications, and the full descriptions in application .desktop files, not just the short name. Also like GNOME Shell, Unity places a number of common functions (including IM or presence options) in a "user menu," although unlike GNOME Shell, Unity uses a separate "system menu" for maintenance tasks and shutdown/reboot functions. Finally, Unity preserves the multiple workspaces/virtual desktop feature that has become a desktop mainstay, but implements it with a non-optional zoom-out/zoom-in effect that Gould said was found in testing to reduce confusion for first-time users.

Unity itself is a plug-in for the Compiz window manager, and relies on Compiz's existing functionality (or other plug-ins) for window stacking, focus, and transition effects. Although he described making it "beautiful" as one of the design goals, Gould said Ubuntu regards Unity as a "picture frame" designed to put emphasis on applications, not the desktop environment itself. Canonical's program of integrating testing and design in the development process, he said, is intended to inspire other projects to adopt a similar approach, and hopefully to take advantage of the new indicator and launcher APIs.

Questions and concerns

When the tour was complete, the questions started coming in from audience members. They fell into two major categories: first, questions about Unity's technical implementation, and second, questions about the changes it brings to user workflow when compared to the GNOME 2.x panel.

In the first category, it was clear that many users thought that Unity either implemented a new window manager, replaced core parts of the GNOME desktop stack other than the panel, or simply lived outside the rest of the desktop environment. For example, one audience member thought that Unity removed the ability to save files or application shortcuts onto the desktop; Gould explained that the desktop was untouched by Unity, and handled by Nautilus as it is today.

Several audience members asked about customization features, including whether the menu bar and launcher could be themed, resized, or moved. Gould explained that the background color, icons, and font used were inherited from the user's GTK+ theme, and would follow any custom theme choices made by the user, including accessibility settings like high-contrast colors or larger font sizes. Application indicators are single-color "symbolic" icons, but the menu bar tries to pick up whether the background color is "light" or "dark" and choose the appropriate icon color for visibility. The size of the menu bar and launcher cannot be changed in the current version, he said, but there is a patch in development. Similarly, a patch is in development to move the launcher to the right-hand side of the screen, which is a feature asked about by right-to-left language users.

Others asked about customizing window management features like focus behavior and window stacking or tiling. Here, Gould explained several times that Unity is a plug-in for Compiz, and relies on Compiz's existing features for that sort of function. As he put it, if you have a plugin that animates closing windows by burning them down with OpenGL-rendered flames, then they will burn down with OpenGL-rendered flames in Unity. By default, however, Ubuntu's version of Compiz will only ship with a basic set of Compiz plug-ins, and it is possible for users to install others that interfere with how Unity works. Similarly, Unity does not have its own "Expose"-like function to show thumbnails of all the open windows, because Compiz has its own function to do so.

There were multiple questions about accessing applications not permanently docked in the launcher. Gould's slide tour showcased the Zeitgeist-backed search function, which several seemed to interpret as meaning that there was no "browse" function (which there is), and that searching for an application by name was the only way to find it. That confusion might have been averted by highlighting the "Applications" button on the launcher during the tour.

Several thought that searching for an application by name was slower than using GNOME 2's application menu — particularly when you know where the application is. One person said that "docks" like the Unity launcher are historically used only for favorites, and asked whether there was any positive benefit to removing the application menu. Gould replied that the GNOME 2.x application menu suffered from overly broad categories (such as "Internet") and that in testing, users tended to know the name of the application that they wanted to use, but not where to find it among the categories. Thus, hitting the search button and typing the application name was faster than browsing through a menu or grid of application launchers. Also, he added, the search feature returns results based not only on the application name, but the long description in its .desktop file, which makes it possible to narrow down a search query further than an application category alone can.

The user workflow questions tended to be more personal. One audience member expressed concern over the lack of GNOME 2.x-style panel applets because she preferred to use them for rapid-access tasks like taking screenshots. Gould replied by saying that she could add the same screenshot application to the Unity launcher, and for convenience, move it to the bottom of the button list, away from the "favorites" launchers and nearer to the Workspace and Applications buttons.

Another audience member, who installs Linux for new converts, asked about assigning alternate names to applications (such as "Word" for Abiword or OpenOffice Writer), and another asked about custom application categories. Gould answered that there was not a built-in facility to do either, but because the search feature indexes .desktop files, adding alternate names or additional categories to the .desktop file would implement the same behavior.

Finally, there were several people under the impression that the 11.04 Ubuntu release that includes Unity would force them to use it. Gould replied that it was a login session option, and users who wanted to use the 2.x-style panels could choose to do so.

Change

After the talk, I asked Gould briefly about how he thought the Q&A session went. He expressed surprise at some of the questions, most of all the concern about using search to launch applications, instead of a menu, saying that he thought Google had conditioned everyone to search for things as the default way to access content. He also speculated that UI changes are always met with concern because of their potential to disrupt comfortable working patterns.

That is almost certainly true when you look at the "workflow" set of questions. Most of them trace back to audience members being concerned that the new interface will slow down common, repetitive tasks. GNOME Shell, which adopts many of the same features implemented in Unity, has met with a strikingly similar set of concerns. But in both cases, it is very rare that the new interface actually removes any functionality — it just rearranges where its features are located. Communicating those changes looks like an area where both projects could use some improvement.

For example, the features of the GNOME 2.x panel has been split into two separate pieces in Unity. Some applets that behave more like menus already (e.g., Tomboy), can easily use the application indicator API to present the same functionality. Others that behave more like buttons (e.g., the screenshot applet), can offer the same functionality in the launcher. Although Gould did not explicitly say so, the hit-search-and-type-the-application-name sounds like the same basic paradigm as GNOME Do. When he answered the audience questions, the questioners generally seemed relieved. Perhaps that sort of point-by-point comparison ought to be a standard part of Unity and GNOME Shell's introductory documents.

On the other hand, it seems like Canonical has not gotten the message out about what pieces of the GNOME desktop experience Unity does and does not change, and there is no obvious solution to that problem. People were confused about its affect on the desktop, its window management behavior, its inheritance from GTK+, and other core functions. Perhaps part of that is due to the choice of wording — such as referring to Unity as an "environment," rather than as a panel and launcher. But it may also stem from its origin in the Ubuntu netbook edition, which carries with it the suggestion of a "stripped down" environment. Consider the current Wikipedia article on Unity, which highlights Unity's "more efficient use of space" in the first paragraph, and contrasts it with the "full" GNOME environment below. I would never suggest that anyone take Wikipedia as authoritative, but it is good for assessing the current general-consensus-viewpoint.

Because open source software is almost always developed by remote teams (in Canonical's case, often working from home), maybe it will always struggle to consistently introduce major new features. Most of what Gould explained in his answers is documented somewhere on the web, either on a wiki page or in blog posts syndicated on Planet Ubuntu, and yet even those users dedicated enough to spend their weekend at SCALE had misconceptions and questions. GNOME recently held an IRC "user day" where core developers and designers met to answer questions about GNOME 3.0 and GNOME Shell, and had a similar experience: even those users experienced enough to be comfortable with IRC had an endless barrage of questions.

It is always enlightening to see developers interact face-to-face with end users, and that is one of the best services offered by community shows like SCALE. In my estimation, the take-away message for the Unity team is the disconnect between the lay-person's understanding of Unity and what it actually is and can do. How to improve on that, though, is not so obvious. Come up with the answer to that one, and I guarantee you will draw a packed house at SCALE or any other open source software event.

Comments (17 posted)

Replacing regexps

The compactness of the regular expression syntax is part of their power, but also part of the problem. Consider even a very simple expression:

    <A\b[^>]*>(.*)</A>

A reader familiar with this syntax will recognize that this expression matches the HTML <A> tag and sets aside the anchor text for later processing. But even experienced regular expression developers must look at that expression for a moment and think about how the various metacharacters affect each other before being able to say for sure what it does. It takes even longer to notice the subtle bug: this expression will be confused by the presence of multiple <A> tags in the text being searched.

So how might one do better? That was Mike Meyer's question as he sought a more "pythonic" way of doing text matching. Needless to say, he is not the first to ask that kind of question; there are a number of attempts at better string matching out there. The first of those is arguably not "pythonic" at all: it is SnoPy, a port of the venerable SNOBOL language to Python.

SNOBOL was developed during the 1960's; it included pattern matching as a core feature of the language. Unlike regular expressions, SNOBOL was anything but concise. Concatenation of strings was explicit, "[abc]" was "Any("abc")", and so on. Nonetheless, SNOBOL was highly influential in this area, and one can see echoes of the language in current regular expressions. That said, SNOBOL is not heavily used now, and the Python SNOBOL module seems to have suffered the same fate; its last release was in 2002.

Another approach is the rxb.py module by Ka-Ping Yee. This module, posted in 2005, creates a new, relatively verbose but relatively readable language for the creation of patterns. Using this language, the regular expression shown above would look something like:

    <A + any(wordchars + whitespace)> + label(1, anychars) + </A>

(Note to readers; the above is totally untested and should not be relied upon for production use). This module, too, has not seen a great deal of use.

Various other packages are out there. For example, one can try to use Icon-style pattern matching with Python. For something completely different, there is the eGenix mxTextTools module, which allows the creation of text-matching programs in an assembly-like language, complete with goto constructs. mxTextTools is intimidating and not necessarily any easier to read than regular expressions, but it is said to be powerful and fast, and there are a number of real users.

Still, none of these seem likely to replace regular expressions as the first tool Python programmers reach for when they need to perform string matching. Python creator Guido van Rossum thinks things will stay that way:

Pushing aside an established incumbent is always hard, and regular expressions are well established indeed. It is never enough to simply be better in this situation; the proposed replacement has to be a lot better. As Guido noted, nothing seems to have come along which is that much better, and it may be that nothing ever will. For some medium-term value of "ever," anyway.

But, then, one also should not underestimate the ingenuity of free software developers. Or their persistence. People will almost certainly continue to throw themselves against this problem, and, maybe, somebody will come up with something interesting. Until then, we'll have to continue beating our heads against our desks as we try to figure out why our expressions don't work as intended.

Comments (42 posted)

SCALE: Honeywell on Hackerspaces

This year's Southern California Linux Expo (SCALE) ventured into non-Linux territory with one of its keynotes. The kick-off keynote from Leigh Honeywell, Hackerspaces and Free Software, delved into the seemingly new trend of Hackerspaces. I say seemingly, because it turns out that hackerspaces have been around for decades — they've just become much more noticeable recently, particularly in North America.

For those not familiar with hackerspaces, and it seemed that quite a few in the audience were not, Honeywell described them as a "third space" intended to foster "a certain kind of creativity." The term third space comes from Ray Oldenburg, who used the term to describe community spaces where people gather for creative interaction. (A first space being the home, and second space being work.) To that end, hackerspaces act as "sort of a library for stuff" where you will find all manner of equipment that members may not be able to invest in individually — such as laser cutters and 3D printers.

Hackerspaces have been in operation, though perhaps not under that particular name, for 20-plus years in Europe. The Chaos Computer Club (CCC), Metalab, and others have been around for decades. If you haven't heard of CCC in reference to hackerspaces, you still may have heard of its brainchild Project Blinkenlights and/or its role in fighting biometrics in Germany. Honeywell recounted the story of CCC lifting German interior minister Wolfgang Schauble's fingerprints and publishing them in Die Datenschleuder after he supported collecting biometrics from German citizens to fight terrorism.

But if you haven't heard of them recently, perhaps you haven't been paying attention — even the Wall Street Journal took notice of the trend in 2009.

Hackers on a plane

How did hackerspaces make the leap across the pond? Honeywell said that it started in North America with Hackers on a Plane. The "wild trip" was organized in 2007 by Hacker Foundation founder Nick Farr, and took about 40 North American hackers through European hackerspaces to try to spread the idea. According to Wired's coverage, Farr said: "This is expensive, but I think the good works we'll see over the next few years will justify the trip. We're hoping that this trip winds up being a watershed moment for the U.S. scene." Apparently, it not only kickstarted the U.S. scene but overshot to Canada and inspired Hacklab.to, Honeywell's home hackerspace.

Hacklab.to was founded in July of 2008 with 35 members and has about 200 people on a public mailing list. You'll find much more than a 3D printer and laser cutter at Hacklab.to. The about page lists most of their tools for public use. Those include glue guns and glue sticks, a sewing machine, hand tools, a Tektronix Digital Storage Oscilloscope, and a Van de Graaff generator — and, of course, a fire extinguisher and first aid kit.

The space also has goodies for computer geeks: a hefty server and storage capacity, with a 1TB NAS, and PXE Boot environment so members can drop in and do PXE installs of things like Ubuntu. The space also has two laptops, and a workstation for members to use while they are there.

Members get a storage bin at the space, the option to bring in guests and organize events at the space, and access to the WiFi network, private wiki, and mailing lists. (Hacklab.to also has a public lists, and Honeywell recommends that other hackerspaces do so as well.) Honeywell says that one of the interesting questions about hackerspaces is whether they serve as physical extensions for Internet groups, or whether the mailing lists, IRC, and wiki are extensions of the physical space.

But it's not all hack, hack, hack. The space also has "food tools," including a stove, fridge, dishwasher, and the requisite pots, pans, dishes, and glassware. The space is open 24/7. Access is controlled by RFID and a PIN-based deadbolt. When members come in, the access is announced on IRC. (The site also says that access is announced on Twitter, but it must not be the main account.)

Honeywell described some of the events and projects that members have worked on at Hacklab.to — including refurbishing the laser cutter, and creating an Arduino-based device that dispenses candy when a member runs the dishwasher. Honeywell noted that it's a continual challenge to get people to wash their dishes, just like any shared space.

The other advantage of a shared space, says Honeywell, is that you'll usually find someone to help you do something that you want to do "like in-person IRC." Honeywell says that the people are an equally important part of a hackerspace.

One near you: No excuses

Honeywell said that if you live in a reasonably large city, there's a good chance there's already a hackerspace near you. The Hackerspaces site has an extensive list of spaces all over the world. The United States has more than 140 listed, including my home hackerspace Arch Reactor.

But if there is no hackerspace nearby, Honeywell advised the audience to create their own rather than packing bags for a city that has its own. Honeywell says that there are "no excuses" not to start a hackerspace of your own if your chosen city lacks one. If you live in a small town, then rent should be cheap and require fewer members. You don't need to start with a 3D printer or laser cutter — the important thing is to start a space and see what it cultivates.

While Honeywell maintained that there was no excuse not to start a hackerspace, there are challenges — some of which may be familiar to participants in open source projects, others specific to working together in meatspace. As with any business, there's the question of money: How much will dues be? How are memberships structured?

Just like instructions for creating your own gadgets from an Arduino, there are design patterns for creating your own hackerspace. Honeywell said that the patterns are just that — patterns. They can be adjusted or ignored in some cases, but provide a set of guidelines that can be useful. Some examples:

• Meet every week on Tuesdays.
• Don't let people sleep there (too often).
• Don't bother with plants, they will die.
• Set up a mailing list, wiki, and IRC channel.

Why Tuesdays? Honeywell says that every day sucks — Tuesdays work just as well as any other day, because any day of the week will be a problem for someone. You will need to set up a set of rules that fit your hackerspace. Honeywell pointed out that one rule at Hacklab.to is "no food, no humans in laser." This is a good rule, but not necessary until your space actually has a laser.

Honeywell said one common misconception is that hackerspaces are all about hardware. While there is a lot of hardware hacking that takes place, she said that hackerspaces can also be a good place for software hacking or joint hardware/software projects, as well as classes for learning new skills. The Hacklab.to community is a good example of this, with Arduino workshops, LaTeX workshops, and collaborating with other hackerspaces on a "Cupcake Challenge" to mail a cupcake at least 1,600 kilometers in "pristine condition."

Which brought around the last topic of the talk — recruiting people who are not like you. Honeywell said that a hackerspace should "bait newbies," and have open days and events that bring in new blood. She also mentioned that people drop in and out of hackerspaces depending on what's going on in their lives, so there needs to be a continual effort to recruit new members. For example, Honeywell is leading a Soldering Workshop for Women at Hacklab on March 14 to encourage more women to get involved.

One of the things that is particularly interesting about the hackerspace movement is that it's an excellent opportunity for free and open source folks to rub elbows with people who may not (yet) be FOSS proponents. Hackerspaces can attract all manner of "hackers" who may or may not be software-oriented, and provide opportunities to meet people who may know nothing about open source but all about re-wiring a laser cutter. But it is a good opportunity for collaborating on free software as well, especially if your area has user groups or enough people interested in hacking on a project.

Comments (15 posted)

Page editor: Jonathan Corbet

Security

Seunshare, /tmp directories, and the "sticky" bit

The /tmp directory has been an unceasing source of security problems going back decades; there are still regular reports of vulnerabilities from insecure usage of temporary files. Part of the problem is that /tmp (and /var/tmp) are shared resources that can be written to by any process, which allows attackers to use various race conditions (typically time-of-check-to-time-of-use (TOCTTOU) races) in insecurely written programs to elevate their privileges. It is a bit ironic, then, that a utility specifically geared toward running a program with a private /tmp directory (for application sandboxing) would run afoul of a somewhat different kind of temporary file vulnerability—one that was long-ago excised by the advent of "sticky" directories. But that is just what Tavis Ormandy found.

The basic problem is that insecure programs often open files in /tmp after checking to see whether the file exists. In the window between the time that the test is done and the time that the file is opened, a malicious program can swap in a file of its choosing (or, more likely, a symbolic or hard link to a file of its choosing). When that happens, the buggy program is operating on a file that it does not expect and that can cause all manner of mayhem. For normally privileged programs, that mayhem is largely restricted, but for setuid programs, it can lead to full system compromise.

Long ago, attackers could use the world-writable attribute of /tmp to delete files that were created by setuid programs. The attacker could then replace the file with a link, and when a privileged program re-opened the file—something that is, in general, a bad practice with temporary files—it would be opening a file of the attacker's choice. But, the advent of the "sticky" bit as applied to directory permissions closed that loophole by only allowing the file owner (or root) to delete a file in a sticky directory. Since that time, lots of code has been written with a sticky /tmp directory in mind.

As part of its efforts to use SELinux to provide application sandboxes, Red Hat created the seunshare utility. That utility will run a command with alternate /tmp and home directories, along with a given SELinux context. seunshare will "unshare" the default mount namespace (so that the command has its own view of the filesystem hierarchy), mount the specified directories over top of /tmp and the home directory, and instruct the kernel to execute the command in the (optionally) given SELinux context. Since the temporary directory specified is under the control of the user, it doesn't necessarily have the sticky bit set, which leads to the vulnerability.

In Ormandy's example, he uses ksu to show how the /etc/passwd file could be overwritten by running ksu under seunshare. There are likely other setuid programs that make the assumption that their temporary files are in sticky directories, and quite possibly some where the consequences could be more severe than just trashing the password file. So a mechanism that was meant to provide more security actually left a hole behind. Unfortunately, this is not an uncommon occurrence in the security realm.

This particular case also shows the value of disclosing security vulnerabilities. Ormandy reported the bug back in September and, though there was a flurry of discussion about it, that discussion died off in late November (at least in the bug report). Things didn't pick up again until Ormandy posted a request for an update, along with notice that he was ready to publish an advisory, on February 18. Hearing no complaint, he did so on February 23.

After that, the discussion picked up again, with solutions being proposed, though no fix is yet available for Fedora or RHEL. One has to wonder how long this potential local privilege escalation might have languished had Ormandy not released his advisory. As a temporary mitigation, Ormandy suggests removing the setuid bit from seunshare or restricting access to it. The solution that Dan Walsh has proposed removes the -t tmpdir argument to seunshare and instead mounts a tmpfs on /tmp (with the sticky bit set). Presumably that will be released in the near future.

There has been an attempt to harden the behavior of sticky directories to try to avoid some of the longstanding /tmp directory problems—though that would not have thwarted this particular vulnerability because it relies on the directory being sticky. There has been resistance to that effort because it is seen as something of an ugly hack to work around badly written code, so it has not made it into the mainline (though Ubuntu and other kernels do have that hardening). But temporary file vulnerabilities of various sorts still rear their head with depressing frequency. We will undoubtedly see others crop up in the future.

Comments (6 posted)

Brief items

Security quotes of the week

Comments (1 posted)

Wallach: Things overheard on the WiFi from my Android smartphone

Comments (21 posted)

Firefox and Thunderbird security updates

Mozilla has released Firefox 3.6.14 and 3.5.17 and Thunderbird 3.1.8, each of which fix some security vulnerabilities, including some that are marked "critical". Mozilla strongly recommends that all users upgrade to the new releases. Each Firefox release fixes eight critical, one high, and one moderate vulnerability (3.6.14, 3.5.17), while the Thunderbird release fixes two critical, and one moderate flaw (3.1.8).

Comments (1 posted)

New vulnerabilities

abcm2ps: multiple vulnerabilities

Comments (none posted)

acroread: multiple vulnerabilities

Comments (none posted)

avahi: denial of service

Comments (none posted)

clamav: arbitrary code execution

Comments (none posted)

firefox: multiple vulnerabilities

Comments (none posted)

fuse: denial of service

Comments (none posted)

gimp: multiple vulnerabilities

Comments (none posted)

kernel: denial of service

Comments (none posted)

logwatch: privilege escalation/arbitrary code execution

Comments (none posted)

openjdk: privilege escalation

Comments (none posted)

pam-pgsql: buffer overflow

Comments (none posted)

pango: arbitrary code execution

Comments (none posted)

php: casting vulnerability

Comments (none posted)

ruby: multiple vulnerabilities

Comments (none posted)

samba: denial of service

Comments (none posted)

wireshark: code execution

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 2.6.38-rc7, released on March 1. "There really isn't a lot to report here. Driver updates (random one-liners and some sound soc codec and smaller dri updates) and a few filesystem updates (in particular btrfs fiemap and ENOSPC handling), but most of it really is pretty tiny. Regressions fixed, hopefully none introduced." Full details can be found in the long-format changelog.

Stable updates: The 2.6.37.2 stable kernel was released on February 24. The 2.6.32.30 longterm kernel was released on March 2, with a note of appreciation: "Many thanks again to Maximilian Attems who dug around in a lot of different distro kernels and forwarded to me the original git commit ids that should be applied to this tree. Red Hat didn't make this very easy due to their "one giant patch" format, and his skill is helping everyone out here."

Comments (7 posted)

Quotes of the week

Comments (5 posted)

The debloat-testing kernel tree

Comments (31 posted)

Intel announces a BIOS Implementation Test Suite (BITS)

Full Story (comments: 28)

Red Hat's "obfuscated" kernel source

One of the key points behind the RPM and Debian package formats is that source is shipped in its upstream form, with patches shipped separately and applied at build time. Red Hat has always followed this convention; the failure to do so with the RHEL 6 kernel is a new and discouraging change of behavior. Distribution in this form should satisfy the GPL, but it makes life hard for anybody else wanting to see what has been done with this kernel. Hopefully it is simply a mistake which will be corrected soon.

Comments (250 posted)

Kernel development news

Waking systems from suspend

But let's just say, if you were an extraordinary cat-napper, and you had some downtime between numerous kernel compiles while doing a long git-bisect: You could make it work, but first you would need a good alarm clock. The same can be said of computers.

The RTC

The RTC (Real Time Clock) is a fairly minor bit of hardware on your computer. It usually keeps track of the wall-clock time while the system is off or suspended. It also can be used to generate interrupts in a number of different modes (periodic, one-shot alarm, etc). This is all fairly normal functionality for a hardware timer device. But one of the most interesting features that most modern RTCs support is that an alarm interrupt can be generated even when the system is suspended (or in some hardware hibernation) forcing the machine to wake up.

On Linux the RTC is exposed to user space via the generic RTC driver infrastructure, which creates sysfs entries and a character device which can be used to set hardware alarms, change the interrupt mode, etc. A few applications out there make use of this interface, such as MythTV DVRs, which can trigger alarms so that media computers can be suspended until the start of a TV show that needs to be recorded.

The exposed interface is very much a low-level driver interface, where the values written by the application are sent directly to the hardware. This is a limitation, as it makes it so only one application at a time can program alarm events to an RTC device. For instance, with only a single RTC device, you can't have your system wake up for a nightly backup and also have it wake up to record your favorite show, unless you have some sort of centralized process managing the wakeups on behalf of other applications. Tutorials such as this one illustrate how complex and limiting this interface can be.

One way to overcome these limitations is to allow the kernel to manage a list of events and have it program the RTC so the alarm will trigger for the earliest event in the list. This avoids the need for user space applications to coordinate in order to share the hardware. To make this sharing possible, a generic "timerqueue" abstraction has been created to manage a simple list of timers that could then be shared with other areas of the kernel, like the high-resolution timers subsystem, that also have to manage timer events. This code was merged for 2.6.38.

The next step is to rework the RTC code so that, when an alarm is set via the character device ioctl() or sysfs interface, an rtc_timer event is created and enqueued into the per-RTC timerqueue instead of directly programming the hardware. The kernel then sets the hardware timer to fire for the earliest event in the queue. In effect, this mechanism virtualizes the RTC hardware, preserving the behavior of the existing hardware-oriented interfaces, while allowing the kernel to multiplex other events using the RTC.

The question now becomes, how to expose this new functionality so it can be used?

CLOCK_RTC

The first approach tried was exporting the new RTC functionality to user space directly via the POSIX clocks and timers interface. With this approach, there is a "clockid" assigned to each RTC device, so a user space application can use the POSIX interfaces to access the RTC. In this approach, clock_gettime() returns the current RTC time, clock_settime() sets the RTC time, and timer_settime() sets a POSIX timer to expire when the RTC reaches the desired time.

This approach is the most straightforward method of exposing the RTC, but it does have some disadvantages. Specifically, the RTC and system time may not be the same. On many systems, the RTC is set to local time rather than universal time. Thus, applications would need to make the extra effort to read the RTC and add to that value the time between now and when they want the timer to fire. Also, the RTC, due to simple clock skew, may not increase at the exact same rate as the system time. Additionally, since there may be multiple RTCs on a system, a single static CLOCK_RTC clockid would not be sufficient. Some form of dynamic clock_id registration is needed in order to export multiple clockids for multiple RTC devices. This functionality is desired for exposing other hardware clocks via the POSIX interface, and it is currently a work-in-progress by Richard Cochran.

Android Alarm Timers

Interestingly, the developers who have been working on Android have extended the RTC to be more useful as well. After all, smartphones are optimized to save power, so they try to stay in suspend as much as possible. But smartphones still have to wake up to do things like notify the user of calendar events or to check for email. In order to do this, The Android team introduced a concept called Android Alarm Timers. These timers use a hybrid approach: when when the system is running, alarm timers trigger a high-res timer to fire when an event is supposed to run; however, when the system goes into suspend, the alarm timers code looks at the list of events and sets the RTC to fire an alarm when the earliest event is to run. This avoids making applications deal with the (possibly unsyncronized) RTC time domain and allows applications to simply set timers and have them fire when expected, whether or not the system is suspended.

While never submitted to the kernel mailing list for inclusion, the Android Alarm Timers implementation would likely meet some resistance from the kernel community. For instance, the user-space interface for applications to use the Android Alarm Timers is via ioctl() to a new special character device (/dev/alarm) instead of using existing system call interfaces. Additionally, the ioctl() interface introduces new names for existing concepts in the kernel, duplicating CLOCK_REALTIME (which provides UTC wall time) and CLOCK_MONOTONIC (which counts from zero starting at system boot, and is not modified by settimeofday() calls) via the names ANDROID_ALARM_RTC and ANDROID_ALARM_SYSTEMTIME respectively.

The Android Alarm Timers interface does introduce some new useful concepts. For instance, the CLOCK_MONOTONIC clock does not increment during suspend. This is reasonable behavior when you want suspend to be transparent to applications, but when the system spends the majority of its time in suspend and you want to schedule events that wake the system up having only CLOCK_REALTIME increment over suspend can be limiting. So Android Alarm Timers introduces the ANDROID_ALARM_ELAPSED_REALTIME clock, which is similar to CLOCK_MONOTONIC, but includes time spent in suspend. But again, it is only introduced via an ioctl() to their special character device, and is not exposed via any other standard timekeeping interface.

Posix Alarm Timers

All in all, the Android Alarm Timers are a very interesting use case, and others in the community have suggested a similar hybrid approach. Inspired by the Android Alarm Timers, I implemented a similar hybrid alarm timers infrastructure on top of the previously-described work virtualizing the RTC interface. However, these timers are exposed to user space via the standard POSIX clocks and timers interface, using the new the CLOCK_REALTIME_ALARM clockid. The new clockid behaves identically to CLOCK_REALTIME except that timers set against the _ALARM clockid will wake the system if it is suspended. Additionally, because it's built upon the virtualized rtc_timers work, this implementation doesn't prohibit applications from making use of the existing legacy RTC interfaces. This gives us all the benefits of Android Alarm Timers, such as not forcing applications to deal with the RTC time domain, while making better use of existing kernel interfaces.

The code that implements the timerqueues and reworks the generic RTC layer to allow for multiplexing of events has been included in the 2.6.38 kernel release. The POSIX alarm timers layer will likely need additional review and discussion, in hopes of making sure the Android developers are able to assess compatibility issues in the design. For instance, I've proposed a new POSIX clock (CLOCK_BOOTTIME, along with a corresponding CLOCK_BOOTTIME_ALARM id) which would provide the incrementing-in-suspend value that the Android developers introduced with ANDROID_ALARM_ELAPSED_REALTIME. Also, while not likely to be included into mainline, Android's wakelocks have some interesting semantics with regards to their alarm timer interface. These semantics are not easily satisfied by the posix timers interface, but it is to be determined if we can get equivalent functionality using modified semantics and the mainline kernel's pm_wakeup interface.

Other open questions that need to be addressed are:

• What capabilities should applications be required to have in order to set POSIX alarm timers?

• In order to avoid systems waking up at inappropriate times (think laptop in a bag in the overhead compartment), should there be additional policy layers added so that user-generated suspends (like closing a laptop) inhibit POSIX alarm timers?

I also can imagine some interesting future work combining this functionality with the "Wake on Directed Packet" feature of some new network cards, which wake the system up any time a packet is sent to it. This feature could be used to allow web servers to function normally, servicing requests and running jobs, while suspending and saving power during longer idle periods.

While I might not be able to sleep on the job, I look forward to my desktop system being able to snooze and save electricity while knowing that cron jobs like nightly backups, downloading package updates or running updatedb will still be done.

Comments (18 posted)

Capabilities for loading network modules

Linux capabilities are still a work in progress. They have been in the kernel for a long time—since the 2.1 days in 1998—but for various reasons, it has taken more than a decade for distributions to really start using the feature. While capabilities ostensibly provide a way to give limited privileges to processes, rather than the all-or-none setuid model, the feature has been beset with incompleteness, limitations, complexity concerns, and other problems. Now that Fedora, Openwall, and other distributions are working on actually using capabilities to reduce the privileges extended to system binaries we are seeing some of those problems shake out.

A patch that was merged for 2.6.32 is one such example. The idea behind it was that the CAP_NET_ADMIN capability should be enough to allow loading network modules, rather than requiring CAP_SYS_MODULE. The CAP_SYS_MODULE capability allows loading modules from anywhere, rather than restricting the module search path to /lib/modules/.... So, by switching to use CAP_NET_ADMIN, network utilities, like ifconfig, could be restricted to only load system modules, rather than arbitrary code.

There is one problem with that scheme, though, as Vasiliy Kulikov pointed out, because it allows processes with CAP_NET_ADMIN to load any module from /lib/modules, not just those that are networking related. Or, as he puts it:

    root@albatros:~# grep Cap /proc/$$/status
    CapInh: 0000000000000000
    CapPrm: fffffffc00001000
    CapEff: fffffffc00001000
    CapBnd: fffffffc00001000
    root@albatros:~# lsmod | grep xfs
    root@albatros:~# ifconfig xfs
    xfs: error fetching interface information: Device not found
    root@albatros:~# lsmod | grep xfs
    xfs                   767011  0 
    exportfs                4226  2 xfs,nfsd

That example deserves a bit of explanation. The first command establishes that the capabilities of the shell are just CAP_NET_ADMIN (capability number 12 of the 34 currently defined capabilities). Kulikov then goes on to show that the xfs module is not loaded until he loads it via ifconfig. That is clearly not the expected behavior. In fact it is now CVE-2011-1019 (which is just reserved at the time of this writing). For those that want to try this out at home, Kulikov gives the proper incantation in his v2 patch:

    # capsh --drop=$(seq -s, 0 11),$(seq -s, 13 34) --

Note that on not-quite-bleeding-edge kernels (e.g. Fedora 14's kernel), the 34 should be changed to 33 to account for the lack of a CAP_SYSLOG, which was just recently added. Running that command will give you a shell with just CAP_NET_ADMIN.

Kulikov's first patch proposal simply changed the request_module() call in the core networking dev_load() function to only load modules that start with "netdev-", with udev expected to set up the appropriate aliases. There are three modules that already have aliases (ip_gre.c, ipip.c, and sit.c) in the code, so the patch changes those to prefix "netdev-". But David Miller was not happy with changing those names, as it will break existing code.

There was also a bit of a digression regarding attackers recompiling modules with a "netdev-" alias, but unless that attacker can install the code in /lib/modules, it isn't a real problem. In this case, the threat model is a subverted binary that has CAP_NET_ADMIN, which is not a capability that would allow it to write to /lib/modules. But Miller's complaint is more substantial, as anything that used to do "ifconfig sit0", for example, will no longer work.

After some discussion of various ways to handle that problem, Arnd Bergmann noted that the backward compatibility problem is only for systems that are not splitting up capabilities (i.e. they just use root or setuid with the full capability set). For those, the CAP_SYS_MODULE capability can be required, while the programs that only have CAP_NET_ADMIN will be new, and thus can use the new "netdev-" names. The code will look something like:

    no_module = !dev;
    if (no_module && capable(CAP_NET_ADMIN))
	    no_module = request_module("netdev-%s", name);
    if (no_module && capable(CAP_SYS_MODULE)) {
	    if (!request_module("%s", name))
		    pr_err("Loading kernel module for a network device "
    "with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev-%s "
    "instead\n", name);

That solution seemed to be acceptable to Miller and others, so we may well see it in the mainline soon. One thing to note, though, is that capabilities are part of the kernel ABI, so changes to their behavior will be difficult to sell, in general. This change is fixing a security problem—and is hopefully not a behavior that any user-space application is relying on—so it is likely to find a reasonably smooth path into the kernel. Other changes that come up as more systems start to actually use the various capability bits may be more difficult to do, though we have already seen some problems with the current definitions of various capabilities.

Comments (11 posted)

Who wrote 2.6.38

The 2.6.38 cycle has seen 9,148 non-merge changesets from 1,136 developers (again, as of this writing). Compared to 2.6.37 (11,446 changesets from 1,276 developers) those numbers may seem small, but they are on a par with most other recent kernel releases:

Release	Changes	Devs
2.6.34	9,443	1,151
2.6.35	9,801	1,188
2.6.36	9,501	1,176
2.6.37	11,446	1,276
2.6.38	9,148	1,136

603,000 lines of code were added in this cycle, and 312,000 were removed, for a net growth of 291,000 lines of code. The most active contributors of that code were:

Most active 2.6.38 developers

By changesets Joe Perches 199 2.2% Chris Wilson 182 2.0% Russell King 147 1.6% Mark Brown 143 1.6% Tejun Heo 107 1.2% Ben Skeggs 107 1.2% Alex Deucher 97 1.1% Eric Dumazet 88 1.0% Felix Fietkau 88 1.0% Mauro Carvalho Chehab 83 0.9% Thomas Gleixner 79 0.9% Jesper Juhl 75 0.8% Lennert Buytenhek 72 0.8% Johannes Berg 70 0.8% Stephen Hemminger 70 0.8% Al Viro 68 0.7% Andrea Arcangeli 67 0.7% Clemens Ladisch 66 0.7% Uwe Kleine-König 66 0.7% Nick Piggin 65 0.7%

By changed lines Vladislav Zolotarov 42524 5.8% Nicholas Bellinger 30797 4.2% Larry Finger 23439 3.2% Hans Verkuil 20978 2.9% Barry Song 14174 1.9% Dimitris Papastamos 12794 1.7% Ben Skeggs 11651 1.6% Rafał Miłecki 11149 1.5% Sven Eckelmann 11081 1.5% Mike Frysinger 10692 1.5% Sonic Zhang 8360 1.1% Michael Chan 8280 1.1% Chris Wilson 8164 1.1% Mark Brown 7690 1.0% Chuck Lever 7457 1.0% Joe Perches 7185 1.0% Shawn Guo 6440 0.9% Paul Walmsley 5671 0.8% Mark Allyn 5424 0.7% Nick Piggin 5402 0.7%

Joe Perches made it to the top of the "by changesets" with a long list of patches removing excess semicolons and casts, adding "static" keywords, and other things of that nature. Chris Wilson's changes were entirely in the Intel graphics driver subsystem, Russell King remains active as the lead ARM maintainer, Mark Brown does large amounts of work in the sound driver subsystem, and Tejun Heo had patches all over the tree, most of which are related to cleaning up workqueue usage.

Vladislav Zolotarov's path to the top of the "lines changed" column ostensibly should not exist anymore; among his many bnx2x driver changes was a large firmware replacement. Nicholas Bellinger is the main author of the LIO SCSI target patches which were merged, after extensive discussion, for 2.6.38. Larry Finger added the Realtek RTL8192CE/RTL8188SE wireless network adapter to the staging tree, Hans Verkuil continues his work straightening out the Video4Linux2 subsystem, and Barry Song added a number of IIO drivers to the staging tree.

Work on 2.6.38 was supported by a minimum of 180 employers, the most active of whom were:

Most active 2.6.38 employers

By changesets (None) 1544 16.9% Red Hat 1145 12.5% Intel 664 7.3% (Unknown) 654 7.1% Novell 383 4.2% IBM 334 3.7% (Consultant) 315 3.4% Texas Instruments 290 3.2% AMD 184 2.0% Broadcom 172 1.9% Wolfson Micro 170 1.9% Nokia 169 1.8% Oracle 136 1.5% Samsung 133 1.5% Google 133 1.5% Atheros 132 1.4% Analog Devices 115 1.3% Fujitsu 112 1.2% Pengutronix 109 1.2% Renesas Tech. 107 1.2%

By lines changed (None) 133902 18.2% Broadcom 97317 13.2% Red Hat 56561 7.7% Intel 44650 6.1% Analog Devices 41083 5.6% Rising Tide Systems 31869 4.3% (Unknown) 30462 4.1% Wolfson Micro 25167 3.4% Texas Instruments 24193 3.3% IBM 16124 2.2% Novell 13939 1.9% (Consultant) 13789 1.9% Freescale 11454 1.6% Nokia 10535 1.4% Oracle 10415 1.4% ST Ericsson 9521 1.3% Renesas Tech. 8534 1.2% Samsung 7988 1.1% AMD 7950 1.1% Oki Semiconductor 7087 1.0%

The most significant new entry is Rising Tide Systems, a storage array company which, unsurprisingly, has an interest in the kernel's SCSI target implementation. Otherwise, the entries at the top of the table have changed little over the last few years; here is a plot showing the trends since 2.6.28:

There is a certain amount of noise, but, over this entire period, non-paid contributors are at the top of the list, followed by Red Hat and Intel, in that order. The most significant trends, perhaps, are TI's steady increase over time, and IBM's slow decline.

Regardless of what individual companies do, though, the real picture that emerges from this data is that the kernel development process remains strong and active. The rate of change remains high, and the community from which those changes come remains large and diverse. There may come a time when the kernel community runs out of ideas and things to do, but it does not seem that things will slow down anytime soon.

[As always, thanks are due to Greg Kroah-Hartman for his assistance in the creation of these numbers. The tool used to calculate these statistics is "gitdm"; it can be had at git://git.lwn.net/gitdm.git. The associated configuration files can be downloaded here.]

Comments (8 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.38-rc7 . (March 1, 2011)

• Greg KH: Linux 2.6.37.2 . (February 25, 2011)

• Greg KH: Linux 2.6.32.30 . (March 2, 2011)

• John W. Linville: ANNOUNCE: debloat-testing kernel git tree . (February 26, 2011)

Core kernel code

• Thomas Gleixner: genirq: Forced threaded interrupt handlers . (February 24, 2011)

• ntl@pobox.com: container-based checkpoint/restart prototype . (March 1, 2011)

• Tejun Heo: Proposal for ptrace improvements . (March 1, 2011)

• Tony Ibbs: RFC: KBUS messaging subsystem . (March 2, 2011)

• Arnd Bergmann: Final BKL removal, take 2 . (March 2, 2011)

• Peter Zijlstra: sched: Reduce runqueue lock contention -v5 . (March 2, 2011)

Development tools

• Burt Triplett: Intel BIOS Implementation Test Suite (BITS) . (February 24, 2011)

• Shubham Goyal: [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. . (March 2, 2011)

• Hui Zhu: Linux Kernel GDB tracepoint module 2011-03-02 . (March 2, 2011)

Device drivers

• Jon Brenner: TAOS 258x: Device Driver . (February 26, 2011)

• Bill Gatliff: Generic PWM Device API . (March 1, 2011)

• Richard Cochran: ptp: IEEE 1588 hardware clock support . (March 1, 2011)

• Po-Yu Chuang: net: add Faraday FTMAC100 10/100 Ethernet driver . (March 1, 2011)

• Alan Cox: gma500: Add 2D acceleration . (March 2, 2011)

• Nicholas A. Bellinger: iSCSI target v4.1.0-rc1 series . (March 2, 2011)

Filesystems and block I/O

• Kyungmin Park: fat: Batched discard support for fat . (February 24, 2011)

• Andrea Righi: blk-throttle: async write throttling . (March 1, 2011)

• Miklos Szeredi: overlay filesystem prototype . (March 1, 2011)

• Theodore Ts'o: Proposed design for big allocation blocks for ext4 . (March 2, 2011)

• Theodore Ts'o: Add basic BIGALLOC support for cluster-based allocation . (March 2, 2011)

Memory management

• Peter Zijlstra: mm: Preemptibility -v9 . (March 2, 2011)

Networking

• Krishna Kumar: [RFC] Implement multiqueue (RX & TX) virtio-net . (March 2, 2011)

• Stephen Hemminger: sched: QFQ - quick fair queue scheduler (v2) . (March 2, 2011)

Architecture-specific

• Corentin Chary: Support Asus WMI capable Notebooks (asus-wmi, asus-nb-wmi and eeepc-wmi) . (February 26, 2011)

• Greg KH: Platform: add Samsung Laptop platform driver . (March 1, 2011)

• Lin Ming: perf: x86, add SandyBridge support . (March 1, 2011)

Security-related

• Jari Ruusu: Announce loop-AES-v3.6b file/swap crypto package . (February 26, 2011)

Virtualization and containers

• Serge E. Hallyn: [PATCH 01/10] Add a user_namespace as creator/owner of uts_namespace . (February 24, 2011)

• Ian Campbell: xen network backend driver . (March 2, 2011)

Miscellaneous

• Pablo Neira Ayuso: conntrack-tools 1.0.0 released . (March 2, 2011)

Page editor: Jonathan Corbet

Distributions

SCALE: Projects and distribution unfriendliness

Fedora engineering manager Tom "spot" Callaway, who actively maintains a Fedora-compatible package repository for the Chromium web browser, said Friday he has "given up" on getting the browser — which does work well on Fedora — into the distribution proper. In November 2009, he explained why Chromium is not an official Fedora package on his blog, and those issues remain. In a talk at the Southern California Linux Expo (SCALE) entitled, "This is why you FAIL," Callaway, who maintains more than 350 packages in Fedora, listed some of what he sees as "points of FAIL," or distribution-unfriendly software development practices, using Chromium as an example for many of them.

"Having your software be distribution-friendly is a key to success," he said. In the Fedora Activity Day event earlier at SCALE, users brought in systems in a variety of "states of disrepair", he said, which were caused by attempting to install third-party software. Distribution-friendliness also reflects on other channels, he said. The same "points of FAIL" that affect distribution maintainers are also problems for intermediaries who are putting the software on embedded devices or running it as a hosted service.

In the talk, which summarized his chapter, "How to tell if a FLOSS project is doomed to FAIL" from the book The Open Source Way, Callaway discussed the bundled dependency problem. It's worth FAIL points to include a private copy of a library on which a program depends, and extra FAIL points for building a modified version. A key problem, Callaway said, is that when a distribution does a security update for a library, it also has to do updates for any packages that include their own copies. (Fedora has a No Bundled Libraries policy, and Fedora package maintainers do modify the build process for software that the distribution packages, in order to make it build with a system copy of the library.) "Chromium is perhaps the worst offender I have ever seen in my entire career," Callaway said.

Chromium developer Evan Martin, who was not at the event but had posted a list of third-party code distributed with Chromium on his blog, replied to that in email:

Callaway did recognize that some versions of system libraries might not work. The build process should make the system copy the default, though. If the build configuration comes up with "I found this library and I can't use it because it has rabies or something," then it could build an alternate copy. Martin said that the Chromium team is willing to accept outside contributions to facilitate this:

Chromium requires a copyright assignment agreement for code changes, which Callaway said he has not accepted. "After review with a lawyer, they advised me that agreeing to that would give Google a license to use my contributions under any copyright license terms that they'd like, including non-free terms," he said later. "I'd be more than willing to give them my changes under the terms of a Free license, but Google wants to continue to distribute a proprietary version of their browser (Google Chrome), and I have no interest whatsoever in helping them with that effort," he added.

Much of the advice in Callaway's talk applied not to large-scale projects like Chromium, but to new, lower-profile projects. He reserved some of the FAIL warnings for projects that don't offer basic documentation, such as how to do a build and how to begin interacting with the source control system. "Lots of programmers coming through school have never seen a source control system," he said. A project web site should include instructions for how to check out the code and how the project wants to receive incoming code changes. It also counts for substantial FAIL if "all your web site has is a picture of a marijuana leaf," as he said one small-scale open source project does.

The build and install process is another key area. "If your code forces an install into /opt or /usr/local you're probably running Oracle and I'm very very sorry," he said. Running "make install" should just work. "Make the decision that you're going to have an installable program that works outside the source directory."

Some software comes in a problematic archive format; RAR archives are a problem, he said, because the format is proprietary. A developer once asked Callaway about making Fedora packages for his new archiving tool, which came in an archive created by the same tool. "He didn't see why this was a problem and I couldn't tell him because I was crying," Callaway said.

A history of having been proprietary is worth more fail the longer it was proprietary before the initial open source release. "Red Hat has bought some real shiny turds," he said, giving Netscape's email server as an example. "We buried it in the backyard," he said.

The good news is that many of the points of FAIL are relatively easy to correct. Including a copy of the software license and setting up a mailing list are minor tasks. The bundled dependency problem, on the other hand, has turned out to require lots of skilled attention from both the project maintainers and distributions, so that one is still with us.

Both library-bundlers and library-splitters have their points. Bundling creates more long-term issues for administrators, but library-splitting takes up valuable development time, especially for cross-platform projects that need to bundle the libraries for target platforms that don't practice library splitting. But bundled libraries are a problem for many Linux distributions and one that we will likely be facing for some time to come.

Comments (7 posted)

Brief items

Fedora 13 ARM Beta Release

Comments (none posted)

FreeBSD 8.2-RELEASE Available

Full Story (comments: none)

Mandriva 2011 Alpha2

Comments (none posted)

MeeGo 1.1 update 3 and MeeGo 1.0 update 7

Full Story (comments: none)

OpenEmbedded Release-2011.03

Full Story (comments: none)

openSUSE 11.4 RC2

Full Story (comments: none)

PC-BSD 8.2 Released

Comments (none posted)

Distribution News

Debian GNU/Linux

Bits from ARM and Embedded Sprint

Full Story (comments: none)

Red Hat Enterprise Linux

Red Hat Lifecycle Support for Red Hat Enterprise Linux 4

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (February 28)
• DistroWatch Weekly, Issue 394 (February 28)
• Fedora Weekly News Issue 264 (February 23)
• openSUSE Weekly News, Issue 164 (February 27)

Comments (none posted)

Linux Mint 11 "Katya" to use GNOME 3 (The H)

Comments (none posted)

Shuttleworth: Mistakes made, lessons learned, a principle clarified and upheld

Comments (78 posted)

Page editor: Rebecca Sobol

Development

Choosing between portability and innovation

Portability is a key concept in the open source ecosystem. Thanks to this killer feature, your author has migrated his desktop operating system during the last ten years from Mac OS X to Linux (various distributions) and eventually to FreeBSD, but throughout that process he could keep using most of the same applications. When you present a recent openSUSE or PC-BSD desktop system to a computer newbie, they won't notice much difference, apart from a different desktop theme, perhaps. The same applications (OpenOffice.org, Firefox, K3b, Dolphin, and so on) will be available. In many circumstances, it just doesn't matter whether your operating system is using a Linux or FreeBSD kernel, as long as it has drivers for your hardware (and that's the catch).

This portability, however, is not always easy to achieve. Now that Linux is the most popular free Unix-like operating system, it shouldn't be a surprise that some projects have begun treating non-Linux operating systems as second-class citizens. This isn't out of contempt for the BSDs or OpenSolaris, it's just a matter of limited manpower: if almost all the users of the application have a Linux operating system and if all the core developers are using Linux themselves, it's difficult to keep supporting other operating systems. But sometimes the choice to leave out support for other operating systems is explicitly made, e.g. when the developers want to implement some innovative features that require functionality that is (at least for now) only available in the Linux kernel.

Xfce 4.8 and udev

In January, version 4.8 of the Xfce desktop environment was released. In the beginning of its announcement, the developers expressed their disappointment that they couldn't offer all the new features of the release on the BSDs:

This somewhat cryptic remark was followed by a summary of the new features, but it was clear that it was aimed at the new desktop frameworks introduced in the last few years, such as udev, ConsoleKit and PolicyKit. udev is only available on Linux, but both ConsoleKit and PolicyKit are already supported by FreeBSD, so as LWN.net commenter "JohnLenz" supposed correctly in a comment on the announcement, the problem is for a large part on the testing side: how many FreeBSD users are using these frameworks? And how many of them test these frameworks regularly and spend the time to report bugs?

The remark in the release announcement probably puzzled a lot of BSD enthusiasts as well, because Xfce developer Jannis Pohlmann followed up a few days later with an explanation on his personal blog. There he named udev as the culprit for the non-portability of some Xfce features:

But then Pohlmann points to the broader context:

Some Linux users, who may be used to six-month release cycles, might not see a problem here, as they now have all those new features. But the BSD operating systems generally have a much slower development life cycle and haven't caught up yet with the whole redesign of the desktop stack. The comments on Pohlmann's blog post are instructive in this regard (although somewhat degenerating into a flame war at the end). For example, one commenter points out that HAL did acquire BSD (and Solaris) support, but only years after it had been mainstream in the Linux world, and the BSD developers only contributed the necessary patches to make it work when Gnome and KDE started making HAL mandatory.

The problem seems to be that udev is not as easy to port to non-Linux systems as HAL was. FreeBSD has the devd daemon to handle volume mounting, but devd's author Warner Losh commented that udev is not well documented, which hampers efforts to port it. However, this didn't stop Alex Hornung from porting udev to DragonFly BSD, although it's not yet a full drop-in replacement. The FreeBSD developers could take a look at his work, as DragonFlyBSD is a FreeBSD derivative.

OpenBSD developer Marc Espie also points to license issues: udev and other software close to the Linux kernel is using GPLv2, which the BSDs don't like to use. For example, OpenBSD developers don't add a component to the base system if it's less free than the BSD license, and the GPL is such a license in their eyes. However, the current problems are also clearly a consequence of different development styles. Components like udisks are part of the Freedesktop specifications (which are supposed to keep X Window System desktops interoperable), but the BSD developers didn't seem to participate in that effort. Maybe the PC-BSD developers can play a role in this, as they want to deliver a modern desktop operating system based on FreeBSD.

All in all, there are two possible solutions to a situation like the one the Xfce 4.8 release is facing. One solution is that the Xfce developers create an abstraction layer supported by as many operating systems as possible. The problem is that currently there is no such abstraction layer for detecting devices, which makes it perfectly understandable that the Xfce developers chose udev. It is used by their major development platform, Linux, and one can't expect them to support frameworks on operating systems they don't use. So the other solution is that some BSD developers port udev to their operating system, which is non-trivial but (as the incomplete DragonFly BSD port shows) doable, or that they propose an abstraction layer that could be supported on non-Linux platforms. As many desktop applications have already been rewritten in the last few years from using HAL to using udev, the latter won't be a popular choice and isn't likely to happen.

X.Org and KMS

Another important desktop component that is becoming more and more Linux-centric in recent years is X.Org. Recent open source video drivers (such as the Nouveau driver) require kernel mode setting (KMS), which is a problem for the BSDs and OpenSolaris, as these operating systems lack kernel support for KMS and Graphics Execution Manager (GEM). So a FreeBSD user who wants to get decent performance out of an Nvidia graphics card, currently has to use the proprietary driver. Fortunately, the FreeBSD Foundation recognized the gravity of this situation and announced last year that it was willing to sponsor a developer to work on KMS and GEM support in the FreeBSD kernel. Last month, the foundation announced that it had awarded a grant, co-sponsored by iXsystems (the developers of PC-BSD) to Russian developer Konstantin Belousov to implement support for GEM, KMS, and Direct Rendering Infrastructure (DRI) for Intel hardware. Matt Olander, Chief Technology Officer at iXsystems, says in the announcement:

More specifically, Belousov will implement GEM, port KMS, and write new DRI drivers for Intel graphics cards, including the latest Sandy Bridge generation of integrated graphic units. After this work, users should be able to run the latest X.Org open source drivers for Intel on their FreeBSD desktop. While the project is limited to Intel graphics, porting other drivers like Nouveau to FreeBSD will become a lot easier once Belusov's work is completed. And when KMS support is in place, FreeBSD users could run the X Server without root privileges, run the Wayland display server, and get access to a lot of other features that are until now only available on Linux.

This case also shows that cutting edge development often happens with Linux primarily in mind. During the last few years, X.Org's drivers were in a constant state of flux, with new technologies like KMS, GEM, translation-table maps (TTM), DRI, Gallium3D and so on being introduced one after another. As these are low-level technologies tightly coupled to the Linux kernel, porting them to FreeBSD is no small task, but fortunately the FreeBSD Foundation and iXsystems have seen that it is very important to follow the lead of Linux here.

systemd

An entirely different case is systemd: Lennart Poettering has no problem with the fact that systemd is tightly glued to Linux features. In a recent interview for FOSDEM, Poettering sums up the Linux-specific functionality systemd relies on: cgroups, udev, the fanotify(), timerfd() and signalfd() system calls, filesystem namespaces, capability sets, /proc/self/mountinfo, and so on. And then comes this quote, explaining why he designed systemd with Linux in mind:

Poettering took this decision because of his experience in writing some other low-level components in the desktop stack:

He even goes further with this provocative invitation to other developers to do the same:

Poettering touches some interesting points here. We have a family of standards that are known as POSIX (Portable Operating System Interface for uniX), defining the API of a Unix operating system. However, the POSIX specifications are not carved in stone and there are few operating systems that are fully compliant (Mac OS X is one of them since the Leopard release). POSIX is really an encapsulation of some choices that various Unix systems made along the way, rather than a body of text that got standardized and then implemented. According to Poettering, Linux should use its position as "market leader" (in the market of free Unix-like operating systems) and try out some new things. If developers don't force themselves into the constraints of the POSIX API, they could develop some really innovative software, like systemd shows. When these new developments happen to turn out really interesting, other operating systems could eventually adopt them as well.

The tension between portability and innovation

These three cases clearly show that there's a constant tension between portability and innovation, which are two important qualities of open source software. In a lot of domains, Linux is taking the lead with respect to innovation, and the BSDs are forced to follow this lead if they don't want to be left behind. While the BSDs will probably not be interested in adopting systemd, implementing KMS is a must-have because one cannot imagine a modern X.Org desktop any more without it. But the biggest portability problems will be in the layers right above the kernel that don't have suitable abstraction layers, such as the Xfce case shows. Will FreeBSD implement udev or will the problem be solved another way? These kinds of questions are important and choosing when to use the POSIX or the Linux API is a delicate balancing act: choosing a Linux-centric approach for a low-level component like systemd is understandable because of the performance and maintenance gains, but most applications won't necessarily benefit from that approach.

But maybe the biggest problem these cases hint at is that Linux development is being done at such a fast pace that other operating systems just can't keep up. Linux distributions and Linux-centric developers are used to the "release early, release often" mantra, including swapping out key components and breaking APIs each release. The BSD world doesn't work that way, and this makes working together on a modern cross-platform open source desktop increasingly difficult. The innovation of Linux inevitably comes at a price: Linux is the de facto Unix platform now, and hence more and more programs will not be portable to other operating systems.

Comments (126 posted)

Brief items

Quotes of the week

+ log_warning("/usr appears to be on a different file system than /. "
+             "This is not supported anymore. "
+             "Some things will probably break (sometimes even silently) "
+             "in mysterious ways.");

Comments (81 posted)

Burt: Canonical's New Plan for Banshee

Comments (20 posted)

Developments in GNU Radio

Comments (1 posted)

Mercurial 1.8 released

Full Story (comments: none)

SciPy 0.9.0 released

Full Story (comments: none)

Upstart 1.0 released

Full Story (comments: 6)

xorg-server 1.10.0

Full Story (comments: 1)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (March 1)
• PostgreSQL Weekly News (February 27)
• Tcl-URL! (March 1)

Comments (none posted)

An Early Look at GNOME 3.0 (Linux.com)

Comments (45 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

The Ada Initiative Announces Advisory Board

Full Story (comments: none)

The Document Foundation achieves its fundraising goal

Full Story (comments: 8)

MeeBlip synthesizer in the wild

Comments (1 posted)

Yocto project and OpenEmbedded align

Comments (14 posted)

Articles of interest

DoD and Open Source Software

Comments (none posted)

Sony's War on Makers, Hackers, and Innovators (Make)

Comments (none posted)

Open source expert takes on the hardest job at Microsoft (Network World)

Comments (12 posted)

U.K. Comes out for Royalty-Free Standards for Government Procurement (The Standards Blog)

Comments (12 posted)

Implementing Open Standards in Open Source (Software Tech News)

Comments (none posted)

Tom's Definitive Linux Software Roundup: Audio Production (Tom's Hardware)

Comments (5 posted)

New Books

The Book of Audacity--New from No Starch Press

Full Story (comments: none)

Programming Amazon EC2--New from O'Reilly

Full Story (comments: none)

Calls for Presentations

Desktop Summit CFP announced and Registration open

Full Story (comments: 6)

2011 Linux Symposium Call for Participation

Comments (none posted)

13th Real Time Linux Workshop - 1st Call for Papers

Full Story (comments: none)

Upcoming Events

LibreOffice conference announced for October 12-15

Full Story (comments: none)

Events: March 10, 2011 to May 9, 2011

If your event does not appear here, please tell us about it.

Audio and Video programs

Allison: Geek Time with Jim Zemlin

Comments (none posted)

Page editor: Rebecca Sobol