LWN.net Weekly Edition for February 23, 2012

Changes and complaints

Free software is always progressing—at least by someone's definition of "progress". Over the last few years, or more, we have seen huge controversies erupt around changes that were being made to desktop environments, system software, and other parts of the free software stack. The controversies themselves are not that surprising, as people are often resistant to change (and some proposed changes are not necessarily good ones), but the tone and manner of the complaints is sometimes rather eye-opening. It often sounds like some believe that the developers of these projects set out to ruin people's lives with their changes. Somehow that doesn't seem likely, and starting a discussion from that standpoint seems rather unlikely to change anyone's mind.

It is unquestionably frustrating and maddening to upgrade a package or distribution and find that things no longer work they way they once did; that workflows or other habits are now "obsolete". But, presumably, the upgrade was done for a reason, typically to get new features and fixes. With upgrades come dangers, of course. Sometimes those dangers are from new bugs or incompatibilities, but other times the danger is that something that once "worked" no longer does.

This is a problem that is in no way restricted to the free software world. One could easily argue that the problem is far worse for proprietary systems, as upgrades are sometimes forced, which is pretty uncommon for free software. Distributions do reach their end of life, but there are plenty of alternatives to consider when that happens. Somehow, choices like Windows Mint or Mac OS X "Wheezy" don't seem to be in the cards. Nor does installing an alternative desktop environment, display server, init system, kernel, or audio subsystem. For the most part, you get what the proprietary vendors give you and you like it—or not.

It could also be argued that the free software approach (to the extent there is a single "approach") does not lead to desktop dominance. That has, of course, been argued ad nauseam here and elsewhere. But the fact of the matter is that hackers and projects are making their own decisions, freely, based on their interests and understanding of the problems they are solving. No one has set out to annoy anyone. That may be a side effect of the changes that are being made, but it certainly isn't the intent.

If you peer into the development mailing lists or talk to the proponents of these kinds of changes, they clearly see them as improvements. It's a little hard to imagine that folks would spend significant amounts of their time making the code worse. Opinions will differ on the changes, of course, and making one's opinion known is a time-honored tradition in free software (not to mention the internet and human society in general). But the vehement, sometimes demanding, tone that those complaints take is counter-productive. Part of what seems to be lacking is a certain of level of respect for the projects and the people behind them.

A much more effective way to make one's opinion known is through engagement. Unlike the proprietary world, we in the free software world can directly engage with the developers, describe the problems that we see, and try to change the direction of the project in a way that is more suitable for our needs. Most free software projects discuss planned changes well in advance of their implementation and give users lots of opportunities to try out early versions. But engaging the project is best done with well-reasoned, specific descriptions of problems, missing features, and so on—not endless streams of "Project XYZ sucks!" messages to mailing lists or comment threads.

Beyond just offering suggestions and/or complaints, though, we can also pitch in and fix the problems that we see. Given the large number of vocal opponents of various changes (and proposed changes) that we have seen, there should be a ready supply of developers and others to continue maintaining older code bases, or to work on getting features added to fill in gaps. Even when a project moves on from an earlier version (a la GNOME 3 or KDE 4), it's not like the old code disappears. We have seen some efforts (like the Trinity desktop environment for KDE 3 and MATE for GNOME 2) but, despite all the complaints, a big community has not sprung up around either.

Part of the problem is that the intersection between those who are vocally unhappy and those with the time and skills needed to help is probably fairly small. Free software thrives where people participate, but folks tend to want to work on things that scratch their own itch. If there aren't enough participants with the "right" itches, few of the complaints will actually get solved.

Another related problem is that there seems to be an increasing sense of entitlement from some within our communities. The huge base of free software that we use today has been given as, essentially, a gift, from tens of thousands of contributors worldwide. Just like "those who write the code get to choose the license", "those who work on the project get to set its direction". Users are, of course, an important piece of the puzzle, but frothing, name-calling, endless complaining does not rise to the level of a contribution.

It may well be that some or all of the problems that people see in various projects (GNOME 3, KDE 4, Wayland, systemd, the Journal, grub2, ...) are serious and will drive users away from the projects or the distributions that adopt them. But there's really only one way to find out. If users vote with their feet and move elsewhere, one suspects the projects will follow along behind.

None of this is meant to downplay the importance of users making their problems known, but there is a point at which the repetitive, often content-free, complaints don't serve any purpose—other than venting perhaps. These projects have most certainly heard most of the complaints by now, sometimes an enormous number of times. Have they seen constructive attempts to address those problems in even a small fraction of that volume? That, unfortunately, seems unlikely.

In the end, though, people eventually either get used to the changes or find some alternative that suits them better. Sometimes that alternative involves a fork of the existing code and it is not unheard of that the fork eventually rejoins or takes over from the original project. The EGCS/GCC split comes to mind, for example, and we may be seeing something like that play out again with LibreOffice and Apache OpenOffice. It's also worth remembering that GNOME 2 was originally met with a lot of unhappiness, perhaps even from some of the same folks that are now complaining that GNOME 3 "took away" things they were used to in GNOME 2. Over time, these things have a way of working out, but in the meantime, it's worth at least thinking about whether that venom-filled post is really going to be very effective.

Comments (175 posted)

LibreOffice releases version 3.5.0

Just over eight months after its last major release, the LibreOffice project has unveiled version 3.5.0 of its open source, cross-platform office suite. The enhancements are numerous, with improvements touching each of the major application components, along with several brand new features. The project has been busy in other respects over the past few months as well, incorporating a foundation, launching a new support service, and exploring post-desktop interface options.

As always, the new LibreOffice builds are available in 32-bit and 64-bit flavors for Linux, packaged in Debian or RPM format, in addition to bundles for Windows and Mac OS X. According to the release notes, 3.5.0 is unchanged from RC3, so bleeding-edge types may not need to upgrade. There are a few warnings in the notes as well, most notably that Windows users cannot upgrade directly from 3.4.5 (so they must uninstall 3.4.5 first), and that Linux users are strongly encouraged to use OpenJDK instead of GCJ.

Users should also expect some difficultly with Microsoft Office 2010 not recognizing some of the new features in Open Document Format (ODF) 1.2 that are now implemented in LibreOffice for the first time. If all of those caveats do not scare you off, however, there is much to see in the new release.

Suite-wide changes

LibreOffice underwent an extensive cleanup in the 3.5 development cycle; Michael Meeks noted the removal of more than 3,000 unused methods between July and December 2011. The subsequently leaner-and-meaner code base adds a few new suite-wide features worth mentioning, starting with support for Java 7 (although Java 6 is still supported as well, for backward compatibility).

The updater and extension manager have been made more user-friendly, allowing users to configure how frequently LibreOffice should check for new releases, and letting users sort extensions by their origin (i.e., those bundled with LibreOffice versus those installed by the user). There are several important UI changes, including clearer alert messages when a user tries to save a file in a format that will lose formatting information, and an easier-to-understand "more" indicator for when the toolbar menu is too small to display all of the buttons.

At the lower levels, password-protected ODF files are now encrypted by AES-256, replacing the weaker Blowfish cipher used in previous versions. LibreOffice now also incorporates the ttfautohint font hinting engine (which we looked at in November), which will improve rendering quality, particularly on Windows. Also on the text handling front, LibreOffice supports SIL International's Graphite font format, and ships with Graphite versions of the Libertine font family, which include a number of new typesetting features [PDF] in the latest update.

Finally, the Base database front-end (which can be used by the other components in the office suite) now includes native drivers for PostgreSQL. 3.5.0 supports versions of Postgres up through 8.3; support for 8.4 and newer is slated to arrive with LibreOffice 3.5.1.

Writer

Easily the most talked-about new feature in 3.5.0 is the integration of a grammar checker into the Writer word processing component. There have been several grammar-checking extensions in the past (and those are still installable), but this is the first such built-in tool. The grammar checker is called LightProof, and it supports English, Hungarian, and Russian.

There is an extensive look at LightProof's functionality on developer László Németh's blog. There he discusses the philosophy employed to hopefully make LightProof more useful and less annoying — a shortcoming that he said leads many users to disable grammar checking in Microsoft Office and other products. The gist is to not try and do too much, limit "false positives," and allow user-control over the options. The rules that the grammar checker uses are configurable, and each rule is linked to a detailed explanation for educational purposes.

Smaller UI improvements include a nicer page-break indicator and a revamped interface for creating and modifying headers and footers. Both are editable using an on-canvas drop-down menu, rather than having settings buried in the menus. You can now toggle the display of non-printing characters (such as paragraph breaks), which should help when hunting down white space problems or formatting issues. The word-count tool, which in previous releases needed to be manually re-executed to update its numbers, is now "modeless" and maintains a live count of the documents words as you work.

There are also lots of smaller improvements to layout issues (such as how to handle tab stops that extend past the outside margins of the page), and internationalization improvements (such as support for using Arabic letters or Persian words as the "numbers" in ordered lists). Last but not least, when Writer automatically generates a table of contents (TOC) for a document, it can now hyperlink the TOC entries to the correct place in the body of the text.

Calc

The Calc spreadsheet also picks up some new features and UI improvements. At the purely functional level, Calc got several new functions, including the trigonometric functions secant and cosecant (along with their hyperbolic analogs) and basic bitwise operations. The new version of Calc also allows for an unlimited number of user-defined rules for conditional formatting (e.g., changing the text style or background color of a cell based on particular criteria). There is still an upper limit on the number of individual sheets that a Calc spreadsheet document can contain, but 3.5.0 bumps that limit up to 10,000.

The interface changes include a multi-line input box for entering cell data. This is a change from the traditional one-line input bar most users are accustomed to, but it makes for an easier time entering formulas or long text strings in a spreadsheet. Graphs and charts should look better in 3.5.0, with the addition of several more point-marker styles (the dots rendered for data points in a scatterplot or line graph), and several fixes to the line-drawing code. Apparently, in previous releases, it was possible for a line that was intended to connect several data points to end up only touching the beginning and end points; the new B-spline code fixes that problem and results in smoother-looking lines all around.

Finally, an import bug-fix lands in the new release, in which Calc now gracefully handles the situation when a cell or formula's data comes from an external source (such as a database), but the external source is unreachable or unreadable. In previous releases, this resulted in errors that cascaded through the spreadsheet's calculations; instead, now the old value of the cell is used and an alert is triggered to tell the user that the external source is irretrievable.

Draw, Impress, and Math

The other components in the suite have shorter new-feature lists in 3.5.0, but some of the enhancements are significant. For example, the Draw vector editor gains the ability to import Microsoft Visio files, and picks up several new styles of line-ending "arrowheads" — including several that are designed to work with Unified Modeling Language (UML) diagrams. Draw can also embed several types of swatch palettes, which includes color palettes and the less-frequently seen gradient or fill palettes. That allows a single change to one of the palettes to alter colors, gradients, and fills throughout a large document.

The LibreOffice version of the Impress presentation application includes the "presenter console" feature that had been an optional extension in OpenOffice.org — though one that most Linux distributions had included for quite some time. This feature lets a user drive a connected projector in the usual manner while keeping his or her accompanying notes visible (or the next slide to be shown) on the screen of the laptop or PC. Unfortunately Impress can get confused sometimes about the displays, which can transpose the screen that is showing the slides and the one showing the notes; 3.5.0 adds a handy screen-swapper button so users can correct the problem at presentation time with a single click. Impress also makes launching the new-presentation-wizard at start-up time optional, and picks up improvements to importing vector shapes and "Smart Art" graphics from Microsoft PowerPoint files.

Math, the LibreOffice formula editor, gains the ability to both import and export expressions from Microsoft's Office Open XML (a.k.a. DOCX) format. It also acquires a few new symbols, such as the impressively-named "negated existential quantifier" (better known as the "does not exist" symbol ∄), and a set of symbols used in game theory.

Still to come

In addition to all of the work that went into the LibreOffice 3.5.0 code itself, the project has been busy on several other fronts in the past few months. On February 1, it legally incorporated its governing organization The Document Foundation as a community-driven nonprofit foundation based in Germany. Deputy Chairman of the Board Thorsten Behrens described the move as a legal affirmation of the project's community spirit, "independent from any single vendor.”

The project also launched a StackOverflow-style community-support site named Ask LibreOffice, which offers a vote-driven way for users to find answers to their questions. The site runs on the open source Askbot web application.

Development continued in new directions, too. A GTK+3 port is underway for Linux, which is an important milestone in its own right, but also clears the path for a web-based LibreOffice interface somewhere down the line, thanks to GTK+3's HTML5 back-end. Work is also underway to port LibreOffice to Android and Apple's iOS. Both mobile OSes are increasingly popular in the workplace on tablets, so there is a case to be made that these new ports are as important (if not more so) as the traditional desktop targets.

Neither the HTML5 nor mobile OS ports of LibreOffice made it to stable status for the release of 3.5.0, however. Meeks told FOSDEM that he is hopeful an online version of LibreOffice will be stable by the end of 2012, and that at least an ODF document reader will be available soon for Android, if not a complete LibreOffice suite. For a project as hefty as LibreOffice, that is a brisk pace, but the past year has shown The Document Foundation and its development community capable of working at a rapid clip — as 3.5.0 demonstrates.

Comments (1 posted)

Handset cohabitation: Ubuntu for Android

Ubuntu for Android is intended for handsets that can be docked and will, thus, have a keyboard, mouse, and display available. In that setting, it will provide the usual, Unity-based Ubuntu experience on that external display; the Ubuntu system essentially runs inside its own container on top of the Android kernel. The interface on the handset itself, meanwhile, remains pure Android. So Ubuntu for Android can be thought of as providing two distinct personalities for the device. There is some data sharing between the two - the contacts database, for example - but they remain mostly separate from each other. Rather than create a single integrated interface to the handset, Canonical has made something closer to a dual-boot system - except that the two can run simultaneously on their respective displays.

According to Canonical, the split system is the best solution:

Even a well-equipped phone does not have vast amounts of storage by contemporary standards. But even with more storage, it seems likely that users of Ubuntu for Android would want to have their files available outside the handset as well. So it is not surprising that this system is cloud-heavy. So there is no LibreOffice by default; instead, the system expects to use the Google Docs service. It does provide Thunderbird, though one might imagine that its storage-intensive indexing has been disabled by default. For good measure, Ubuntu TV has also been built into the system.

The hardware requirements (found on the features page) rule out a lot of devices, but are certainly not out of line for a current high-end device. Ubuntu for Android wants a dual-core CPU (clocked at 1GHz or higher), video acceleration and the ability to produce HDMI output from a secondary frame buffer device, and 512MB of RAM. The need for a dock for the phone to provide HDMI and USB ports is implied; few devices have the requisite connectors without a dock. As Canonical points out, the hardware requirements are easily satisfied by devices that are in development now.

So Ubuntu for Android seems like a useful and feasible development. The unfortunate part is that it is not available for users or developers to play with. Canonical is clearly hoping to sell this offering to device manufacturers and carriers; as this page makes clear, shipping it will involve per-unit royalties. Canonical clearly believes that vendors may find those royalties worthwhile, though, as a way to sell more high-end devices:

Canonical also pitches the idea that a bundled Ubuntu desktop will drive demand for fast broadband offerings (LTE, for example) from the carriers. And they claim that it could be especially attractive in parts of the developing world where high-end handsets are being sold to customers who have never owned a computer before. Such people, Canonical says, have "no legacy attachment to the desktop" and will find a combined offering compelling.

This reasoning may make some sense; it is possible that hybrid, handheld Linux-based systems will bring about the year of the Linux desktop after all. But there are a couple of concerns worthy of note. One is that users may quickly tire of having two different interfaces to the same system, leaving Ubuntu for Android vulnerable to a competing system with a more integrated experience. One can imagine, after all, that, if this idea goes anywhere at all, there will be Windows- and Mac OS-based variants available in short order - and, perhaps, other Linux-based implementations as well. Some of these systems may look like less of a hybrid and, as a result, be more successful.

The other concern is that Canonical appears to be taking a step toward proprietary systems. If there are plans to offer this functionality directly to users, or to enable it to be bundled with a distribution like CyanogenMod, Canonical has not disclosed them yet. Instead, we have a system that, by all appearances, will only be available in binary form from manufacturers or carriers. Source for GPL-licensed components will naturally be available, but it is far from clear that Ubuntu for Android will be all free software; vendors like Citrix and Adobe feature prominently on the product's page. It is also not clear that device owners will be able to modify the distribution to their own liking and run the result on their devices. A handset or tablet that can run a full Ubuntu system has some appeal; one running a locked-down Ubuntu system would be rather less exciting.

Ubuntu for Android is clearly an important step in the evolution of the "desktop" away from traditional personal computer systems. It has a lot of potential as a practical replacement for bulkier systems. But, to be commercially successful, Canonical will have to convince a lot of people that the Unity-based desktop is what customers want. And to be successful as free software, it will have to result in free systems under the control of their owners. It will be a sad day if the Ubuntu community of the future is focused on the creation and propagation of tools to jailbreak their Ubuntu systems.

Comments (91 posted)

Page editor: Jonathan Corbet

Security

Capsicum: practical capabilities for UNIX

The Capsicum capabilities framework has been around for a couple of years now, and support for it was added to the recent FreeBSD 9.0 release. Capsicum takes a very different approach from other capabilities systems (like Linux capabilities or POSIX capabilities), and is geared toward sandboxing applications to limit the damage that can be caused by buggy or misbehaving programs. While the FreeBSD support is "experimental", it is available for researchers and others to try out.

Capsicum came out of a collaboration between the University of Cambridge's Computer Laboratory and Google. That resulted in a prototype implementation for FreeBSD along with modification of several different programs to take advantage of Capsicum. One of the main applications of interest is the Chromium web browser, but several FreeBSD utilities (tcpdump, dhclient, and gzip) were also converted, as described in the Capsicum paper [PDF].

The idea behind Capsicum is to extend the standard Unix APIs by adding ways that applications can "self-compartmentalize". Essentially, applications can choose to restrict themselves to a sandbox that will disallow many "dangerous" operations, while still allowing them to get their job done via the capabilities they allow for themselves or those that are passed in using special file descriptors (which are also, perhaps unfortunately, called capabilities). It is, in some ways, conceptually similar to programs that drop their privileges using the setuid() call but, instead of being restricted to what a particular user is allowed to do (which is often far more than the application needs), Capsicum allows much finer-grained control over what restrictions are in place.

The starting point for a Capsicum-enabled process is the new cap_enter() system call. This is a one-way gate that puts a process and any subsequent children into "capability mode". It turns off "ambient authority", which is a term for the normal Unix process model where a process has all of the permissions of the UID it is running as. Capability mode restricts access to any of the global namespaces, like the filesystem namespace, PID namespace, network namespace, and others. Any system calls that operate on these global namespaces are either disallowed entirely, or their arguments are constrained.

For example, the sysctl() call is constrained to only allow around 30 (of a possible 3000) of the different system parameters to be examined via that call. The shared memory creation call, shm_open(), is only allowed to create anonymous memory objects, while the openat() family is restricted to allow access to files at or below the directory file descriptor passed in (by essentially disallowing "/" or ".." at the start of the path). There are some other miscellaneous restrictions that come with capability mode including disallowing the loading of kernel modules or the execution of setuid and setgid binaries.

Capsicum wraps normal file descriptors with additional capability information that restricts what can be done with the file. If a capability file descriptor has the CAP_READ capability, that's all that can be done to it, unlike a file descriptor for a file that is opened read-only which can still be used to make metadata changes (via fchmod() for example). In order to change positions in the file, the CAP_SEEK capability is required. A capability file descriptor can also wrap a directory file descriptor, which allows the capability set to be applied to all members of that directory. That would allow Apache to set up workers that only have access to a certain subset of the web directory hierarchy, or for a sandboxed application to access a library path, for example.

The capability file descriptors can be already open at the time that cap_enter() is called (and wrapped by a set of capabilities specified in an earlier cap_new() call) or passed to the process using Unix sockets. That means that a fairly simple program can decrease its ability to cause harm by setting up the file descriptors it needs and then calling cap_enter() before performing more "dangerous" operations. The tcpdump example given in the paper is instructive, as it simply enters capability mode after setting up the packet filter (which is a privileged operation), but before entering the processing loop. That way, errors in the packet decoding code are very limited in the kind of damage they can cause.

The simple two-line change to tcpdump() did expose a few problems, however. For example the glibc DNS resolver code requires access to the filesystem (/etc/resolv.conf) and to the network namespace (to talk to the DNS server), which led to reduced functionality. Switching tcpdump to use a lightweight local resolver restored that feature.

In addition to the "raw" Capsicum interface using cap_enter(), the framework provides a libcapsicum that can be used to more thoroughly isolate the sandboxed processes without each application having to do its own start-up management of a sandboxed process. It handles closing all undelegated file descriptors (those that are not meant for the sandbox), forking the new sandboxed process, flushing the address space using fexecve(), and setting up a Unix socket that can be used for communication between the privileged and unprivileged processes. None of the examples in the paper use libcapsicum as it generally requires major changes to the application in order to be used, so it may be more suitable for new development.

The examples do show that substantial improvements in the security of programs can be had with minimal code changes, though. Roughly 100 new lines of code were all that was required to use Capsicum in Chromium on FreeBSD, largely because the browser was written with privilege separation in mind. Chromium already uses various techniques, depending on the OS, to separate the rendering process from other renderers and the rest of the browser. That made it fairly straightforward to adapt Chromium and the paper says that switching to a libcapsicum-based implementation should not be significantly harder.

Capsicum is an interesting idea that bears watching as it rolls out in FreeBSD. The 9.0 release only contains the kernel changes required for Capsicum but doesn't ship any applications that use the facility. 9.1 is slated to have some of those, presumably starting with Chromium. Beyond this brief introduction, those interested should take a look at the paper, this article [PDF] from ;login: magazine, as well as the documentation page.

Comments (5 posted)

Brief items

Security quotes of the week

Comments (1 posted)

RSA keys not as random as they should be (The H)

Comments (16 posted)

Weekend Project: Get Started with Tahoe-LAFS Storage Grids (Linux.com)

Comments (1 posted)

Mozilla's message to certificate authorities

Comments (33 posted)

New vulnerabilities

busybox: code execution

Comments (none posted)

chromium: multiple vulnerabilities

Comments (none posted)

conga: cross-site scripting

Comments (none posted)

drupal7-field_permissions: missing permissions

Comments (none posted)

flash_plugin: multiple vulnerabilities

Comments (none posted)

horde3: cross-site scripting

Comments (none posted)

horde3-dimp: cross-site scripting

Comments (none posted)

ibutils: code execution

Comments (none posted)

initscripts: network traffic sniffing

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

jetty5: denial of service

Comments (none posted)

libpng: code execution

Comments (none posted)

libvorbis: code execution

Comments (none posted)

libxml2: denial of service

Comments (none posted)

mozilla: use after free

Comments (none posted)

mumble: information disclosure

Comments (none posted)

rocksndiamonds: arbitrary file overwrite

Comments (none posted)

wicd: information disclosure

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

Stable updates: the 3.0.22 and 3.2.7 stable updates were released on February 20; they contain the usual list of important fixes.

Comments (none posted)

Quote of the week

if (ret) {
        one_line_statement();
}

Comments (12 posted)

A sys_poll() ABI tweak

Most of the time, things just work. The int and long types tend to be the same on most architectures, and, in cases where they are different, glibc sign-extends the timeout value appropriately. Things go wrong, though, when a 32-bit process is running on an x86-64 system. In that case, the 32-bit sys_poll() function just passes the timeout value directly to the native kernel version, without sign extension. So if the timeout value is negative (an indication that poll() should wait forever if need be), the kernel will eventually see a large, positive timeout instead.

There are various ways this problem could be fixed. What Linus has chosen to do, though, is to just change the type of the timeout parameter to int inside the kernel. Since the timeout is now a 32-bit quantity on all systems, that particular source of confusion is removed. There is a small risk to this approach, though: it is possible that some program somewhere was actually making use of 64-bit timeouts. Doing so would require replacing or bypassing glibc (because its sign extension makes 64-bit timeouts unusable), so it's unlikely that anybody has bothered, but one never knows. If this change were to break a real application, it would have to be reverted in favor of a more complicated solution.

Linus's patch was merged for 3.3-rc5, so anybody who objects has a few weeks to make their concerns known.

Comments (5 posted)

dma-buf and binary-only modules

It will be unsurprising to most readers that the response to Robert's complaint was not 100% sympathetic. After a while, the discussion died down without any real resolution. Recently, though, Rob Clark has reported on a discussion held at the Embedded Linux Conference:

Since then, there has been no discussion at all; there has also been no move to change the symbol exports in the mainline kernel. But the shift in tone suggests that positions may be softening, and that the buffer-sharing API may eventually be made available to proprietary modules.

Comments (none posted)

Oracle offering DTrace for Linux

Comments (39 posted)

Kernel development news

Short sleeps suffering from slack

The C-library sleep() function is defined to put the calling process to sleep for at least the number of seconds specified. One might think that calling sleep() with an argument of zero seconds would make relatively little sense; why put a process to sleep for no time? It turns out, though, that some developers put such calls in as a way to relinquish the CPU for a short period of time. The idea is to be nice and allow other processes to run briefly before continuing execution. Applications that perform polling or are otherwise prone to consuming too much CPU are often "fixed" with a zero-second sleep.

Once upon a time in Linux, sleep(0) would always put the calling process to sleep for at least one clock tick. When high-resolution timers were added to the kernel, the behavior changed: if a process asked to sleep on an already-expired timer (which is the case for a zero-second sleep), the call simply returned directly back to the calling process. Then came the addition of timer slack, which can extend sleep periods to force multiple processes to wake at the same time. This behavior will cause timers to run a little longer than requested, but the result is fewer processor wakeups and, thus, a savings of power. In the case of a zero-second sleep, the addition of timer slack turns an expired timer into one that is not expired, so the calling process will, once again, be put to sleep.

The default timer slack, at 50µs, is unlikely to cause visible changes to the behavior of most applications. But it seems that, on some systems, the timer slack value is set quite high - on the order of seconds - to get the best power behavior possible. That can extend the length of a zero-second sleep accordingly, leading to misbehaving applications.

Matthew Garrett, working under the notion that breaking applications is bad, submitted a patch making a special-case for zero-second sleeps. The idea is simple: if the requested sleep time is zero, timer slack will not be added and the process will not be delayed indefinitely. The problem with this approach is that the process will still not get the desired result: rather than yielding the processor, it will have simply performed a useless system call and gone right back to whatever it was doing before. Without timer slack, a request to sleep on an expired timer will return directly to user space without going through the scheduler at all.

An alternative would be to transform sleep(0) into a call to sched_yield(). But that idea is not hugely popular with the scheduler developers, who think that calls to sched_yield() are almost always a bad idea. It is better, they say, to fix the applications to stop polling or doing whatever else it is that they do that causes developers to think that explicitly yielding the CPU is the right thing to do.

According to Matthew, the number of affected applications is not tiny:

Normal practice in kernel development would be to try to avoid breaking those applications if possible. Even in cases where applications are relying on undefined and undocumented behavior - certainly the case here - it is better if a kernel upgrade doesn't turn working code into broken code. Some participants have suggested that the same approach should be taken in this case.

The situation with sleep(0) is a little different from others, though. Application developers cannot claim a long history of working behavior in this case, since the kernel's response to a zero-second sleep has already changed a few times over the course of the last decade. And, according to Thomas Gleixner, it is hard to know when the special case applies or what should be done:

Thomas worries that there may be calls for special cases for similar calls - single-nanosecond calls to nanosleep(), for example - and that the result will be an accumulation of cruft in the core timer code. So, rather than try to define these cases and maintain the result indefinitely, he thinks it is better just to let the affected code break in cases where the timer slack has been set to a large value. And that is where the discussion faded away, suggesting that nothing will be done in the kernel to reduce the effect of timer slack on zero-second sleeps.

Comments (25 posted)

The Linaro Connect scheduler minisummit

I had the privilege of acting as moderator/secretary for the recent Scheduler Mini-Summit at Linaro Connect, which was attended by Peter Zijlstra (Red Hat), Paul Turner (Google), Suresh Siddha (Intel), Venki Pallipadi (Google), Robin Randhawa (ARM), Rob Lee (Freescale assigned to Linaro), Vincent Guittot (ST-Ericsson assigned to Linaro), Kevin Hilman (TI), Mike Turquette (TI assigned to Linaro), Peter De Schrijver (Nvidia), Paul Brett (Intel), Steve Muckle (Qualcomm), Sven-Thorsten Dietrich (Huawei), and was ably organized by Amit Kucheria (Linaro). Rough notes from the session can be found here.

The main goals of the mini-summit were as follows:

1. Take first step towards planning any Linux-kernel scheduler changes that might be needed for ARM's upcoming big.LITTLE [PDF] systems to work well (see also Nicolas Pitre's LWN article).

2. Create a power-aware infrastructure for scheduling and related Linux kernel subsystems. For example, integrate dyntick-idle, cpufreq, cpuidle, sched_mc, timers, thermal framework, pm_qos, and the scheduler.

3. Provide a usable mechanism that reliably allows all work (present and future) to be moved off of a CPU so that said CPU can be powered off and back on under user-application control. CPU hotplug is used for this today, but has some serious side effects, so it would be good to either fix CPU hotplug or come up with a better mechanism—or, in the best Linux-kernel tradition, both. Such a mechanism might also be useful to the real-time people, who also need to clear all non-real-time activity from a given CPU.

How well did we meet these goals? Read on and decide yourself! To that end, the remainder of this article is organized as follows:

1. Overview of ARM big.LITTLE Systems
2. Major Issues Considered
3. Future Work and Prospects
4. Conclusions

Following this is the inevitable Answers to Quick Quizzes.

Overview of ARM big.LITTLE Systems

ARM's big.LITTLE systems combine the Cortex-A7 and Cortex-A15 processors. Both processors are implementations of the ARMv7 architecture and they execute the same code. ARM stated the little Cortex-A7 design was focused on energy efficiency at the expense of performance. The bigger Cortex-A15 design was, instead, focused on performance at some cost to energy efficiency. In practice this means the little core will be somewhat quicker and a lot more power efficient than today's Cortex-A8: a multi-core configuration of these little cores could run today's smartphones. The big core will significantly outperform Cortex-A9 within a similar power budget.

For some implementations, thermal limitations would require that the Cortex-A15 CPUs be used only for short bursts at maximum frequency, as was discussed at length at the summit. However, I have since learned that many other implementations are expected to be fully capable of running the Cortex-A15 CPUs at maximum frequency indefinitely.

The switch between the Cortex-A7 and Cortex-A15 CPUs is implemented in firmware, but Grant Likely, Nicolas Pitre, and Dave Martin are moving this functionality into the Linux kernel.

In many big.LITTLE designs, it is also possible to run both the Cortex-A7 and Cortex-A15 CPUs concurrently in an shared-memory configuration. However, this means that the Linux kernel sees the big.LITTLE architecture, which in turn raises the issues discussed in the next section.

Major Issues Considered

Those of you who know the personalities in attendance will not be surprised to hear that the discussions were both spirited and wide-ranging. However, most of the discussion centered around the following four major issues:

1. Benchmarks and Synthetic Workloads
2. Parallel Hardware/Software Development
3. What Do You Do With a LITTLE CPU?
4. CPU Hotplug: Kill It or Cure It?

Each of these issues is covered in one of the following sections:

Benchmarks and Synthetic Workloads

The biggest and most pressing issue facing SMP-style big.LITTLE systems is the lack of packaged Linux-kernel-developer-friendly benchmarks and synthetic workloads. C programs and sh, perl, and python scripts can be friendly to Linux-kernel developers, while benchmarks requiring (for example) an Android SDK or a specific device will likely be actively ignored.

It is critically important for benchmarks to provide a useful “figure of merit”, which should encompass both user experience and estimated power consumption. For example, a synthetic workload that models a user browsing the web on a smartphone might have a smaller-is-better estimate of average power consumption, but also have the constraint that the system respond to emulated browser actions within (say) 500ms. If the response time is within the 500ms constraint, then the figure of merit is the estimated average power consumption, but if that constraint is exceeded, the figure of merit is a very large number. The exact computation of the figure of merit will vary from benchmark to benchmark.

Currently, some rough and ready workloads are in use. For example, Vincent Guittot used cyclic test in his work. While this did get the job done for Vincent, something more adapted to embedded/mobile workloads instead of real-time computing would be quite welcome. Zach Pfeffer of Linaro will be doing some workload creation in his group, however, given the wide variety of mobile and embedded workloads, additional contributions would also be welcome.

Finally, the scheduler maintains a great number of statistics and tracepoints. A “schedtop”-style tool that provides a mobile/embedded view of this information would be very valuable.

Parallel Hardware/Software Development

Even if you don't know exactly when a given piece of hardware will be available, it is a good bet that it will become available too late to get the needed software running on it. It is therefore critically important to have some way to develop the needed software before the hardware is available. Thankfully, there are a number of ways to test big.LITTLE scheduler features before big.LITTLE hardware becomes available.

One crude but portable method is to create a SCHED_FIFO thread on each LITTLE-designated CPU, and to have this thread spin, burning CPU, for (say) one millisecond out of every two milliseconds. This approach perturbs the scheduler's preemption points, particularly the wake-up preemptions. Nevertheless, this approach is likely to be quite useful.

A less portable but more accurate approach is to constrain the clock frequency of the CPUs so that the big-designated CPUs have a lower bound on their frequency and the LITTLE-designated CPUs have an upper bound on their frequency. The way to do this is via the sysfs files in the /sys/devices/system/cpu/cpu*/cpufreq directories, the most pertinent of which are described below.

  cd /sys/devices/system/cpu/cpu1/cpufreq
  sudo echo 800000 > scaling_max_freq

    2534000 2533000 1600000 800000

For more information, see Documentation/cpu-freq in the Linux kernel source directory.

There was also some discussion of ways that the linsched user-mode scheduler simulator might help with prototyping.

Finally, it is possible to use T-states on Intel platforms to emulate a big.LITTLE system. According to Paul Brett:

What Do You Do With a LITTLE CPU?

If you have both big and LITTLE CPUs, how do you decide what tasks will be banished to the slower LITTLE CPUs? Similarly, if your workload is currently running all on LITTLE CPUs, how do you decide when to take the step of starting up one of the the power-hungry big CPUs?

Right now for SMP-configured big.LITTLE systems, “you” is the application developer, who can use facilities such as CPU hotplug, affinity, cpusets, sched_mc, and so on to manually direct the available work to the desired subsets of the CPUs. These facilities constrain the scheduler in order to ensure that nothing runs on CPUs that are to be powered down.

Decisions on what CPUs to use should include a number of considerations. First, if a LITTLE CPU is able to provide sufficient performance, it provides better energy efficiency, at least in cases where race to idle is inappropriate. Second, because mobile platforms have no fans and are sometimes sealed, some devices might not be able to run all the big CPUs at maximum clock rate for very long without overheating. Of course, such devices might also need to limit the heat produced by analog electronics and GPUs as well (see Carroll's and Heiser's 2010 USENIX paper [PDF] and presentation [PDF] for a power-consumption analysis of a ca. 2008 smartphone). Third, some workloads can adapt themselves to lower performance. For example, some media applications can reduce performance requirements by dropping frames and reducing resolution. Fourth, there is more to performance than CPU clock speed: For example, it is possible that a workload with high cache-miss rates can run just as fast on a LITTLE CPU as it can on a big CPU. Finally, many workloads will have preferred ways of using the CPUs, for example, some mobile workloads might use the LITTLE CPUs most of the time, but bring the big CPUs online for short bursts of intense processing.

Keeping track of all this can be challenging, which is one big reason for thinking in terms of automated assistance from the scheduler. Some of the proposed work towards this end is listed in the Future Work and Prospects section. But first, let's take a closer look at CPU hotplug and its potential replacements.

CPU Hotplug: Kill It or Cure It?

Although CPU hotplug has a checkered reputation in many circles, it is what almost all current multicore devices actually use to evacuate work from a given CPU. This is a bit surprising given that CPU hotplug was intended for infrequently removing failing CPUs, not for quickly bringing perfectly good CPUs into and out of service. It is therefore well worth asking what CPU hotplug is providing that users cannot get from other mechanisms:

• Migrating timers off of a given CPU. This can likely be fixed, but a synchronous fix that prevents any further timers from being set may be more challenging.

• Shutting off a CPU with a single simple action. This can likely be fixed, but requires coordinating interrupts, the scheduler, timers, kthreads, and so on.

• Preventing all possible wakeup events from causing that CPU to power back on until the user explicitly permits it to power back on. (Some platforms may have wakeup events that cannot be shut off.)

• Synchronous action, so that userspace can treat it atomically.

• Coordinating user applications based on hotplug events. (However, there is no known embedded or mobile use of this feature, so if you need it, please let us know. Otherwise it will likely go away.)

These CPU-hotplug features are valuable outside of the mobile/embedded space, for example, some real-time applications will take a CPU offline and then immediately bring it back online to make it fully available for the application—in particular, to clear timers off of the CPU. Furthermore, people really do make use of CPU hotplug to offline failing CPUs.

But this brings up another question. Given that CPU hotplug does all these useful things, what is not to like? First, CPU-hotplug operations can take several seconds, as shown here. An ideal power-management mechanism would have latencies in the 5ms range. It might be possible to make CPU hotplug run much faster. Second, CPU-hotplug offline operations use the stop_machine() facility, which interrupts each and every CPU for an extended time period. This sort of behavior is not acceptable when certain types of real-time or high-performance-computing (HPC) applications are running. It might be possible to wean CPU hotplug from stop_machine().

Third, a given CPU's workqueues can contain large numbers of pending items of work, and migrating all of these can be quite time consuming, as can re-initializing all the workqueue kernel threads when a given CPU comes online. Other CPU-hotplug notifiers have similar problems, which can hopefully be addressed by coming up with a good low-overhead way to “park” and “unpark” kernel threads that are associated with an offline CPU.

Such a parking mechanism faces the following challenges:

• Many per-CPU kernel threads are (quite naturally) coded with the assumption that they will always run on the corresponding CPU.

• If a kthread that has an affinity to a given CPU is awakened while that CPU is offline, the scheduler prints an error message and removes the affinity, so that the kthread will now be able to run on any CPU.

• Wakeups can be delayed so that they do not arrive at the kthread until after the corresponding CPU has gone offline.

• All kernel threads parked for a given offline CPU must sleep interruptibly, because otherwise the kernel will emit soft-lockup messages.

• When a given CPU goes offline, any work pending for that CPU must either be completed immediately (thus delaying the offline operation), migrated to some other CPU (thus increasing complexity), or deferred until the CPU comes back online (which might be never).

Finally, CPU-hotplug operations can destroy cpuset configuration, so that cpusets need to be repaired when CPUs are brought back online. This topic is currently the subject of spirited discussions.

Perhaps these CPU-hotplug shortcomings can be repaired. But suppose that they cannot. What should be done instead in order to evacuate all work from a given CPU?

Tasks can be moved off of a given CPU by use of explicit per-task affinity, cgroups, or cpusets, although interactions with other uses of these mechanisms need more thought. In addition, interactions among all of these mechanisms can have unexpected results because of a strong desire that the scheduler generally consume less CPU than the workload being scheduled.

However, interrupts can still happen on a given CPU even after all tasks have been evacuated. Interrupts must be redirected separately using the /proc/irq directory. This directory in turn contains one directory for each IRQ, and each IRQ directory contains a smp_affinity file to which you can write a hexadecimal mask to restrict delivery of the corresponding interrupts. You can then examine the /proc/interrupts file to verify that interrupts really are no longer being delivered to the CPUs in question. See Documentation/IRQ-affinity.txt in the kernel sources for more information. One caution: that document notes that some irq controllers do not support affinity, and for such controllers it is not possible to direct irq delivery away from a given CPU.

Finally, evacuating tasks and interrupts from a given CPU can still leave timers running on that CPU. As noted earlier, there is currently no mechanism other than CPU hotplug to migrate timers off of a given CPU, but it should be possible to create such a mechanism. An asynchronous mechanism (which would allow each timer one final ride on the outgoing CPU) is straightforward. A synchronous mechanism would be more complex, but should be doable.

So, what should be done? In the near term, the only sane approach is to attack on both fronts: (1) attempt to cure CPU hotplug of its worst ills (especially given that it will likely continue to be needed for removing failing CPUs), and (2) attempt to improve the alternative mechanisms so that they can do more of the work that can currently only be done by CPU hotplug—hopefully avoiding at least some of the complexity currently inherent to CPU hotplug.

Future Work and Prospects

In the short term, the following actions need to be taken:

• Create an email list for the attendees and other interested parties. This is now available here, courtesy of Amit Kucheria and Loic Minier.

• Document best practices for using existing Linux kernel facilities (including CPU hotplug, cgroups, cpusets, affinity, and so on) to manage big.LITTLE systems in an SMP configuration. This documentation should include measurements of latencies (keeping in mind the 5ms goal for evacuating work from a CPU and for restarting it) and power consumption. Vincent Guittot's presentation and git tree are a good start in this direction.

• Create software to emulate big.LITTLE systems on current hardware, for example, using one or more of the approaches describe in the Parallel Hardware/Software Development section.

• Produce Linux-kernel-developer-friendly synthetic workloads and benchmarks for mobile applications and use cases, as discussed in the Benchmarks and Synthetic Workloads section. There will be some Linaro work in this direction, but additional workloads and benchmarks are welcome from any and all.

• Experiment with improving cpusets and cgroups as discussed in the CPU Hotplug: Kill It or Cure It? section.

• Experiment with curing CPU hotplug, also discussed in the CPU Hotplug: Kill It or Cure It? section.

• Accumulate a list of the attributes that system-on-a-chip (SoC) vendors believe to be important to scheduling and managing big.LITTLE systems. An initial list was accumulated during the scheduler summit:
  • Power-domain and clock-domain constraints. For example, many ARM SoCs require that all CPUs in a cluster run at the same clock rate.

  • Thermal tradeoffs. For example, some SoCs might impose a tradeoff between the number of CPUs running at a given time and the frequency at which they are running if they are to avoid thermal throttling.

  • Thermal feedback, e.g., temperature sensors.

  • Process type, where the amount of leakage current can affect the optimal strategies for power-efficient operation.

  • Relative benefit of reducing frequency of several CPUs as opposed to consolidating workload on a small number of CPUs.

  • Instruction-per-clock (IPC) measurements and correlation between clock rate and useful forward progress.

• Remove sched_mc.

• Port Frederic Weisbecker's OS-jitter-reduction patchset to ARM. Geoff Levand of Huawei is leading up an effort along these lines.

• Contact gaming companies (e.g., Epic) to see if their 3D gaming engines (which run on both iPhone and Android) can make good use of big.LITTLE, even in the presence of thermal throttling.

• Investigate alternative scheduler disciplines. For example, the prototype SCHED_EDF patchset would allow tasks to specify deadlines, which would allow the scheduler to better decide between race-to-idle and run-at-low-frequency. Other related scheduler disciplines such as EVDF might be useful—there may be other real-time technologies that can be commandeered to energy-efficiency purposes.

  If SCHED_EDF looks useful to mobile/embedded, someone needs to forward-port the patch and fix a number of issues in it. This is not a trivial project. (Paul Turner is looking into bringing some Google resources to bear, and Juri Lelli has been doing some recent deadline-scheduler work.)

  One common mobile/embedded requirement is to consolidate the workload down to the minimum number of CPUs that can support acceptable user experience, then spread the load across that minimal set of CPUs.

• Investigate modal scheduling. Paul Turner gave the following list of modes as an example:
  • Low load of interactive, low-utilization tasks might favor race to idle.

  • Moderate load of periodic media-feeding tasks might lower frequency to the smallest value that allows the task to keep up with its hardware.

  • High load of CPU-bound tasks in the absence of thermal limitations might increase frequency.

  Some hysteresis will be required. It is usually OK to delay the decisions a bit, especially given that ARM provides relatively fast transitions between power states. Paul Turner posted a first step in this direction, with a patch series that improves the scheduler's ability to better estimate the effect of migration of each CPU's load. A more up-to-date series is maintained here.

Conclusions

The scheduler mini-summit at Linaro Connect was quite productive, with work already in progress to implement some of the recommendations. For example, some code and patches are in flight to reduce RCU's dependence on stop_machine(), which is a first step towards weaning CPU hotplug from stop_machine(). For another example, Srivatsa Bhat is doing some good work on curing CPU hotplug of some of its ills.

So how did we do against the goals? Let's check:

1. Take first step towards planning any Linux-kernel scheduler changes that might be needed for ARM's upcoming big.LITTLE systems work well.

   The most important actions toward this goal are the emulation of big.LITTLE systems, the mobile/embedded synthetic benchmarks/workloads, and the list of SoC attributes. This information will help work out which of the longer-term actions are most important.

2. Create a power-aware infrastructure for scheduling and related Linux kernel subsystems. For example, integrate dyntick-idle, cpufreq, cpuidle, sched_mc, timers, thermal framework, pm_qos, and the scheduler.

   The mobile/embedded synthetic benchmarks/workloads is the most important first step in this direction, as is the list of SoC attributes. The removal of sched_mc is a first implementation step, on the theory that one must tear down before one can build.

3. Provide a usable mechanism that reliably allows all work (present and future) to be moved off of a CPU so that that CPU can be powered off and back on under user-application control. CPU hotplug is used for this today, but has some serious side effects, so it would be good to either fix CPU hotplug or come up with a better mechanism—or, in the best Linux-kernel tradition, both. Such a mechanism might also be useful to the real-time people, who also need to clear all non-real-time activity from a given CPU.

   This goal received the most discussion, and the medium-term actions for curing CPU hotplug on the one hand or improving the alternatives to CPU hotplug on the other.

Work on these three goals has only just begun, but with continued effort, we can make the Linux kernel work better for big.LITTLE in particular and for mobile/embedded workloads on asymmetric systems in general.

Acknowledgments

I am grateful to the scheduler mini-summit attendees for many useful and enlightening discussions, and especially to Amit Kucheria for organizing the mini-summit. We all owe thanks to Zach Pfeffer, Peter Zijlstra, Amit Kucheria, Robin Randhawa, Jason Parker, Rusty Russell, Vincent Guittot, and Dave Rusling for helping make this article human readable. I owe thanks to Dave Rusling and Jim Wasko for their support of this effort.

Answers to Quick Quizzes

Quick Quiz 1: But what if there is a different number of Cortex-A7s than of Cortex-A15s?

Answer: In that case, it is necessary to remove the excess CPUs from service, for example, using CPU hotplug, before carrying out the switch.

Back to Quick Quiz 1.

Quick Quiz 2: Why scale down? Isn't it always better to run full out in order to race to idle?

Answer: Although it often is best to race to idle, there are some important exceptions to this rule in certain mobile/embedded workloads. For one example, imagine a codec that required the CPU to occasionally do some work to provide the codec with data. Because CPU power consumption often rises as the square of the core clock frequency, you typically get the best battery life by running the CPU at the lowest frequency that gets the work done in time. As always, use the right tool for the job!

Back to Quick Quiz 2.

Quick Quiz 3: I typed the following command:

    sudo echo 800000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq

Despite the sudo, I got “Permission denied”. Why doesn't sudo give me sufficient permissions?

Answer: Although that command does give echo sufficient permissions, the actual redirection is carried out by the parent shell process, which evidently does not have sufficient permissions to open the file for writing. One way to work around this is sudo bash followed by the echo, or to do something like:

    sudo sh -c 'echo 800000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq'

Another approach is to use tee, for example:

    echo 800000 | sudo tee /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq

Yet another approach uses dd as follows:

    echo 800000 | sudo dd of=/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq

Back to Quick Quiz 3.

Quick Quiz 4: Why would anyone use an Intel system to test out an ARM capability?

Answer: The scheduler is core code, and for the most part does not care about which instruction-set architecture is running. The important thing is not the ISA, but rather the performance characteristics.

Back to Quick Quiz 4.

Quick Quiz 5: Why not just use SIGSTOP to park per-CPU kthreads?

Answer: It might well work, at least given appropriate adjustments. Please try it and let us know how it goes.

Back to Quick Quiz 5.

Quick Quiz 6: What other mechanisms could be used to park per-CPU kthreads?

Answer: Here are some possibilities:

• Kill the kthreads at CPU-offline time and recreate them at CPU-online time. This is used today, and is quite slow.

• Kill the kthreads at CPU-offline time and recreate them at CPU-online time, as above, but create a separate CLONE_ flag to prevent the parent from waiting until the child runs. This waiting behavior exists to work around an old bash bug, and is not needed for in-kernel kthreads.

• Kill the kthreads at CPU-offline time, but don't recreate them until they are actually needed, perhaps using a separate high-priority kthread to allow the creation to be initiated from environments where blocking is prohibited. This might work well in some situations, but does increase the state space significantly.

• Have the kthreads block while their CPU is offline. This approach faces some complications:
  • Wake-ups can be delayed, so that a delayed wakeup might arrive at a kthread after the corresponding CPU has gone offline.

  • If a task that has an affinity to a given CPU awakens while that CPU is offline, the scheduler prints a warning message and breaks affinity. This breaking of affinity can cause failures in kthreads written to assume that they only run on their CPU.

  • Sleeping uninterruptibly while a CPU is offline can result in spurious soft-lockup warnings.

• Use an explicit rendezvous to park each kthread, setting its affinity mask to cover all CPUs and informing it that it needs to remain quiescent. This operation would be reversed when the CPU comes back online. This works, but is often surprisingly difficult to get right, particularly on busy systems where wakeups can be delayed.

• Your idea here.

It is quite likely that different approaches will be required in different situations.

Back to Quick Quiz 6.

Comments (20 posted)

Subtle interactions in the embedded world - what bugs can teach us

I must admit that I love bugs. I also hate bugs which leads to a very conflicted relationship with the blighters, but for now I just want to focus on the positive - I love bugs. The bugs I am talking about are, of course, the incorrect behavior of software systems due to incorrect code and the reason that I love them is that they are so educational. There are a number of reasons for this, the significant one for the present being that they provide motivation and focus to read and study new code.

One of the freedoms that free and open source software provides is the freedom to study the source code. However having that freedom doesn't mean it's easy to use it. When faced with a large body of code such as the Linux kernel it can be hard to know where to start, or when to move on. There is no story-line, no curriculum, no plot to guide your study. This is where a bug comes in: it can provide a story line. This may be a particular sequence of events or a particular combination of features, or just a starting point to spiral out from. But by providing a clear focus and a clear finishing point, it makes study easier and more rewarding.

More than a phone

I have recently been spending some time trying to make the Linux kernel run well on the GTA04 "Phoenux" replacement motherboard for the Openmoko "Freerunner" and "Neo 1973" mobile phones. This is an attempt to revitalize the Openmoko effort to provide an open (or at least "as open as possible") platform for a mobile computing device. It fits in the same case, uses the same display and speakers, but provides more memory, faster processor, and faster networking plus a few other extras. During this (ongoing) effort I have discovered a number of bugs and missing features, most of which only came with smaller learning experiences. One, though, stands out for leading me on a somewhat longer path of discovery than most, and so I would like to share that path and those discoveries so that others might benefit. I should note first that the path is still open. I have not actually fixed the bug, so others could follow the path themselves. Anyone keen to do that should stop after the next paragraph, get a GTA04, and start hunting themselves. Others can read on.

The symptom of the bug is easy to describe. Like most computer systems the GTA04 has an RTC (real time clock) which has an alarm function. When the time in the clock matches the time in the alarm, it generates an interrupt. The problem is that, though this interrupt works perfectly while the system is awake, it does not reliably wake the device when it is asleep (i.e. in suspend-to-RAM state). Other interrupts configured much the same way (for example data received on the serial console) wake the system with 100% reliability. However the RTC alarm almost always fails. Understanding this bug leads on a path through the interrupt management code, the suspend management code, and even the USB code. But it must start with a brief description of the hardware.

In the GTA04 there are two particular integrated circuits (ICs) that are relevant. One is the OMAP3 SoC (system-on-a-chip) from TI. The other is the PMIC (power management IC) which is referred to as "twl4030" in the various drivers - various because, like the OMAP3, it is a multi-function chip combining diverse functions such as battery charging, keyboard control, and the USB electrical interface. There are a number of connections between these chips but only two are relevant for now. These are a single interrupt line and an I2C bus.

The interrupt line signals the interrupt controller (INTC) in the OMAP3 that something has happened. The I2C (inter-integrated-circuit) bus is a simple 2-wire bus that allows the CPU to read and write registers in the PMIC. In particular it can be used to find out which of the several functions generated the interrupt, and why. It can also be used to clear the source of the interrupt.

Having one interrupt line that indicates any of a number of possible events like this is not uncommon. For example the UARTs (Universal Asynchronous Receiver Transmitter - a serial port) in the OMAP3 each have a single interrupt, but it can be generated by a character arriving, a character transmit having completed, or an error condition. As with most such devices there is a register that can be inspected which has one bit for each possible interrupt source.

When all the interrupt sources are related to a single device and handled by a single driver this is all quite easy to manage. However when the different interrupt sources are related to logically separate devices, having to manage all these behind a single interrupt could get clumsy.

So many interruptions

To address this situation, Linux has an abstraction called an "irq_chip". An irq_chip represents a set of interrupt sources each of which is assigned an interrupt number (as listed in /proc/interrupts). It provides functions to enable or disable each interrupt, to allow the interrupt to wake the device from suspend, to set the trigger type (edge or level), and various other functions. It also arranges that the interrupt handler for each interrupt will be called when appropriate.

irq_chips with sources

Using this, the driver for the INTC in the OMAP3 defines an irq_chip for all the interrupts that it knows about directly, and the driver for the PMIC can define a separate irq_chip which describes the various interrupt sources in the PMIC. Each irq_chip will provide very different functions to configure and control the interrupt sources, but will provide a uniform interface to all device drivers. When the single PMIC interrupt arrives at the INTC, it will call an interrupt handler which will examine the PMIC registers and then call the appropriate handler that was registered with the second irq_chip. This way the drivers for the various parts of the PMIC don't need to know too much about each other - the irq_chip mediates between them.

In our PMIC there is a "Primary Interrupt Status Register" which has one bit for each module that can generate interrupts. Each of these modules has their own "Secondary Interrupt Status Register" to report what actually caused the interrupt, much like the UART described earlier. There is normally no need for a tertiary irq_chip to represent these registers as the one driver manages the whole module and all of its interrupts.

There are two exceptions, only one of which interests us. There is a "power interrupts" module in the PMIC which combines interrupts from a diverse range of sources: press of the power button, insertion of USB cable, over-temperature warning, and the RTC alarm. These are sufficiently varied that a separate irq_chip makes sense here. The twl4030-irq driver manages all this quite elegantly, providing a secondary irq_chip for the device as a whole, and supporting tertiary irq_chips, including for the Power Interrupts module.

So when the RTC alarm triggers, the primary interrupt is handled by the INTC. This calls an interrupt handler which inspects the PMIC, determines that a power interrupt is active and calls the relevant handler. It in turn examines the PMIC again and finds that the RTC alarm has fired, and so calls the handler that was registered for that particular interrupt. This all works perfectly. But it doesn't seem to wake the device from suspend. At least, not always. It seems that something happens on the way to suspend that interferes with this.

A funny thing happened on the way to suspend

One of the many things that happens on the way to suspend is that each individual interrupt gets disabled - unless it is flagged as IRQF_NO_SUSPEND in which case it is left alone. However this doesn't mean exactly what it sounds like it means. Being "disabled" just means that the handler routine will not be run, it doesn't mean that the interrupt will not be generated. We have a different word for that, which is "masking". When an interrupt is masked the originating source of the interrupt is told to never post that interrupt.

Linux uses a lazy scheme for disabling interrupts. When the disable request is made, the fact is recorded in internal data structures, but that is all. If the interrupt is subsequently delivered, only then might the interrupt be masked. This can be a useful optimization as masking an interrupt can take a lot longer than just setting a flag in memory.

So, on the way to suspend, interrupts are disabled but not masked. If the interrupt does actually arrive before we reach full suspend, the fact is recorded. If it was an interrupt that should wake from suspend, this is detected in check_wakeup_irqs() and suspend aborts. If the interrupt doesn't arrive before full suspend, then it is still unmasked and will successfully wake up the device, which will resume and then handle the interrupt. This might all seem a bit complex, but once it is fully understood it is actually quite neat and it works well ... except for my RTC alarm.

Who turned out the lights?

As promised we will need to glimpse at the USB code as well and now is the time to bring that in. Like most modern phones, the GTA04 can be attached to a computer or a charger via a USB port. When the USB interface detects 5 volts on the power line, it will route this to the battery charger (BCI) which will use it to charge the battery and power the device.

This is relevant because - when tracing is added to the twl4030-irq driver - we see that during suspend we get an interrupt from the PMIC which turns out to be due to the battery charger losing power and saying "Hey, just thought you should know that I'm running on battery again now". What is happening is that when the USB port is told to go to sleep it turns off its various power supply regulators. One of these (USB3V1) also powers the voltage comparator which determines if 5V is present. As soon as that presence is no longer signaled, the current stops being routed to the BCI, and the BCI justifiably complains.

This results in the interrupt line from the PMIC being asserted which causes the interrupt management code to run. This notices that the interrupt was disabled, and so masks it. When we get to check_wakeup_irqs(), the PMIC IRQ is pending, but as it has not been set for wakeup it is just ignored. The net result here is that all interrupts from the PMIC are ignored during suspend, so the RTC alarm doesn't work, the power button doesn't wake the device up, and plugging or unplugging the USB cable has no effect either.

So now that we understand the problem ...

So, what to do? We could just unplug the USB cable when testing the RTC alarm. Then the BCI would not notice the current disappearing and so wouldn't generate an interrupt. This certainly works - once. It doesn't seem to work a second time, but we'll leave that issue for the moment.

We could tell the USB controller not to turn off USB3V1 when the BCI is active and power is available on the USB bus. This is certainly a good idea and would mean that the battery can charge during suspend. It also brings up the interesting issue of how these separate drivers should communicate, but it doesn't really solve the general problem, just this specific one.

It could be that an interrupt occurs that we really do want to ignore during suspend, that we have even explicitly disabled (but due to lazy-masking has not been masked). So we want to be able to mask those interrupts without masking the main interrupt. A few moments reflection on what we have learned so far suggests that setting IRQF_NO_SUSPEND on the intermediate interrupts should help. That way they will not be disabled during suspend so the interrupt management will trace through the chain of irq_chips far enough to find the originating interrupt source and will mask just that. The other interrupt source will still be available.

This may sound convincing, but reality has a way of getting in the way: testing shows that just setting IRQF_NO_SUSPEND is not enough. Diving back into the code and the data sheets reveals an interesting and important fact. The PMIC allows each individual interrupt source to be masked, but it does not allow a whole module as a set of interrupt sources to be masked. So when an interrupt arrives from the BCI (one of the modules) the IRQ manager notices that interrupt is disabled and so the secondary irq_chip is asked to mask it. But as the irq_chip cannot mask it (the hardware doesn't support that), it simply ignores the request. You can imagine what happens next - an unending stream of attempts to mask the BCI interrupt, none of which are effective and the interrupt keeps re-firing.

The root problem seems to be that while the TWL4030 PMIC appears to have an interrupt structure that lends itself to being represented as a small tree of irq_chips, this isn't actually the case. The details of the behavior of suspend makes it essential that a working "mask" function be provided, and that cannot be provided for modules in the twl4030, only for individual interrupts. So it seems that a complete fix requires some change to the structure of the irq_chips in the twl4030 driver. Possibly we could make the structure simpler (no tree), or possibly we could make the implementation more clever in some way. The important point is that each interrupt that is visible to the Linux IRQ management code must support being masked in the hardware in some way.

Another funny thing on the way to resume

As mentioned, after one successful wake-from-alarm, the RTC alarm in the GTA04 doesn't work a second time. In fact it stops working completely, even when not suspended. It seems that something has gone wrong on the way to resume.

During resume all the interrupts that were disabled by suspend are re-enabled. The three interrupts which relate to the RTC - the base PMIC interrupt, the interrupt for the "power interrupts" module, and the actual RTC interrupt - are enabled in that order with PMIC first. As this interrupt is still asserted (as it is what triggered resume) the PMIC interrupt handler runs as soon as it is enabled. It reads the interrupt status register to see which module was responsible, and triggers the relevant interrupt for that module.

This secondary interrupt is still disabled so nothing happens. In particular the interrupt source is not cleared, nor is it masked, so we again get a repeated sequence of attempts to handle an interrupt which cannot be handled. This one isn't unending as the loop that enables all the interrupts is still running and it eventually gets to the one for the target module (the power interrupts module). This is triggered again and this time it actually does something. It reads the final status register (thus clearing the interrupt) and calls the interrupt handler for the RTC alarm. Unfortunately that is still disabled so nothing else happens.

Eventually the RTC interrupt is enabled but by now it is too late. In some cases an interrupt that fires while disabled gets the IRQS_PENDING flag set so that when it is re-enabled, the interrupt is resent. But that is not the case when the twl4030-irq interrupt controller tries to handle a tertiary IRQ. Whether this is a bug or a misunderstood feature is as yet unclear. Another journey of discovery would be needed to fully understand that issue. Meanwhile, because the interrupt is delivered but the interrupt service routine isn't run, the handshaking between the driver and the device gets confused and no other interrupt is ever seen from the RTC.

We can work around this problem by a simple expedient. When re-enabling the interrupts during resume, do it in the reverse order. This ensures that the RTC interrupt is enabled before it is called, and everything works. This even feels like the "right" thing to do - it is balanced for "enable" to happen in the reverse order of "disable". But even if it is "right", it is not really a fix. There would still be bugs lurking and ready to spring out again.

Journey's end

The point of this little tour is not to point out bugs in the code or to shame the relevant developers. In fact the code is in general quite elegant, well structured, largely very effective, and the developers are only to be congratulated. Rather, the point is to highlight the complexity of the task being tackled by developers working the embedded space:

• There are complex interactions between distinct hardware. Here the USB interface and the battery charger have important interdependencies that need to be reflected in the drivers. Some interdependencies already are but there are subtleties that are easy to miss.

• The requirements of an "irq_chip" are not really documented anywhere, and given the current rate of development there is a good chance that such documentation would be out of date. Extracting the requirements from the code is far from trivial. In this case a seemingly correct and obvious implementation was only found to be deeply wrong by a fairly obscure test case.

• The IRQF_NO_SUSPEND flag is clearly important, but not easy to understand. It is fairly obvious what it does, but much less obvious why you would want to do that. I imagine that as a developer my approach would be to not set it until I found a bug which could be fixed by setting it. Then I would set it and hope it didn't break anything. This is not really a robust way to do development.

With complex hardware interactions and complex software dependencies it is not surprising that we seem to see a new class of problems in the embedded space. As we have observed, there are often ways to work around the bug rather than fix it, and that is usually much simpler and quicker. They may be tempting, but for this developer at least they are nowhere near as satisfying. Because if you follow the bug all the way to the root cause, and if you understand all that is happening and find the right fix, then you have not only helped other future developers, but have learned a whole lot in the process. And that is why I love bugs.

Comments (11 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.3-rc4 . (February 19, 2012)

• Greg KH: Linux 3.2.7 . (February 20, 2012)

• Greg KH: Linux 3.0.22 . (February 20, 2012)

• Steven Rostedt: 3.0.22-rt37 . (February 21, 2012)

Core kernel code

• Grant Likely: irq_domain generalization and rework . (February 16, 2012)

• Oleg Nesterov: make vfork() killable . (February 17, 2012)

• Jason Baron: jump label: introduce very_[un]likely + cleanups + docs . (February 22, 2012)

• Rafael J. Wysocki: PM: Implement autosleep and "wake locks", take 2 . (February 22, 2012)

Development tools

• Jiri Olsa: perf tool: parser generator for events parsing . (February 17, 2012)

• Borislav Petkov: RAS trace event proto . (February 21, 2012)

Device drivers

• Laxman Dewangan: Open drain/open source support from gpio library . (February 17, 2012)

• Stephen Warren: pinctrl: API change, config in mapping table . (February 20, 2012)

• Mugunthan V N: Adding new TI Common Platform ethernet SWitch driver . (February 21, 2012)

• Sakari Ailus: [PATCH v3 0/33] V4L2 subdev and sensor control changes, SMIA++ driver and N9 camera board code . (February 21, 2012)

• Aaron Sierra: mfd: Add LPC driver for Intel ICH chipsets . (February 21, 2012)

• Aaron Sierra: gpio: Add support for Intel ICHx/3100/Series[56] GPIO . (February 21, 2012)

Documentation

• Laurent Pinchart: Kernel Display and Video API Consolidation mini-summit at ELC 2012 - Notes . (February 17, 2012)

Filesystems and block I/O

• Miao Xie: Btrfs: introduce extent buffer cache for each i-node . (February 17, 2012)

• Tao Ma: ext4: Add inline data support . (February 21, 2012)

• David Howells: Union Mount [ver #2] . (February 22, 2012)

• Tejun Heo: blkcg: accumulated blkcg updates . (February 22, 2012)

Memory management

• KAMEZAWA Hiroyuki: page cgroup diet v5 . (February 17, 2012)

• Marek Szyprowski: Contiguous Memory Allocator . (February 17, 2012)

• Aneesh Kumar K.V: hugetlbfs: Add cgroup resource controller for hugetlbfs . (February 20, 2012)

• Hugh Dickins: mm/memcg: per-memcg per-zone lru locking . (February 21, 2012)

• Glauber Costa: memcg kernel memory tracking . (February 22, 2012)

Networking

• Javier Martinez Canillas: af_unix: add multicast and filtering features to AF_UNIX . (February 21, 2012)

Architecture-specific

• Mahesh J Salgaonkar: [RFC PATCH v7 00/10] fadump: Firmware-assisted dump support for Powerpc. . (February 16, 2012)

• H. Peter Anvin: RFC: x32 support . (February 20, 2012)

• Thierry Reding: Add PWM framework and device tree support. . (February 22, 2012)

• Jason Cooper: arm: kirkwood: add dreamplug/devicetree support . (February 22, 2012)

Security-related

• Kees Cook: fs: hardlink creation restrictions . (February 21, 2012)

Miscellaneous

• Douglas Gilbert: lsscsi-0.26 released . (February 20, 2012)

Page editor: Jonathan Corbet

Distributions

FOSDEM: Multiarch on Debian and Ubuntu

In his talk at FOSDEM (Free and Open Source Software Developers' European Meeting) 2012 in Brussels, Wookey (who is working for Linaro on Linux for ARM and doesn't have a first name) talked about what multiarch is and why it's important. Multiarch is a general solution for installing libraries of more than one architecture on the same system.

By "general" we mean more than just the lib and lib64 directories for 32 and 64-bit x86 libraries. Currently the Filesystem Hierarchy Standard (FHS) attempts to address the use of these libraries on the same system by requiring that /usr/lib be reserved for 32-bit libraries, while 64-bit libraries are located in /usr/lib64. This so-called "biarch" design was adopted by Red Hat and SUSE, but not by Debian and Ubuntu. A general solution should not only scale to other architectures, but it should also "remove all corresponding bodgery we have in Debian, such as ia32-libs and biarch packages," Wookey says. The Debian developers have been working on a multiarch solution for years and multiarch support is a release goal for the coming Debian 7 "Wheezy" release, expected in 2013.

The basic idea behind multiarch is to generalize the biarch design to arbitrary architectures, and the way it is done is actually quite simple, Wookey maintains: you put your libraries into architecture-specific paths. For instance /usr/lib/libfoo goes into /usr/lib/x86_64-linux-gnu/libfoo if your machine has an x86_64 architecture, into /usr/lib/i386-linux-gnu/libfoo for an i386 architecture, into /usr/lib/powerpc64-linux-gnu/libfoo for a ppc64 architecture, and /usr/lib/arm-linux-gnueabi/libfoo for an armel architecture.

The multiarch paths contain the GNU triplets used by GCC to describe architectures. For instance, in x86_64-linux-gnu "x86_64" stands for the processor type, "linux" designates the kernel, and "gnu" stands for the user-space ABI. However, multiarch adopts the GNU triplets with some adjustments. For instance, both the i486-linux-gnu and i586-linux-gnu GNU triplets will be translated to the /usr/lib/i386-linux-gnu/libfoo path because, according to Wookey, a few minor instruction set differences do not add up to a different ABI requiring its own triplet. The advantage of this partial rethinking of the file system hierarchy is that all libraries have a canonical path. There are no special cases for the locations of native, cross-built, or emulated (with QEMU) libraries: they are all the same.

What can we do with it?

So what can we do with multiarch? As already mentioned, multiarch makes cross-compilation much simpler: it is no longer a special case and essentially you're getting it for free as a byproduct of multiarch. This is primarily because the library loader path is baked into every executable by the linker, and thanks to multiarch's canonical path based on the system's architecture, this path is the same whether the library is built or cross-built.

In the classical approach of cross-building for armel (with dpkg-cross), the build-time library path is for instance /usr/arm-linux-gnueabi/lib/libfoo, while the runtime library path is just /usr/lib/libfoo. With the multiarch approach for cross-compilation, the library path is /usr/lib/arm-linux-gnueabi both at build time and at run time, so "it's much harder for libtool to screw it up," Wookey said. Another advantage is that you can just run the build tools under QEMU via binfmt-misc for testing.

Multiarch also allows for a better support for binary-only software, which tends to be only available for 32-bit systems. Thanks to multiarch, you can more easily install 32-bit versions of a 32-bit proprietary program's dependencies on a 64-bit system. Wookey gave as examples the Flash plugin, Skype or Xilinx development tools. Multiarch also allows cheap emulated environments: you can emulate only the parts you need.

The slow genesis of multiarch

Wookey quotes Tollef Fog Heen, who said in 2005: "ia32-libs [is now] the biggest source package in Debian." That is because currently any 32-bit software that has to run on an amd64 (which is the name Debian uses for x86_64) installation depends on the package ia32-libs, which contains i386 versions of all of the libraries, so its source package currently weighs in as a 555 MiB tarball. Ia32-libs was always intended as a temporary solution for the i386/amd64 case, but unfortunately (as often happens with these things), developing the proper general replacement took a lot longer. There were talks about a solution at Debconf 4 and 5 (in 2004 and 2005, respectively), there was a multiarch meeting at FOSDEM 2006, and in June 2006 the first multiarch patches for dpkg were uploaded.

In May 2009, the apt and dpkg maintainers agreed on a package management specification for multiarch at the Ubuntu Developer Summit in Barcelona. To avoid further delays, they restricted the scope to multiarch libraries. In August 2010, the first proposal for multiarch directory names was drafted, and in February 2011 a dpkg multiarch implementation (sponsored by Linaro) landed in Ubuntu. A month later, the normalized GNU triplets were adopted for the multiarch directory names, and then the Ubuntu 11.04 release came with 83 libraries multiarched. Together with 14 multiarch libraries in a PPA (Personal Package Archive), this was already enough to cross-install the 32-bit Flash plugin on a 64-bit system.

Currently, the Ubuntu core is almost completely multiarch: at the time of Wookey's talk, 110 out of the 112 source libraries in the Ubuntu 12.04 main repository were multiarched, as well as 175 out of the 176 binary libraries. Obviously all libraries have to be made multiarch-ready, but also most -dev packages need converting as well, using a similar directory naming scheme as for libraries. That makes it possible to co-install include files that differ between architectures. But on top of this, any tool that is aware of library paths had to be fixed, including libc, dpkg, apt, compilers, make, pkg-config, pmake, cmake, debhelper, lintian, libffi, OpenJDK's lib-jna, and dpkg-cross.

Wookey made it clear that the multiarch development is a classic example of a significant distribution-wide change, which is generally very difficult to do right. One of the factors in the success of the multiarch development is that they used written specifications to record a shared understanding. As can be seen from the project's history, another key factor is that they split the work into bite-sized deliverables.

How does it work?

Normally, a package of the same name but a different architecture is not co-installable. Multiarch-ready packages, though, are given an extra field Multi-Arch in the package specification. This field has one of three possible values, depending on the type of package. A library has the value "same": it can be co-installed with the same package from another architecture, but it can only be used to satisfy the dependencies of a package within the same architecture. An executable has the value "foreign": it cannot be co-installed with the same package from another architecture, but it should be allowed to satisfy the dependencies for any architecture (of course, preference is given to a package for the native architecture if available). And a package that contains both libraries and executables has the value "allowed". An example of this is the python package. The depending packages specify how they use it.

The Debian wiki has some information about how package maintainers can convert their package to multiarch, as well as some general information about multiarch support in Debian. Note that a package for a foreign architecture is only installable if all of its (recursive) dependencies are either marked as multiarch or do not have corresponding packages installed for the native architecture.

An interesting implementation detail is that co-installability doesn't mean that documents from a package get installed twice when you install two architectural versions of it: according to Wookey, dpkg has support for reference-counting of documentation files from co-installable packages that overlap. So an identical documentation file in a 32 and 64-bit x86 version of a library only gets installed once, and it doesn't get removed until both versions of the library are removed.

In practice, you can easily add a new architecture to your machine's Debian or Ubuntu installation. For instance, when you have an amd64 installation and you want to install some i386 libraries, you can add the latter architecture with a simple "dpkg --add-architecture i386" command. Use "dpkg --print-foreign-architectures" to get a list of the foreign architectures you have added, and "dpkg-architecture -qDEB_HOST_MULTIARCH" to see the multiarch pathname for your system's native architecture. The entries in /etc/apt/sources.list also get an extra arch field, for instance:

    deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu precise main

Things multiarch (currently) doesn't do

Currently the multiarch solution is limited to libraries. This means that you can't install executables from more than one architecture in /bin or /usr/bin with multiarch. Co-installable executables could be useful (for instance to reuse a single network-mounted root partition on systems of multiple architectures with no modification), but it is deliberately left out of the initial implementation because it would complicate matters further than they already are. Other than a multiarch path for executables, such a system would require kernel support or boot-time symlinking. Before implementing this, the multiarch developers need a detailed specification as they have done with the implementation for libraries, Wookey warned.

Another interesting but currently not implemented feature is that you could "cross-grade" your machine from one architecture to another one. For instance, if you have installed a 32-bit x86 distribution on your 64-bit machine, you could convert it to a 64-bit distribution without having to reinstall it. This could be possible by first manually installing the 64-bit versions of dpkg and apt and then changing which architecture is used by default, after which you could reinstall all installed software, but from the 64-bit architecture. This should work the same way for a cross-grade from arm to armel and from armel to armhf.

Wookey ended his talk with the message that Debian and Ubuntu have now done the hard work for multiarch and shown that it works. However, it could be useful beyond Debian and its derivatives. The multiarch directory scheme will be a target for FHS/LSB standardization in the future, but even if that doesn't happen, it's a much more scalable solution than the current one.

(Those wanting more details can watch the video of Wookey's talk posted by the conference.)

Comments (30 posted)

Brief items

Distribution quotes of the week

Comments (none posted)

Debian Position on Software Patents

Full Story (comments: none)

Mageia 2 beta 1 available

Comments (6 posted)

openSUSE 12.2 Milestone 1

Comments (none posted)

Scientific Linux 6.2

Comments (2 posted)

Ubuntu 10.04.4 LTS released

Full Story (comments: none)

Ubuntu for Android

Comments (32 posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (February 20)
• DistroWatch Weekly, Issue 444 (February 20)
• Maemo Weekly News (February 20)
• Ubuntu Weekly Newsletter, Issue 253 (February 19)

Comments (none posted)

Lea: The Unity design process (and how you can play a part in it)

Comments (107 posted)

Open webOS governance model described

Comments (6 posted)

What's new in Red Hat Enterprise Linux 5.8 (The H)

Comments (none posted)

Cox: Enterprise Linux 5.7 to 5.8 risk report

Comments (none posted)

Page editor: Rebecca Sobol

Development

Collaborative book authoring with Booktype

Sourcefabric, the free-media-software project behind the Airtime radio broadcasting application (covered here last week), has also launched a new book authoring effort dubbed Booktype. At its heart, Booktype is a collaborative, web-based rich document editor, but it is designed to streamline the process of creating and publishing complete books — using the same text, regardless of whether the end result is exported for print, ebook readers, or the web. In addition to its editing and formatting capabilities, the software integrates tools to manage teams and communities around a book project, which may make it a better fit for documentation jobs and user communities than a generic content management system (CMS).

Booktype is the product of a collaboration itself. The code is an evolution of the Booki editing platform used to great success by FLOSS Manuals. Although FLOSS Manuals had been using Booki for its own documentation projects and book sprints for several years, the code had never been packaged and distributed as a general-purpose tool. The partnership with Sourcefabric was announced on February 14 by FLOSS Manuals founder Adam Hyde, and the new version of the code was unveiled at the O'Reilly Tools of Change For Publishing conference the same day.

The code is licensed under the AGPL version 3, and is written primarily in Django using PostgreSQL for storage. The source is hosted at Sourcefabric's Github repository, where the installation instructions list several other library dependencies. The most notable are Espri and Objavi, the content-import and rendering engines (respectively), developed for Booki. At the moment, neither is packaged for distribution, although there is an empty-at-present Objavi repository, and the code is accessible from the Booki repository. Instead of using installed versions of those components, the Booktype code links directly to FLOSS Manuals' hosted versions. However, the project does make both freely available for outside use, but until they are released as stand-alone packages, the hard-coded links will have to suffice.

The initial Booktype release is numbered 1.5, to reflect the code's Booki-era history. Sourcefabric's Adam Thomas said that the principle changes include an overhaul to the user interface, full localization, and the addition of non-ASCII character support in auxiliary functions (such as the Django slug creator). It is now possible to upload exported PDFs directly to a Lulu.com publishing-on-demand account, and to create EPUB and MOBI output tailored for Apple's iPad and Amazon's Kindle.

Anatomy of a book publishing job

The application is web-based, geared towards drag-and-drop simplicity in the editor, although there are a handful of command-line tools for administering a Booktype server. Editing a text is similar to using Google Docs, PiratePad, or any other contemporary collaborative editor. The main editing window uses TinyMCE, the same WYSIWYG rich text editor widget found in WordPress, Joomla, Plone, and many other open source CMSes. Simultaneous editing by multiple users is supported, as is revision control and roll-back of earlier changes. The site advertises full Unicode support as well as support for bi-directional text.

What makes Booktype a "book publishing" system rather than a web editor is its support for long, multi-part documents, its output options, and its focus on a collaborative workflow. Every user must have an account on the Booktype server, and the creator of a new book is automatically its owner. The owner can keep the title private, or open it up to the public — either the world at large, or a specific group of users — and can optionally assign user roles to distinguish between authors, editors, and proofreaders. Editing a book requires creating one or more chapters, and each chapter is essentially treated as a separate document in the application.

The "book view" allows an editor to rearrange the chapters using drag-and-drop. Users can also break the book into named "sections" and move the chapters freely between them. Booktype keeps track of the order of the material, and when exporting the final product, automatically numbers the sections and chapters, adds page numbers if print output is created, and generates a table of contents. The changes to the book's structure are noted in the interface's History tab (which, like MediaWiki's, allows you to compare different revisions side-by-side).

Whenever a book is ready for publication, its owner can freeze a revision of the text, which gets tagged with a software-style Major.minor release number. Version numbers are strictly-increasing, however; there is no "undo." The project describes this feature as being useful for textbooks and other projects that need to produce updates where it is always clear which revision is the most recent. When the text is finalized, publishing is a matter of choosing the output format — the same text is used to generate print, web, and ebook output.

The publishing feature currently offers five formats: print-ready PDF, ebook (with iPad, Kindle, and generic EPUB options), screen-resolution PDF, LibreOffice ODT, and the aforementioned Lulu.com upload. Internally, book text is stored as HTML and styled with CSS. The export function uses WebKit with built-in CSS templates to convert the HTML to the other output formats. At the moment, there is not much documentation of the templates used or their options. The "Publish" tab walks you through several set-up screens where you can alter settings like font style, size, margins, and gutters, but without a preview. As the Booktype manual notes, you can enter custom CSS on the final screen under "Advanced Options," although this is not particularly user-friendly.

According to Hyde, the exporter supports all of WebKit's CSS features, which mean only partial support for advanced text formatting and font features. However, he added, "what is most interesting is that we can support CSS3 and JavaScript, There are some very nice JavaScript typography libraries and we have been experimenting with the integration of kerningjs. It's experimental but looks promising."

Socializing

The "social dimension" of Booktype is its other distinguishing feature. It has been designed from the ground up to allow users to create their own book projects at will, and to contribute to each others' texts. That plays out in the ease with which one can start a new book, and with Booktype's ability to import chapters from public-domain works (such as the Internet Archive) and Creative Commons-licensed ebooks. The application also allows you to create groups in addition to isolated user accounts, and to assign editing rights to a group. User and group profile pages are minimal at the moment, but the feature is clearly intended to support large-scale group efforts akin to FLOSS Manuals' own contributor community.

In addition to the broader social networking designs, there are nice collaboration features built in to the application itself. During the writing and editing process, all changes are logged and are visible to other logged-in users on the same project in a sidebar next to the main editor pane, above a list of other users currently working on the same book. The automatically-logged changes include simple "User Nate has created new chapter 'Introduction'" and "User Nate has saved chapter 'Chapter 2'" messages, but collaborators can leave longer text notes for one another, or send chat messages through the sidebar if both are logged in at the same time.

Book owners and editors can flag chapters as "content needed," "review needed," or any other tag. This stops a little short of permitting a book owner to assign tasks to participants, but Booktype seems designed to fit ad-hoc groups of willing participants, rather than paid employees at a publishing house. FLOSS Manuals has certainly gotten significant mileage out of the volunteer-driven, casual workflow of Booki, which is an endorsement of the idea.

Coda

There are a few features discussed in the press release and project page that do not appear to be active in the live Booktype demo, such as publishing directly to Amazon.com or Apple iBooks, or output to static (i.e., non-Booktype) HTML. Coupled with the still-unpackaged status of the Espri and Objavi components, it is possible that the initial release of Booktype was not quite as ready as the project would have liked when the Tools of Change For Publishing event arrived.

Nevertheless, what is there is an easy-to-use collaboration platform suitable for writing-intensive tasks like documentation. FLOSS Manuals' output is of high quality for a number of reasons — including the skills and time contributed by its volunteers — but over the years the project's software has developed into something other communities ought to examine as well. Personally, I would much rather see software documentation written in Booktype than in MediaWiki, with its idiosyncratic syntax and the pages of empty outlines that inevitably result. The workflow tools in Booktype allow an editor to orchestrate a large volume of text with minimal effort, and the application takes care of the formatting automatically. Both features free writers to produce better writing.

Thomas and Hyde both said that work is already underway for the next revision of the code, which should include support for audio, video, and interactive elements. Until then, Booktype is ready to craft PDF, ebook, and print-on-demand output, which gives the rest of us plenty to read.

Comments (none posted)

Brief items

Quotes of the week

Comments (10 posted)

Apache HTTP Server 2.4.1 released

Full Story (comments: none)

VLC 2.0 released

Comments (48 posted)

Toward Wayland 1.0

Full Story (comments: none)

Kuhn: Busybox GPL enforcement concerns resolved

Full Story (comments: 36)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (February 21)
• Perl Weekly (February 20)
• PostgreSQL Weekly News (February 19)

Comments (none posted)

Gettys: Diagnosing Bufferbloat

Comments (15 posted)

Torrone: The {Unspoken} Rules of Open Source Hardware

Comments (none posted)

Gräßlin: The Costs of Supporting Legacy Hardware

Comments (82 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

The Document Foundation incorporated

Comments (none posted)

No more Flash for Firefox on Linux

Comments (60 posted)

Time zone database suit withdrawn

Comments (16 posted)

Articles of interest

Seigo: Spark pre-order registration is open!

Comments (53 posted)

Event Reports

ABS/ELC videos available

Comments (4 posted)

Upcoming Events

Upcoming Debian Bug Squashing Parties

Full Story (comments: none)

Python Game Programming Challenge (PyWeek)

Update: As this correction notes, the dates were wrong in the original (as reflected in the strike-through above).

Full Story (comments: none)

PyCon DE 2012

Full Story (comments: none)

Events: February 23, 2012 to April 23, 2012

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol