LWN.net Weekly Edition for February 7, 2013

LCA: Linux gaming at last

At the beginning of his talk on Linux game development at linux.conf.au 2013, Eric Anholt noted that his original motivation for getting into graphics driver development was that he wanted to be able to play games on alternative operating systems. At that time, there were a few games available, but the graphics drivers were in such a poor state that he ended up working on drivers instead of playing games. Thus, he has now been working in Intel's graphics driver team for seven years; currently, he works mainly on Mesa's OpenGL graphics drivers.

Eric's talk took the form of a short review of some recent history in Linux game development followed by a description of his experiences in the Intel open source graphics driver team working with Valve Software to port the Steam game platform to Linux.

Recent history of game development on Linux

Eric started with a summary of significant changes that have taken place in the graphics stack over the last seven years. These changes include kernel mode setting and improvements to memory management so that multiple processes can reliably employ memory on the graphics card. On the OpenGL side, things have improved considerably. "Back when I started, we were about ten years behind". The then-current OpenGL version on Linux was 2.1, no modern games ran on Linux, and there was no OpenGLES on Linux. However, by now, Linux supports OpenGL 3.1 and has reached Khronos-certified OpenGLES 2.0 conformance, and Open GLES 3.0 certification seems to be quite close.

Development of open source games seems to have lagged. Eric suggested a number of reasons for this. One of these is that creating a game requires building a multi-talented team composed of artists, developers, and designers. It's difficult to put a team like that together. And then, even if one does manage to assemble such a team, it's hard to agree on a direction: when it comes to game design, the design space is so large that it can be difficult to agree on what you are creating. Finally, the move into open source game development means that you spend less time doing the thing you want to do: playing games.

Nevertheless, there have been a few open source games, such as etracer (Extreme Tux Racer), Neverball, Xonotic, Foobillard, and Wesnoth. In addition, there were closed source games such as Quake (later open sourced), Unreal Tournament 2004, Loki, Minecraft, and whatever users could force to run with Wine.

In May 2010, the Humble Indie Bundle appeared. The concept was a package of games made available DRM-free, with the user choosing the price that they would pay. "They've actually released some surprisingly good games for Linux." One of those games was Braid, and Eric noted that the developers who participated in Humble Bundle learned an important lesson from an earlier attempt to port that game to Linux.

The developer of Braid, Jonathan Blow, made a blog post asking for help on how to port Braid to Linux, asking questions such as "how do I deal with mouse grabs, so that mouse clicks only go to the game window?" and "how do I output sound?" The community did try to help: in all, the blog post got 251 responses, many of them containing directly conflicting advice. In response, the developer gave up: he couldn't justify spending the time to determine the correct way to do the port for what was a small target market.

The lesson that the Humble Bundle developers learned from the Braid experience was that game developers should not be burdened with the task of porting games. So instead, they employed Ryan C. Gordon, a developer who had already ported a number of games to Linux, to port all of their games. This approach has been surprisingly successful at quickly getting games to run on Linux.

Working with Valve on the Linux port of Steam

There have been petitions for Valve Software to port Steam to Linux for as long as Steam has been around, Eric said. The Intel graphics driver team started working with Valve Software in July 2012. During the porting project, the Intel team had access to the Steam source code and worked with Valve on both tuning Steam to work better with the Mesa graphics library and tuning Mesa to work better with Steam. The closed beta test for the port was started in November 2012, and the open beta started in December. The port included the Steam's Source engine and the game Left For Dead 2.

The cooperative work between the graphics developers and Valve proved to be quite productive, but in the process, the graphics developers learned about a number of things that Valve found really disappointing. The first of these disappointments concerned ARB_debug_output. This is an extension in the Windows development environment for OpenGL that provides a general event logging infrastructure between the graphics layer on one side and middleware and applications on the other side. This is an important debugging tool in the Windows environment, "where you don't really have the console so much". There is an implementation of ARB_debug_output in Mesa, but it is rudimentary, supporting only two event types.

Another major disappointment concerned bad debugging and performance-measurement tools. The Valve developers described the tools they used for debugging on Windows, and wanted to know what equivalents existed on Linux. In response, the graphics developers were excited to show Valve an API-tracing tool that they had developed. API tracing is a technique that is extremely useful for driver developers because it allows them to obtain a reproducible trace of the graphics operations requested by the application. This means driver developers can get the application out of the way while they replay the trace in order to debug problems in the driver or improve the driver's performance. However, this sort of low-level tool provides little assistance for analyzing performance at the application level.

By contrast, Windows has some good tools in this area that allow the application programmer to insert tracepoints to track application steps such "starting to draw the world", "starting to draw a figure", and "starting to draw a helmet". This enables the application developer to obtain a hierarchical view of performance and determine if (say) drawing helmets takes a long time. Linux has to do a lot to catch up in this area.

The Valve developers also complained that Mesa was simply too slow. Many of the code paths in Mesa are not well optimized, with enormous switch statements containing nested conditionals. One possible solution is to offload some of the graphics work onto a separate thread running on a different core. Some work has been done in this area, but, so far, performance is no better than for the existing single-threaded approach. More work is required.

Notwithstanding the disappointments, there were other aspects of working on Linux that the Valve developers loved. For example, they greatly appreciated the short development cycles that were made possible when using open source drivers. Although the support for ARB_debug_output was poor on Linux, the Valve developers were impressed when Eric was able to quickly implement some instrumentation of driver hotspots, so that the driver would log messages when the application asked it to carry out operations that were known to perform poorly. The Valve developers were also surprised that the logging could be controlled by environment variables, so that the same driver would be used in both quiet and "logging" mode.

A final pleasant surprise for the Valve developers was that drivers could be debugged using breakpoints. That possibility is unavailable with closed source Windows drivers. More generally, the Valve developers don't have much insight into the closed source drivers that they use on Windows (or, for that matter, closed source drivers on Linux). Thus, they have to resort to experimentation to form mental models about performance problems and how to get around them.

Concluding remarks

For gaming enthusiasts, the announcement by Valve—one of the largest producers of game software—that Steam would be ported to Linux was something of a watershed moment. The Linux gaming landscape is poised to grow much bigger, and even those of us who are not gamers can reflect that a much-improved games ecosystem will at the very least widen interest in Linux as a platform for desktop computer systems.

Comments (17 posted)

FOSDEM: State of the GNUnion

The Free Software Foundation (FSF) has received criticism in recent months for its copyright assignment policies and for being slow in dealing with reported GPL violations. In a talk at FOSDEM on February 3, John Sullivan, the Executive Director of the FSF, addressed some of these issues. In his "State of the GNUnion" talk, Sullivan highlighted the FSF's recent licensing and compliance activities and described challenges that are important to the organization for 2013.

Licensing and Compliance Lab

Sullivan started with an overview of the members of the Licensing and Compliance Lab and its activities. The team is led by Josh Gay, the former FSF campaigns manager, and Donald Robertson, who has been handling copyright assignments for some time. While Sullivan helps to define the overall strategy employed for licensing in order to promote freedom, the team is supported by Richard Stallman, Bradley Kuhn (a Director of the FSF) and lawyers from the Software Freedom Law Center, in particular Aaron Williamson and Eben Moglen. Finally, there's a team of volunteers that helps out with questions that come in through the licensing@fsf.org address. Sullivan noted that it is important for the FSF to communicate with people about license choices and related topics.

The Licensing and Compliance Lab focuses on a number of areas. A big one is the production of educational materials about GNU licenses. It also investigates license violations, especially for code entrusted to the FSF. Finally, it certifies products that use and require only free software.

Licensing is important, Sullivan said, because all software is proprietary by default. The GPL grants rights to users and he believes that the GPL is the right license to boost free software adoption. He mentioned claims that the use of the GPL is declining, but criticized those studies for not publishing their methodology or data. His own study, based on Debian squeeze, showed that 93% of packages contained code under the GPL family. He noted the difficulty of measuring GPL adoption: does a package count if it contains any GPL code, or should you count lines of code? And what about software that is abandoned? Sullivan noted that his interest in GPL adoption is obviously not because the FSF makes money from licensing but because of his belief that the GPL provides the "strongest legal protection to ensure that free software stays free".

Sullivan highlighted a new initiative to create more awareness of GNU licenses. The lab has started publishing interviews on its blog to share insights about the license choice of different projects. Recent posts have featured Calibre and Piwik.

Compliance efforts and copyright assignment

The FSF collects copyright assignments in order to enforce the GPL, Sullivan said, but there are a number of misconceptions about that. He explained that the GNU project does not mandate copyright assignment and that individual projects have a choice when they join the GNU project. However, if a project has chosen to assign copyright, all contributions to that project have to be assigned to the FSF.

The FSF hears frequent complaints that the logistics of copyright assignment slow down software development within the GNU project. It has made a number of changes to improve this process. Historically, the process involved asking for information by email, then mailing out a paper form, getting it signed, and then sent back. These days, the FSF can email out forms more quickly. It also accepts scanned versions in the US and recently expanded this option to Germany after getting a legal opinion there. Sullivan noted that the laws in many places are behind the times when it comes to scanned or digital signatures. Having said that, the FSF is planning to accept GPG-signed assignments for some jurisdictions in the future.

Sullivan lamented that the FSF's copyright assignment policy is often used by companies to justify copyright assignment. He noted that there are significant differences between assigning copyright to an entity like the FSF and a company with profit motives. Not only does the FSF promise that the software will stay free software but it would also jeopardize FSF's non-profit charity status if they were to act against their mission.

One reason the FSF owns the copyright for some GNU projects is to perform GPL enforcement on behalf of the project. He discussed recent complaints that the FSF is not actively pursuing license violations, notably the issues raised by Werner Koch from the GnuPG project. Sullivan explained that this was, to a large degree, a communication problem. The FSF had in fact gone much further than Koch was aware of, but they failed to communicate that. He promised to keep projects better informed about the actions taken. Unfortunately, a lot of this work is not discussed publicly because of its nature. The FSF usually approaches companies in private and will only talk about it in public if no agreement can be reached. Also, if it comes to legal action, the FSF once again cannot talk about it in public.

The lab closed 400 violation reports in 2012, Sullivan said. Out of those, some turned out not to be violations at all, but the majority of violation reports were followed up by actions from the lab that resulted in compliance. He also noted that the FSF is planning to add additional staff resources in order to respond to reported violations more quickly.

JavaScript and non-free code running in browsers

Sullivan then went on to cover a number of challenges facing the free software world. Richard Stallman described the "JavaScript Trap" a few years ago, which is the problem of non-free code running in web browsers.

Sullivan explained that these days browser scripts can be quite advanced programs but "for some reason we've been turning a blind eye" to their typically non-free nature. The FSF is spending a lot of time on tackling this problem and has created LibreJS, which is an extension for Mozilla-based browsers. LibreJS identifies whether JavaScript files are free software, and it can be configured to reject any script that's not free.

In order for this to work the FSF developed a specification that web developers can follow to mark their JavaScript code as free software. Developers can either put a specific header in their JavaScript files or publish license information using JavaScript Web Labels. Gervase Markham pointed out that Mozilla uses web server headers and Sullivan agreed that LibreJS could be enhanced to support that method too.

Sullivan added that many JavaScript files are free software already, but that developers have to start marking them as such. They are working with upstream projects, such as MediaWiki and Etherpad Lite, on doing so.

Certification program

The FSF has launched a certification program to identify hardware that only uses free software. It wants to make it easy for people to care. Sullivan emphasized that the label has to be attractive and hopes it will cause manufacturers to respect user freedom more. He showed a different label similar in style to the warning note on a cigarette package ("This product may contain material licensed to you under the GNU General Public License") and explained that this "is not what we want to do". The actual logo (seen at right) shows the Liberty Bell along with the word "freedom". The first product to achieve certification is the LulzBot 3D printer.

User freedom

As an alternative to Android, Sullivan recommended Replicant—a fully free Android distribution—for those willing to sacrifice some functionality (such as WiFi and Bluetooth on Sullivan's mobile phone) for freedom. He also encouraged Android users to take advantage of the F-Droid Repository to download free software apps for their devices. F-Droid also provides the option to make donations to the authors of the free software apps.

Sullivan also briefly commented on UEFI secure boot. He said that while the FSF is obviously "annoyed by it", it is not fully opposed—there is nothing inherently wrong with secure boot as long as the power remains with the users. However, it's important to make a distinction with what he called "restricted boot". Restricted boot can be found on ARM-based Windows devices which lock down the device and don't give users any choice. This is obviously not acceptable, according to the FSF.

Concluding remarks

Sullivan gave an interesting overview of the FSF's recent activities and upcoming challenges it intends to tackle. He is aware of concerns that have been expressed by members of the GNU community in recent months and is keen to improve the situation. The talk showed that the FSF is working on many activities and that it hopes to improve and expand its work as funding allows.

Comments (28 posted)

Trademarks and their limits

The core idea behind a trademark is that it gives the owner the exclusive right to apply the trademarked name to a product or service. Thus, for example, the Mozilla Foundation owns the trademark for the name Firefox as applied to "computer programs for accessing and displaying files on both the internet and the intranet"; a quick search on the US Patent and Trademark Office site shows other owners of the name for use with skateboards, bicycles, wristwatches, power tools, and vehicular fire suppression systems. Within the given domain, the Mozilla Foundation has the exclusive right to control which programs can be called "Firefox". The Foundation's trademark policy has been seen by some as being overly restrictive (almost no patches can be applied to an official release without losing the right to the name); that is why Debian's browser is called "Iceweasel" instead. But those same restrictions allow the Mozilla Foundation to stop distribution of a program called "Firefox" that sends credit card numbers to a third party.

The Document Foundation (TDF) owns a trademark on "LibreOffice" in the US, while the Apache Software Foundation (ASF) owns "Apache OpenOffice" and "OpenOffice.org". Both foundations have established trademark usage policies (TDF, ASF) intended to ensure that users downloading their software are getting what they expect: the software released by the developers, without added malware. Without this protection, it is feared, the net would quickly be filled with corrupted versions of Apache OpenOffice and LibreOffice that would barrage users with ads or compromise their systems outright.

How effective is this protection? To a degree, trademarks are clearly working. Reports of systems compromised by corrupt versions of free office suites are rare; when somebody attempts to distribute malware versions, trademarks give the foundations the ability to get malware distributors shut down relatively quickly. It seems hard to dispute that the application of trademark law has helped to make the net a somewhat safer place.

Questionable distributors

One might ask: safer from whom? Consider, for example, a company called "Tightrope Interactive." Tightrope was sued by VideoLan.org (the developers of the VLC media player) and Geeknet (the operators of SourceForge) in 2010; they were accused of "trademark infringement, cyberpiracy and violating California's consumer protection law against spyware." Tightrope had been distributing "value-added" versions of VLC from its site at vlc.us.com; it was one of many unwanted VLC redistributors during that time. That litigation was settled in 2011; the terms are mostly private, but they included the transfer of vlc.us.com over to VideoLan.org, ending the use of that channel by Tightrope.

On Friday, April 15, 2011, Oracle announced that OpenOffice.org would be turned into a "community project" of an (at that point) unspecified nature. On April 18 — the next business day — Tightrope Interactive filed for ownership of the OpenOffice trademark in the US. That application was eventually abandoned, but not willingly; as Apache OpenOffice contributor Rob Weir recently noted in passing, "It took some special effort and legal work to get that application rejected." Companies in this sort of business clearly see the value in controlling that kind of trademark; had Tightrope Interactive been successful, it would have been able to legally distribute almost any software under the name "OpenOffice." The fact that the project successfully defended the trademark in this case should impede the distribution of corrupted versions of Apache OpenOffice in the future.

Sample OpenOffice ads

A quick search of the net will turn up complaints (example) about unwanted toolbars and adware installed by redistributed versions of OpenOffice, including Tightrope's version. This apparently happens often enough that the Apache OpenOffice project felt the need to put up a page on how to safely download the software, saying:

This problem is not restricted to Apache OpenOffice; a search for LibreOffice will turn up a number of similar sites. Given that, one might well wonder whether trademarks are actually living up to the hopes that have been placed on them. Isn't this kind of abusive download site just the sort of thing that trademarks were supposed to protect us from?

One answer to that question can be found on one of the LibreOffice download sites, where it is noted that clicking on the "Download" button will start with the "DomaIQ" installer. This bit of software is described in these terms:

Herein lies the rub. The version of Apache OpenOffice or LibreOffice offered by these sites is, most likely, entirely unmodified; they may well be shipping the binary version offered by the project itself. But the handy "installer" program that runs first will happily install a bunch of unrelated software at the same time; by all accounts, the "suggestions" for "additional free software" tend to be hard to notice — and hard to opt out of. So users looking for an office suite end up installing rather more software than they had intended, and that software can be of a rather unfriendly nature. Once these users find themselves deluged with ads — or worse — they tend to blame the original development project, which had nothing to do with the problem.

The purveyors of this software are in complete compliance with the licensing and trademark policies for the software they distribute; at least, those that continue to exist for any period of time are. That software is unmodified, links to the source are provided, and so on. What they are doing is aggregating the software with the real payload in a way that is quite similar to what Linux distributors do. Any attempt to use trademark policies to restrict this type of aggregation would almost certainly bite Linux distributors first.

Consider an example: a typical Linux distribution advertises the fact that it includes an office suite; it also comes with an installer that can install software that presents advertisements to the user (the music stores incorporated into media players, for example, or Amazon search results from Unity), phones home with hardware information (Fedora's Smolt) or exposes the system to external compromise (Java browser plugins). It is hard to imagine a trademark policy that could successfully block the abuses described in this article while allowing Linux distributors to continue to use the trademarked names. Free software projects are generally unwilling to adopt trademark policies of such severity.

As a result, there is little that the relevant projects can do; neither copyright nor trademark law can offer much help in this situation. That is why these projects are reduced to putting up pages trying to educate users about where the software should actually be downloaded from. The conclusion that one might draw is that trademarks are only partially useful for the purpose of protecting users. They can be used as a weapon against the distribution of overtly compromised versions of free software programs, but they cannot guarantee that any given distribution is safe to install. There is still no substitute, it seems, for taking the time to ensure that one's software comes from a reliable source.

Comments (58 posted)

Page editor: Jonathan Corbet

Security

LCA: CSP for cross-site scripting protection

At linux.conf.au 2013 in Canberra, Mozilla's François Marier presented a talk on the Content Security Policy (CSP), the browser-maker's proposed approach to thwarting cross-site scripting attacks with a framework of granular restrictions on what types of content a page can load.

We covered CSP in July 2009, just a few months after development started. Since then, the idea has been expanded, and, in November 2012, version 1.0 was declared a Candidate Recommendation by the World Wide Web Consortium (W3C).

Cross-site scripting attacks, Marier explained, usually occur when input and variables in a page are not properly escaped. An unsanitized variable such as a user input field allows an attacker to inject JavaScript or other malicious code that is loaded by a visitor's browser. Even the templating systems used by modern content management systems (CMS)—many of which auto-escape content—are not foolproof. CSP offers an additional layer of protection, argued Marier, because it is implemented as an HTTP header to be delivered by the web server and not by the CMS. Thus, for an attacker to defeat a CSP-equipped site, he or she would have to compromise the web server, which is arguably more robust than the CMS.

A CSP policy is declarative, in which a site or web application specifies the locations from which it wishes to allow scripts and other page content to load. The header declares one or more src directives, each of which specifies a list of acceptable URIs for a specific content type. For example, the most basic policy

    default-src 'self';

    img-src 'self' data ;

There is also a special reserved expression for allowing inline content (such as inline scripts or CSS), which is somewhat editorially named unsafe-inline as a reminder that permitting such inline content is a risky prospect. The reason this warrants the unsafe moniker being written into the specification itself, said Marier, is that a browser has no way to distinguish inline scripts that are written into the page at the original server from any scripts which are injected into the page content by an attacker.

The default-src directive allows site owners to set a restrictive generic policy, which is then overwritten only by whitelisting specific additional content types, he said. At his personal site, fmarier.org, he has the default-src directive set to none and only turns on additional directives for "minor stuff."

Policy makers

At the moment, CSP is available and "works really well" in Firefox and Chromium/Chrome, and is somewhat functional in Safari 6 or greater. Nevertheless, he continued, one does not need to jump directly into converting one's sites over to full CSP, which can be tricky to get right on the first try. He instead suggested a few steps to implement CSP progressively.

The first step is removing all inline scripts and styles from the site's pages. Simply moving them to external files should not affect page functionality at all, and it removes the need to worry about unsafe-inline (although, it should be noted, external scripts and stylesheets do mean longer load times). The next step is to remove all <javascript:> URIs, which, of course, may entail some rewriting. Then one can proceed to implementing a CSP policy. Marier recommended starting with a "relaxed" and permissive policy, then working one's way progressively toward a stricter policy.

For this, CSP provides a helpful report-uri directive. Unlike the other directives, report-uri does not set policy; it tells the browser to report a policy violation to the URI provided as the value. The example Marier provided is:

    report-uri http://example.com/report.cgi

Marier also recommended that interested site administrators add their CSP rules in the web server, not through their CMS or application framework, specifically to provide the extra layer of protection described above. It is also useful as a reminder that CSP is a complement to standard cross-site scripting hygiene, and not a replacement for input escaping. There are some resources out there for site maintainers to get started with policy writing, he said, such as CSPisAwesome.com, a tool for generating valid policies.

For users who are keen to get the benefits of CSP but cannot wait for their sites to get it rolling, he recommended installing a browser extension that implements CSP on the client-side. There appears to be just one at the moment: UserCSP for Firefox. This extension allows users to write policies for the various sites they visit, which Firefox then applies just as it would a CSP header originating from the server. Obviously, the user needs to be aware of the risks of "injecting" (so to speak) CSP into their browser, since applying a user-crafted policy could break the site's functionality. On the other hand, by putting the policy decision in the user's hands, the user can find his or her own balance between what breaks and what risks are left open—as is the case with other client-side security extensions like NoScript.

HTTPS, almost everywhere

As a "bonus header," Marier also discussed the HTTP Strict Transport Security (HSTS) policy framework with the time remaining in his session. HSTS, like CSP, is an HTTP header mechanism. It is designed to protect again SSL downgrade attacks, in which an HTTPS connection is stripped down to HTTP, presumably without attracting the user's attention. HSTS allows the server to declare that it will only allow browsers to connect over HTTPS. The header does not fix a permanent condition; it includes a max-age directive giving a time in seconds for which the browser should cache the HSTS setting.

Firefox has supported HSTS since Firefox 4, but as a question from the audience revealed, it comes with one hangup: the browser must successfully connect to the server over HTTPS the first time in order to get the HSTS header. Mozilla sought to alleviate the risk of attacks that exploit this by shipping Firefox 17 pre-loaded with a list of verified banking web sites that the browser should access over HTTPS the first time.

HSTS is supported in Chromium/Chrome in addition to Firefox, as well as in Opera. Mozilla cannot do much to implement security policy for other browsers—particularly the proprietary ones—so when asked what to tell users of other browsers, Marier's response was "It works in these browsers. If it doesn't work in your favorite browser ... switch browsers."

That is probably sound advice, which a lot of free software security mavens would echo. But it is interesting to see that, with both CSP and HSTS, Mozilla is pushing forward on web security from the server side as well as within the browser itself.

Comments (4 posted)

Brief items

Security quotes of the week

Comments (28 posted)

“Lucky Thirteen” attack snarfs cookies protected by SSL encryption (ars technica)

Comments (5 posted)

Garrett: Don't like Secure Boot? Don't buy a Chromebook

Comments (70 posted)

New vulnerabilities

abrt and libreport: two privilege escalation flaws

Comments (none posted)

axis: incorrect certificate validation

Comments (none posted)

chromium: multiple vulnerabilities

Comments (none posted)

coreutils: multiple vulnerabilities

Comments (none posted)

couchdb: multiple vulnerabilities

Comments (none posted)

ettercap: code execution

Comments (none posted)

freeipa: multiple vulnerabilities

Comments (1 posted)

jakarta-commons-httpclient: incorrect certificate validation

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

java: multiple unspecified vulnerabilities

Comments (none posted)

keystone: denial of service

Comments (none posted)

libupnp: multiple vulnerabilities

Comments (none posted)

libwebp: denial of service

Comments (none posted)

ndjbdns: ghost domain attack

Comments (none posted)

rhncfg: information disclosure

Comments (none posted)

samba: multiple vulnerabilities in SWAT

Comments (3 posted)

squid-cgi: denial of service

Comments (none posted)

tinymce-spellchecker: code execution

Comments (none posted)

v8: multiple vulnerabilities

Comments (none posted)

virtualbox: unspecified vulnerability

Comments (none posted)

xen: denial of service

Comments (none posted)

xorg-x11-drv-qxl: denial of service

Comments (none posted)

zim: multiple vulnerabilities

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

Stable updates: 3.0.62, 3.4.29, and 3.7.6 were released on February 3; 3.2.38 was released on February 6.

Comments (1 posted)

Quotes of the week

diff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches
--- a/Documentation/SubmittingPatches
+++ b/Documentation/SubmittingPatches
@@ -93,7 +93,9 @@ includes updates for subsystem X.  Please apply."
 
 The maintainer will thank you if you write your patch description in a
 form which can be easily pulled into Linux's source code management
-system, git, as a "commit log".  See #15, below.
+system, git, as a "commit log".  See #15, below.  If the maintainer has
+to hand-edit your patch, you owe them the beverage of their choice the
+next time you see them.

Comments (26 posted)

RAID 5/6 code merged into Btrfs

Full Story (comments: 24)

Kernel development news

User-space lockdep

Lockdep works by adding wrappers around the locking calls in the kernel. Every time a specific type of lock is taken or released, that fact is noted, along with ancillary details like whether the processor was servicing an interrupt at the time. Lockdep also notes which other locks were already held when the new lock is taken; that is the key to much of the checking that lockdep is able to perform.

To illustrate this point, imagine that two threads each need to acquire two locks, called A and B:

If one thread acquires A first while the other grabs B first, the situation might look something like this:

Now, when each thread goes for the lock it lacks, the system is in trouble:

Each thread will now wait forever for the other to release the lock it holds; the system is now deadlocked. Things may not come to this point often at all; this deadlock requires each thread to acquire its lock at exactly the wrong time. But, with computers, even highly unlikely events will come to pass sooner or later, usually at a highly inopportune time.

This situation can be avoided: if both threads adhere to a rule stating that A must always be acquired before B, this particular deadlock (called an "AB-BA deadlock" for obvious reasons) cannot happen. But, in a system with a large number of locks, it is not always clear what the rules for locking are, much less that they are consistently followed. Mistakes are easy to make. That is where lockdep comes in: by tracking the order of lock acquisition, lockdep can raise the alarm anytime it sees a thread acquire A while already holding B. No actual deadlock is required to get a "splat" (a report of a locking problem) out of lockdep, meaning that even highly unlikely deadlock situations can be found before they ruin somebody's day. There is no need to wait for that one time when the timing is exactly wrong to see that there is a problem.

Lockdep is able to detect more complicated deadlock scenarios than the one described above. It can also detect related problems, such as locks that are not interrupt-safe being acquired in interrupt context. As one might expect, running a kernel with lockdep enabled tends to slow things down considerably; it is not an option that one would enable on a production system. But enough developers test with lockdep enabled that most problems are found before they make their way into a stable kernel release. As a result, reports of deadlocks on deployed systems are now quite rare.

Kernel-based tools often do not move readily to user space; the kernel's programming environment differs markedly from a normal C environment, so kernel code can normally only be expected to run in the kernel itself. In this case, though, Sasha Levin noticed that there is not much in the lockdep subsystem that is truly kernel-specific. Lockdep collects data and builds graphs describing observed lock acquisition patterns; it is code that could be run in a non-kernel context relatively easily. So Sasha proceeded to put together a patch set creating a lockdep library that is available to programs in user space.

Lockdep does, naturally, call a number of kernel functions, so a big part of Sasha's patch set is a long list of stub implementations shorting out calls to functions like local_irq_enable() that have no meaning in user space. An abbreviated version of struct task_struct is provided to track threads in user space, and functions like print_stack_trace() are substituted with user-space equivalents (backtrace_symbols_fd() in this case). The kernel's internal (used by lockdep) locks are reimplemented using POSIX thread ("pthread") mutexes. Stub versions of the include files used by the lockdep code are provided in a special directory. And so on. Once all that is done, the lockdep code can be built directly out of the kernel tree and turned into a library.

User-space code wanting to take advantage of the lockdep library needs to start by including <liblockdep/mutex.h>, which, among other things, adds a set of wrappers around the pthread_mutex_t and pthread_rwlock_t types and the functions that work with them. A call to liblockdep_init() is required; each thread should also make a call to liblockdep_set_thread() to set up information for any problem reports. That is about all that is required; programs that are instrumented in this way will have their pthreads mutex and rwlock usage checked by lockdep.

As a proof of concept, the patch adds instrumentation to the (thread-based) perf tool contained within the kernel source tree.

One of the key aspects of Sasha's patch is that it requires no changes to the in-kernel lockdep code at all. The user-space lockdep library can be built directly out of the kernel tree. Among other things, that means that any future lockdep fixes and enhancements will automatically become available to user space with no additional effort required on the kernel developers' part.

In summary, this patch looks like a significant win for everybody involved; it is thus not surprising that opposition to its inclusion has been hard to find. There has been a call for some better documentation, explicit mention that the resulting user-space library is GPL-licensed, and a runtime toggle for lock validation (so that the library could be built into applications but not actually track locking unless requested). Such details should not be hard to fill in, though. So, with luck, user space should have access to lockdep in the near future, resulting in more reliable lock usage.

Comments (5 posted)

A simplified IDR API

Strangeness notwithstanding, the IDR API has changed little since it was documented here in 2004. One includes <linux/idr.h>, allocates an idr structure, and initializes it with idr_init(). Thereafter, allocating a new integer ID and binding it to an internal structure is a matter of calling these two functions:

    int idr_pre_get(struct idr *idp, gfp_t gfp_mask);
    int idr_get_new(struct idr *idp, void *ptr, int *id);

The call to idr_pre_get() should happen outside of atomic context; its purpose is to perform all the memory allocations necessary to ensure that the following call to idr_get_new() (which returns the newly allocated ID number and associates it with the given ptr) is able to succeed. The latter call can then happen in atomic context, a feature needed by many IDR users.

There is just one little problem with this interface, as Tejun points out in the introduction to his patch set: the call to idr_get_new() can still fail. So code using the IDR layer cannot just ask for a new ID; it must, instead, execute a loop that retries the allocation until it either succeeds or returns a failure code other than -EAGAIN. That leads to the inclusion of a lot of error-prone boilerplate code in well over 100 call sites in the kernel; the 2004 article and Tejun's patch both contain examples of what this code looks like.

Failure can happen for a number of reasons, but the mostly likely cause is tied to the fact that the memory preallocated by idr_pre_get() is a global resource. A call to idr_pre_get() simply ensures that a minimal amount of memory is available; calling it twice will not increase the amount of preallocated memory. So, if two processors simultaneously call idr_pre_get(), the amount of memory allocated will be the same as if only one processor had made that call. The first processor to call idr_get_new() may then consume all of that memory, leaving nothing for the second caller. That second caller will then be forced to drop out of atomic context and execute the retry loop — a code path that is unlikely to have been well tested by the original developer.

Tejun's response is to change the API, basing it on three new functions:

    void idr_preload(gfp_t gfp_mask);
    int idr_alloc(struct idr *idp, void *ptr, int start, int end, gfp_t gfp_mask);
    void idr_preload_end(void);

As with idr_pre_get(), the new idr_preload() function is charged with allocating the memory necessary to satisfy the next allocation request. There are some interesting differences, though. The attentive reader will note that there is no struct idr argument to idr_preload(), suggesting that the preallocated memory is not associated with any particular ID number space. It is, instead, stored in a single per-CPU array. Since this memory is allocated for the current CPU, it is not possible for any other processor to slip in and steal it — at least, not if the current thread is not preempted. For that reason, idr_preload() also disables preemption. Given that, the existence of the new idr_preload_end() function is easy to explain: it is there to re-enable preemption once the allocation has been performed.

A call to idr_alloc() will actually allocate an integer ID. It accepts upper and lower bounds for that ID to accommodate code that can only cope with a given range of numbers — code that uses the ID as an array index, for example. If need be, it will attempt to allocate memory using the given gfp_mask. Allocations will be unnecessary if idr_preload() has been called, but, with the new interface, preallocation is no longer necessary. So code that can call idr_alloc() from process context can dispense with the idr_preload() and idr_preload_end() calls altogether. Either way, the only way idr_alloc() will fail is with a hard memory allocation failure; there is no longer any need to put a loop around allocation attempts. As a result, Tejun's 62-part patch set, touching 78 files, results in the net deletion of a few hundred lines of code.

Most of the developers whose code was changed by Tejun's patch set responded with simple Acked-by lines. Eric Biederman, though, didn't like the API; he said "When reading code with idr_preload I get this deep down creepy feeling. What is this magic that is going on?" As can be seen in Tejun's response, one developer's magic is another's straightforward per-CPU technique. As of this writing, that particular discussion has not reached any sort of public resolution. Your editor would predict, though, that the simplification of this heavily-used API will be sufficiently compelling that most developers will be able to get past any resulting creepy feelings. So the IDR API may be changing in a mainline kernel in the not-too-distant future.

Comments (5 posted)

LCA: The Trinity fuzz tester

The Linux kernel developers have long been aware of the need for better testing of the kernel. That testing can take many forms, including testing for performance regressions and testing for build and boot regressions. As the term suggests, regression testing is concerned with detecting cases where a new kernel version causes problems in code or features that already existed in previous versions of the kernel. Of course, each new kernel release also adds new features. The Trinity fuzz tester is a tool that aims to improve testing of one class of new (and existing) features: the system call interfaces that the kernel presents to user space.

Insufficient testing of new user-space interfaces is a long-standing issue in kernel development. Historically, it has been quite common that significant bugs are found in new interfaces only a considerable time after those interfaces appear in a stable kernel—examples include epoll_ctl(), kill(), signalfd(), and utimensat(). The problem is that, typically, a new interface is tested by only one person (the developer of the feature) or at most a handful of people who have a close interest in the interface. A common problem that occurs when developers write their own tests is a bias toward tests which confirm that expected inputs produce expected results. Often, of course, bugs are found when software is used in unexpected ways that test little-used code paths.

Fuzz testing is a technique that aims to reverse this testing bias. The general idea is to provide unexpected inputs to the software being tested, in the form of random (or semi-random) values. Fuzz testing has two obvious benefits. First, employing unexpected inputs mean that rarely used code paths are tested. Second, the generation of random inputs and the tests themselves can be fully automated, so that a large number of tests can be quickly performed.

History

Fuzz testing has a history that stretches back to at least the 1980s, when fuzz testers were used to test command-line utilities. The history of system call fuzz testing is nearly as long. During his talk at linux.conf.au 2013 [ogv video, mp4 video], Dave Jones, the developer of Trinity, noted that the earliest system call fuzz tester that he had heard of was Tsys, which was created around 1991 for System V Release 4. Another early example was a fuzz tester [postscript] developed at the University of Wisconsin in the mid-1990s that was run against a variety of kernels, including Linux.

Tsys was an example of a "naïve" fuzz tester: it simply generated random bit patterns, placed them in appropriate registers, and then executed a system call. About a decade later, the kg_crashme tool was developed to perform fuzz testing on Linux. Like Tsys, kg_crashme was a naïve fuzz tester.

Naïve fuzz testers are capable of finding some kernel bugs, but the use of purely random inputs greatly limits their efficacy. To see why this is, we can take the example of the madvise() system call, which allows a process to advise the kernel about how it expects to use a region of memory. This system call has the following prototype:

    int madvise(void *addr, size_t length, int advice);

madvise() places certain constraints on its arguments: addr must be a page-aligned memory address, length must be non-negative, and advice must be one of a limited set of small integer values. When any of these constraints is violated, madvise() fails with the error EINVAL. Many other system calls impose analogous checks on their arguments.

A naïve fuzz tester that simply passes random bit patterns to the arguments of madvise() will, almost always, perform uninteresting tests that fail with the (expected) error EINVAL. As well as wasting time, such naïve testing reduces the chances of generating a more interesting test input that reveals an unexpected error.

Thus, a few projects started in the mid-2000s with the aim of bringing more sophistication to the fuzz-testing process. One of these projects, Dave's scrashme, was started in 2006. Work on that project languished for a few years, and only picked up momentum starting in late 2010, when Dave began to devote significantly more time to its development. In December 2010, scrashme was renamed Trinity. At around the same time, another quite similar tool, iknowthis, was also developed at Google.

Intelligent fuzz testing

Trinity performs intelligent fuzz testing by incorporating specific knowledge about each system call that is tested. The idea is to reduce the time spent running "useless" tests, thereby reaching deeper into the tested code and increasing the chances of testing a more interesting case that may result in an unexpected error. Thus, for example, rather than passing random values to the advice argument of madvise(), Trinity will pass one of the values expected for that argument.

Likewise, rather than passing random bit patterns to address arguments, Trinity will restrict the bit pattern so that, much of the time, the supplied address is page aligned. However, some system calls that accept address arguments don't require memory aligned addresses. Thus, when generating a random address for testing, Trinity will also favor the creation of "interesting" addresses, for example, an address that is off a page boundary by the value of sizeof(char) or sizeof(long). Addresses such as these are likely candidates for "off by one" errors in the kernel code.

In addition, many system calls that expect a memory address require that address to point to memory that is actually mapped. If there is no mapping at the given address, then these system calls fail (the typical error is ENOMEM or EFAULT). Of course, in the large address space available on modern 64-bit architectures, most of the address space is unmapped, so that even if a fuzz tester always generated page-aligned addresses, most of the resulting tests would be wasted on producing the same uninteresting error. Thus, when supplying a memory address to a system call, Trinity will favor addresses for existing mappings. Again, in the interests of triggering unexpected errors, Trinity will pass the addresses of "interesting" mappings, for example, the address of a page containing all zeros or all ones, or the starting address at which the kernel is mapped.

In order to bring intelligence to its tests, Trinity must have some understanding of the arguments for each system call. This is accomplished by defining structures that annotate each system call. For example, the annotation file for madvise() includes the following lines:

    struct syscall syscall_madvise = {
        .name = "madvise",
        .num_args = 3,
        .arg1name = "start",
        .arg1type = ARG_NON_NULL_ADDRESS,
        .arg2name = "len_in",
        .arg2type = ARG_LEN,
        .arg3name = "advice",
        .arg3type = ARG_OP,
        .arg3list = {
            .num = 12,
            .values = { MADV_NORMAL, MADV_RANDOM, MADV_SEQUENTIAL, MADV_WILLNEED,
                    MADV_DONTNEED, MADV_REMOVE, MADV_DONTFORK, MADV_DOFORK,
                    MADV_MERGEABLE, MADV_UNMERGEABLE, MADV_HUGEPAGE, MADV_NOHUGEPAGE },
        },
        ...
    }; 

This annotation describes the names and types of each of the three arguments that the system call accepts. For example, the first argument is annotated as ARG_NON_NULL_ADDRESS, meaning that Trinity should provide an intelligently selected, semi-random, nonzero address for this argument. The last argument is annotated as ARG_OP, meaning that Trinity should randomly select one of the values in the corresponding list (the MADV_* values above).

The second madvise() argument is annotated ARG_LEN, meaning that it is the length of a memory buffer. Again, rather than passing purely random values to such arguments, Trinity attempts to generate "interesting" numbers that are more likely to trigger errors—for example, a value whose least significant bits are 0xfff might find an off-by-one error in the logic of some system call.

Trinity also understands a range of other annotations, including ARG_RANDOM_INT, ARG_ADDRESS (an address that can be zero), ARG_PID (a process ID), ARG_LIST (for bit masks composed by logically ORing values randomly selected from a specified list), ARG_PATHNAME, and ARG_IOV (a struct iovec of the kind passed to system calls such as readv()). In each case, Trinity uses the annotation to generate a better-than-random test value that is more likely to trigger an unexpected error. Another interesting annotation is ARG_FD, which causes Trinity to pass an open file descriptor to the tested system call. For this purpose, Trinity opens a variety of file descriptors, including descriptors for pipes, network sockets, and files in locations such as /dev, /proc, and /sys. The open file descriptors are randomly passed to system calls that expect descriptors. By now, it might start to become clear that you don't want to run Trinity on a system that has the only copy of your family photo albums.

In addition to annotations, each system call can optionally have a sanitise routine (Dave's code employs the British spelling) that performs further fine-tuning of the arguments for the system call. The sanitise routine can be used to construct arguments that require special values (e.g., structures) or to correctly initialize the values in arguments that are interdependent. It can also be used to ensure that an argument has a value that won't cause an expected error. For example, the sanitise routine for the madvise() system call is as follows:

    static void sanitise_madvise(int childno)
    {
        shm->a2[childno] = rand() % page_size;
    } 

This ensures that the second (length) argument given to madvise() will be no larger than the page size, preventing the ENOMEM error that would commonly result when a large length value causes madvise() to touch an unmapped area of memory. Obviously, this means that the tests will never exercise the case where madvise() is applied to regions larger than one page. This particular sanitize routine could be improved by sometimes allowing length values that are larger than the page size.

Running trinity

The Trinity home page has links to the Git repository as well as to the latest stable release (Trinity 1.1, which was released in January 2013). Compilation from source is straightforward; then Trinity can be invoked with a command line as simple as:

     $ ./trinity

With no arguments, the program repeatedly tests randomly chosen system calls. It is also possible to test selected system calls using one or more instances of the -c command-line option. This can be especially useful when testing new system calls. Thus, for example, one could test just the madvise() system call using the following command:

     $ ./trinity -c madvise

In order to perform its work, the trinity program creates a number of processes, as shown in the following diagram:

The main process performs various initializations (e.g., opening the file descriptors and creating the memory mappings used for testing) and then kicks off a number (default: four) of child processes that perform the system call tests. A shared memory region (created by the initial trinity process) is used to record various pieces of global information, such as open file descriptor numbers, total system calls performed, and number of system calls that succeeded and failed. The shared memory region also records various information about each of the child processes, including the PID, and the system call number and arguments for the system call that is currently being executed as well as the system call that was previously executed.

The watchdog process ensures that the test system is still working correctly. It checks that the children are progressing (they may be blocked in a system call), and kills them if they are not; when the main process detects that one of its children has terminated (because the watchdog killed it, or for some other reason), it starts a new child process to replace it. The watchdog also monitors the integrity of the memory region that is shared between the processes, in case some operation performed by one of the children has corrupted the region.

Each of the child processes writes to a separate log file, recording the system calls that it performs and the return values of those system calls. The file is synced just before each system call is performed, so that if the system panics, it should be possible to determine the cause of the panic by looking at the last recorded system call in each of the log files. The log file contains lines such as the following, which show the PID of child process, a sequential test number, and the system call arguments and result:

    [17913] [0] mmap(addr=0, len=4096, prot=4, flags=0x40031, fd=-1, off=0) = -1 (Invalid argument)
    [17913] [1] mmap(addr=0, len=4096, prot=1, flags=0x25821, fd=-1, off=0x80000000) = -541937664
    [17913] [2] madvise(start=0x7f59dff7b000, len_in=3505, advice=10) = 0
    ...
    [17913] [6] mmap(addr=0, len=4096, prot=12, flags=0x20031, fd=-1, off=0) = -1 (Permission denied)
    ...
    [17913] [21] mmap(addr=0, len=4096, prot=8, flags=0x5001, fd=181, off=0) = -1 (No such device)

Trinity can be used in a number of ways. One possibility is simply to leave it running until it triggers a kernel panic and then look at the child logs and the system log in order to discover the cause of the panic. Dave has sometimes left systems running for hours or days in order to discover such failures. New system calls can be exercised using the -c command-line option described above. Another possible use is to discover unexpected (or undocumented) failure modes of existing system calls: suitable scripting on the log files can be used to obtain summaries of the various failures of a particular system call.

Yet another way of using the trinity program is with the -V (victim files) option. This option takes a directory argument: the program will randomly open files in that directory and pass the resulting file descriptors to system calls. This can be useful for discovering failure modes in a particular filesystem type. For example, specifying an NFS mount point as the directory argument would exercise NFS. The -V flag can also be used to perform a limited kind of testing of user-space programs. During his linux.conf.au presentation, Dave demonstrated the use of the following command:

    $ ./trinity -V /bin -c execve

This command has the effect of executing random programs in /bin with random string arguments. Looking at the system log revealed a large number of programs that crashed with a segmentation fault when given unexpected arguments.

Results

Trinity has been rather successful at finding bugs. Dave reports that he has himself found more than 150 bugs in 2012, and many more were found by other people who were using Trinity. Trinity usually finds bugs in new code quite quickly. It tends to find the same bugs repeatedly, so that in order to find other bugs, it is probably necessary to fix the already discovered bugs first.

Interestingly, Trinity has found bugs not just in system call code. Bugs have been discovered in many other parts of the kernel, including the networking stack, virtual memory code, and drivers. Trinity has found many error-path memory leaks and cases where system call error paths failed to release kernel locks. In addition, it has discovered a number of pieces of kernel code that had poor test coverage or indeed no testing at all. The oldest bug that Trinity has so far found dated back to 1996.

Limitations and future work

Although Trinity is already quite an effective tool for finding bugs, there is scope for a lot more work to make it even better. An ongoing task is to add support for new system calls and new system call flags as they are added to the kernel. Only about ten percent of system calls currently have sanitise routines. Probably many other system calls could do with sanitise routines so that tests would get deeper into the code of those system calls without triggering the same common and expected errors. Trinity supports many network protocols, but that support could be further improved and there are other networking protocols for which support could be added.

Some system calls are annotated with an AVOID_SYSCALL flag, which tells Trinity to avoid testing that system call. (The --list option causes Trinity to display a list of the system calls that it knows about, and indicates those system calls that are annotated with AVOID_SYSCALL.) In some cases, a system call is avoided because it is uninteresting to test—for example, system calls such as fork() have no arguments to fuzz and exit() would simply terminate the testing process. Some other system calls would interfere with the operation of Trinity itself—examples include close(), which would randomly close test file descriptors used by child processes, and nanosleep(), which might put a child process to sleep for a long time.

However, there are other system calls such as ptrace() and munmap() that are currently marked with AVOID_SYSCALL, but which probably could be candidates for testing by adding more intelligence to Trinity. For example, munmap() is avoided because it can easily unmap mappings that are needed for the child to execute. However, if Trinity added some bookkeeping code that recorded better information about the test mappings that it creates, then (only) those mappings could be supplied in tests of munmap(), without interfering with other mappings needed by the child processes.

Currently, Trinity randomly invokes system calls. Real programs demonstrate common patterns for making system calls—for example, opening, reading, and closing a file. Dave would like to add test support for these sorts of commonly occurring patterns.

An area where Trinity currently provides poor coverage is the multiplexing ioctl() system call, "the worst interface known to man". The problem is that ioctl() is really a mass of system calls masquerading as a single API. The first argument is a file descriptor referring to a device or another file type, the second argument is a request type that depends on the type of file or device referred to by the first argument, and the data type of the third argument depends on the request type. To achieve good test support for ioctl() would require annotating each of the request types to ensure that it is associated with the right type of file descriptor and the right data type for the third argument. There is an almost limitless supply of work here, since there are hundreds of request types; thus, in the first instance, this work would probably be limited to supporting a subset of more interesting request types.

There are a number of other improvements that Dave would like to see in Trinity; the source code tarball contains a lengthy TODO file. Among these improvements are better support for "destructors" in the system call handling code, so that Trinity does not leak memory, and support for invoking (some) system calls as root. More generally, Trinity's ability to find further kernel bugs is virtually limitless: it simply requires adding ever more intelligence to each of its tests.

Comments (7 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.8-rc6 . (February 1, 2013)

• Greg KH: Linux 3.7.6 . (February 4, 2013)

• Thomas Gleixner: 3.6.11-rt26 . (February 4, 2013)

• Greg KH: Linux 3.4.29 . (February 4, 2013)

• Steven Rostedt: 3.4.29-rt41 . (February 6, 2013)

• Steven Rostedt: 3.4.28-rt40 . (February 4, 2013)

• Ben Hutchings: Linux 3.2.38 . (February 6, 2013)

• Steven Rostedt: 3.2.38-rt56 . (February 6, 2013)

• Greg KH: Linux 3.0.62 . (February 4, 2013)

• Steven Rostedt: 3.0.62-rt87 . (February 6, 2013)

• Steven Rostedt: 3.0.61-rt86 . (February 4, 2013)

Core kernel code

• Thomas Gleixner: CPU hotplug rework - episode I . (February 1, 2013)

• Lai Jiangshan: workqueue: enhance locking and record global worker id for work data . (February 4, 2013)

• Tejun Heo: idr: implement idr_alloc() and convert existing users . (February 4, 2013)

• Viresh Kumar: CPUFreq: Implement per policy instances of governors . (February 4, 2013)

• Frederic Weisbecker: printk: Support for full dynticks mode . (February 6, 2013)

• Frederic Weisbecker: 3.8-rc6-nohz4 . (February 6, 2013)

Development tools

• Jiri Olsa: perf tool: Add PERF_SAMPLE_READ sample read support . (February 4, 2013)

• Stephane Eranian: perf: add new uncore command . (February 4, 2013)

• Oleg Nesterov: uprobes/perf: pre-filtering . (February 5, 2013)

Device drivers

• Takashi Iwai: firmware: Make user-mode helper optional (v5) . (February 1, 2013)

• Simon Wood: USB: HID: SRW-S1 Gaming Wheel Driver . (February 1, 2013)

• Philipp Zabel: Add generic driver for on-chip SRAM . (February 4, 2013)

• Neil Horman: i2c: Adding support for Intel iSMT SMBus 2.0 host controller . (February 5, 2013)

• Hongbo Zhang: Add ST-Ericsson AB8500 HWMON driver . (February 6, 2013)

Documentation

• Michael Kerrisk (man-pages): man-pages-3.46 is released . (February 4, 2013)

Filesystems and block I/O

• Jan Kara: Mapping range lock . (February 4, 2013)

• Darrick J. Wong: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel . (February 4, 2013)

• J. Bruce Fields: Implement NFSv4 delegations, take 6 . (February 4, 2013)

• Zheng Liu: ext4: extent status tree (step2) . (February 4, 2013)

• Aaron Lu: block layer runtime pm . (February 5, 2013)

Networking

• Vlad Yasevich: Add basic VLAN support to bridges . (February 5, 2013)

• Andy King: VM Sockets for Linux upstreaming . (February 5, 2013)

Architecture-specific

• Andi Kleen: Basic perf PMU support for Haswell v2 . (February 4, 2013)

• dirk.brandewie@gmail.com: Add P state driver for Intel Core Processors . (February 4, 2013)

• Rik van Riel: [PATCH -v5 0/5] x86,smp: make ticket spinlock proportional backoff w/ auto tuning . (February 6, 2013)

• Matt Porter: DMA Engine support for AM33XX . (February 4, 2013)

• Peter De Schrijver: Tegra114 clockframework . (February 4, 2013)

• Joonsoo Kim: introduce static_vm for ARM-specific static mapped area . (February 5, 2013)

• Nicolas Pitre: multi-cluster power management . (February 6, 2013)

Security-related

• Dmitry Kasatkin: initramfs with digital signature protection . (February 5, 2013)

• Paolo Bonzini: Corrections and customization of the SG_IO command whitelist (CVE-2012-4542) . (February 6, 2013)

Miscellaneous

• Sasha Levin: liblock: userspace lockdep . (February 4, 2013)

Page editor: Jonathan Corbet

Distributions

Systemd lightweight containers

Linux containers, which are implemented using kernel namespaces and control groups, allow processes to operate in an isolated manner, so that the interactions with other processes and kernel services are limited. That makes containers attractive for a variety of tasks, including many that might have once been done using chroot(). As namespace support in the kernel matures, tools to set up and use containers are becoming more prevalent—and easier to use. A feature proposed for Fedora 19 will make use of systemd to create and manage containers.

At first blush, systemd does not really seem like a container-management tool. In fact, detractors might see that as feature creep. But systemd already has infrastructure to spawn containers in the form of the systemd-nspawn command. In addition, creating a new process ID (PID) namespace means that an init program (i.e. PID 1) is needed, which is, of course, the role that systemd normally fills.

Beyond that, systemd is designed around the idea of "socket activation", so that services can be started when the first connection is made to them. That idea can be applied to containers, so that a new container gets started when a connection is made to a certain port. This "container activation" feature is reminiscent of a similar idea in the SELinux-based secure containers feature that was added to Fedora 18. Unlike the secure containers, though, those created with systemd-nspawn are not primarily intended for security. With proper care and feeding, however, they can provide another layer of a "defense in depth".

One goal of the "systemd lightweight containers" feature is to make it easy to run an unmodified Fedora 19 inside the containers created by systemd-nspawn. But it isn't just Fedora that could run in those containers, Debian is another candidate; other distributions are possible too. By installing a minimal system into a directory somewhere—using yum or debootstrap for example—and then pointing systemd-nspawn at it, a usable version of the distribution can be run. Users can log into it from the "console", set up a service or services to run inside of it, and so on. Rudimentary directions on setting that up are part of the feature proposal.

By default, systemd-nspawn sets up separate PID, mount, IPC (inter-process communication), and UTS (host and domain name) namespaces, and executes the given command inside of them. If invoked with the -b option, it will search for an init binary to execute, and pass any arguments to that program. This command:

    systemd-nspawn -bD /srv/rawhide 3

    systemd-nspawn --capability=cap_audit_write,cap_audit_control -bD /srv/rawhide 3

With a simple unit file, the container can be turned into a service that can be started, stopped, and monitored with systemctl. Fans of the systemd journal can use the -j option of systemd-nspawn to effectively export the container's journal to the host. A "journalctl -m" command on the host will then show merged journal entries from the host and any containers.

Multiple containers can be started using the same directory and they won't be able to see each other. Changes to the filesystem will be immediately visible in any container using it, but processes in one container cannot interact with processes in another, nor with the processes on the host.

Using the techniques described in "systemd for Administrators, Part XX", these containers can easily be made socket activated. An incoming connection on a particular host port would spawn the container, which would have unit files that recognized the incoming connection to start the right service on the inside. Users will likely also want to set up sshd inside the container to run on a different port (the host presumably already uses 22) for ease of accessing the container.

There is also an option to run the container in a separate network namespace (--private-network), which essentially turns off networking for the container. Only the loopback interface is available to the container, so no network connections of any kind can be made, though it could still read and write using socket file descriptors that were passed to it. That would be a way to isolate an internet-facing service, for example.

There are a number of different use cases for the feature, but it also looks like something that will be built upon in the future. Allowing for tightened security, possibly using user ID namespaces, would be one possibility. Adding support for network namespaces that have more than just the loopback interface could be interesting as well. Since FESCo approved the feature for Fedora 19 at its February 6 meeting, more users of the feature can be expected. That means that more use cases will be found, which seems likely to lead to expanded functionality, but it's a useful feature as it stands.

Comments (10 posted)

Brief items

Distribution quote of the week

Comments (none posted)

Fedora 18 for ARM released

Full Story (comments: 4)

Linaro 13.01 released

Comments (none posted)

Distribution News

Red Hat Enterprise Linux

Red Hat Enterprise Linux 3 - 1-Year End Of Support Notice

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (February 4)
• DistroWatch Weekly, Issue 493 (February 4)
• Ubuntu Weekly Newsletter, Issue 302 (February 3)

Comments (none posted)

5 Ubuntu alternatives worth checking out (ExtremeTech)

Comments (1 posted)

Page editor: Rebecca Sobol

Development

The GNOME developer experience hackfest

For three days just prior to FOSDEM, GNOME contributors gathered in Brussels to discuss and work on the "developer experience". The developer experience hackfest was well attended, attracting people from all over the world. It was also quite productive, judging from various blog postings about the meeting. The headline decision—making JavaScript the preferred language for developing GNOME applications—was certainly noteworthy, but there were other plans made as well.

The JavaScript choice makes sense on a number of levels, but it also has clearly caused a fair amount of disdain to be directed at the GNOME project. Any language chosen would have likely resulted in much the same level of discontent. As Travis Reitter noted in his announcement: "The important thing was that we had to make a decision." That helps in having a single answer to the "how do I develop a GNOME application?" question, which is part of the justification for the move:

None of the existing supported languages are being deprecated because of the decision; it's really just a matter of focus for the project. While there have been a fair number of complaints heard about the choice, here at LWN and elsewhere, it's not clear how many GNOME contributors are among those disgruntled. On the other hand, some GNOME developers who might seem like candidates for grumbling are on-board with the change. John "J5" Palmieri is a big Python fan who worked on GObject Introspection for that language, but is pleased with the decision:

Another interesting discussion took place in the "application distribution and sandboxing" subgroup, as reported by Alexander Larsson. The group considered two different pieces of the application infrastructure puzzle: how to deploy (i.e. create, install, and run) application bundles and how to protect the rest of the user's session from application misbehavior.

For deployment, GNOME is considering having applications declare their library dependencies, in either a coarse or fine-grained manner, and then installing and running them in an environment that guarantees those dependencies regardless of what the system itself is running. That would be done using containers that provide a private view showing the platform dependencies that the application said it requires. As Larsson describes, there are some benefits to that approach:

Beyond that, the idea of application isolation can be extended to provide a fully isolated sandbox for untrusted applications. It would require D-Bus routing in the kernel, which is proving to be a hard sell, but "hopefully this will work out this time", Larsson said. There is interest in adding a facility like the Android Intents system to allow sandboxed applications to communicate. Since that kind of communication implies a security domain transition, the group came up with the name "Portals" for this new feature. The discussion continues post-hackfest, he said, "hopefully we can start [to] implement parts of this soon".

The JavaScript decision was also helpful for the documentation subgroup, as Meg Ford and Allan Day reported. Those two, along with several others, started reworking the GNOME developer web site, including adding new tutorials for first-time application developers. Significant redesign of the site with several new features was discussed. Ford described some of those:

It would seem that the GNOME project has chosen the direction for its "developer story". That story is an important piece of the puzzle when trying to attract application developers to a platform. One could argue about the choices made, but it is significant (and important as Reitter said) that a choices was made. JavaScript is certainly popular, and many developers are already comfortable using the language. That may prove helpful in attracting some of those developers, but having one place to point when someone asks about developing a GNOME application is likely to be more important still.

Comments (1 posted)

Brief items

Quotes of the week

Comments (23 posted)

KDE 4.10 Release of Plasma Workspaces, Applications and Development Platform

Full Story (comments: none)

Barman 1.2.0 released

Barman, the Backup and Recovery Manager for PostgreSQL, version 1.2.0 has been released. This release introduces "automated support for retention policies based on redundancy of periodical backups or recovery window." Such policies are "integrated by a safety mechanism that allows administrators to specify a minimum number of periodical backups that must exist at any time for a server."

Full Story (comments: none)

Topaz: a new Ruby implementation

Comments (13 posted)

MySQL Community Server 5.6.10 available

Version 5.6.10 of the MySQL database server has been released. Despite the x.x.10 version number, this is the first stable release of the 5.6 series to officially be declared "general availability" (GA). The set of changes included is extensive; the project has a full changelog available on its site.

Full Story (comments: none)

Firefox 18.0.2

Comments (3 posted)

Krita 2.6 released

Version 2.6 of Krita, the KDE-flavored painting and natural-media simulation application, has been released. This version includes improvements to OpenRaster support, Photoshop (.PSD) file export, and new support for the OpenColorIO color-management system used in some professional video workflows.

Full Story (comments: none)

Newsletters and articles

Development newsletters from the past week

• Caml Weekly News (February 5)
• What's cooking in git.git (February 1)
• What's cooking in git.git (February 4)
• What's cooking in git.git (February 6)
• Haskell Weekly News (January 30)
• Openstack Community Weekly Newsletter (February 1)
• Perl Weekly (February 4)
• Ruby Weekly (January 31)

Comments (none posted)

Reitter: Answering the question: "How do I develop an app for GNOME?"

Comments (159 posted)

Sourcefabric: Open source search with Solr leads Newscoop development

Sourcefabric, creator of open source journalism applications such as the Airtime radio station management system, has released a newsletter that rounds up recent developments in its software frameworks. This edition includes the addition of Apache Solr searching in the Newscoop CMS and Booktype's appearance at the Tools of Change conference.

Full Story (comments: none)

Firefox and Chrome have a chat

Comments (6 posted)

Page editor: Nathan Willis

Announcements

Brief items

25 Women in 10 Free Software Organizations for GNOME's Outreach Program for Women

Comments (136 posted)

The FSF licensing team's 2012 report

Comments (none posted)

linux.conf.au 2013 videos online

Comments (16 posted)

FOSDEM videos

Comments (none posted)

Articles of interest

Free Software Supporter -- Issue 58, January 2013

Full Story (comments: none)

Final FOSDEM 2013 speaker interviews

Comments (none posted)

Guide to Open Source and Linux-Compatible 3D Printers (Linux.com)

Comments (none posted)

A pair of UEFI updates

Meanwhile, James Bottomley has put up a report on his work with the Linux Foundation's secure boot loader. "The upshot of all of this is you can now use Pre-BootLoader with Gummiboot (as demoed at LCA2013). To boot, you have to add two hashes: one for Gummiboot itself and one for the kernel you’re booting, but actually this is a good thing because now you have a single security policy controlling all of your boot sequence. Gummiboot itself has also been patched to recognise a failure due to secure boot and pop up a helpful message telling you which hash to enrol."

Comments (14 posted)

Calls for Presentations

LAC 2013: the Linux Audio Conference - Deadline Extension

Full Story (comments: none)

1st Call For Papers, 19th Annual Tcl/Tk Conference 2013

Full Story (comments: none)

Upcoming Events

SCALE 11X: Birds of a Feather flocking at SCALE

Full Story (comments: none)

LPI participates in CeBIT 2013 Poland Partner Events

Full Story (comments: none)

LibrePlanet 2013: Commit Change

Full Story (comments: none)

IPv6 Summit

Full Story (comments: none)

Events: February 7, 2013 to April 8, 2013

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol