LWN.net Weekly Edition for July 14, 2011

Copyright, copyleft, and culture

Nina Paley has certainly stirred things up with her recent "rantifesto" on free culture and free software. It has spawned numerous responses on various blogs, both from supporters and those who disagree with her contention that the Free Software Foundation (FSF) is being hypocritical in its licensing of its web pages and other non-software works. For some people it is a bit galling to see an organization that is set up to ensure the right to create and distribute derivative works (subject to some conditions, of course) of software, be so steadfast in its refusal to apply those same freedoms to text and other works.

Paley's main example is quite cogent. In her essay, she restates the famous "four freedoms" from the FSF's free software definition and applies them to free culture. In doing so, she has arguably created a derivative work of the FSF's definition, which is not compatible with the "verbatim copying license" that governs text on the GNU web site. Though the FSF web site (unlike the GNU web site) is covered by the Creative Commons Attribution-NoDerivs (CC BY-ND) license, and Paley confuses the two a bit, either of the those two licenses would restrict derivative works. It's a little hard to argue, however, that what Paley has done is not in keeping with the spirit of free software even though it has been applied to text that is specifically licensed to restrict that kind of activity.

In the unlikely event it ever becomes an issue, an argument could certainly be made that Paley exercised her "fair use" rights when creating the four freedoms of free culture. But fair use is a weak and uncertain defense—at best—for anyone wishing to make use of a restrictively-licensed work. Fair use is jurisdiction-dependent and, even in places like the US where it is an established precedent, judges can interpret it in wildly different ways. There is also the small matter of the cost to defend against a claim of copyright violation even if it seems to be fair use. In some ways, it's reminiscent of the uphill battle faced by a software company accused of a patent violation for a patent with "obvious" prior art—it's extremely costly to defend against the suit without any real assurance of getting a sensible ruling. A license that explicitly allows derivative works provides much more certainty.

The argument that Paley is making is not that all works should be licensed freely—though, of course, that is the argument that the FSF makes for software—but that the champion of the copyleft movement should more liberally license its non-software works. The FSF has already run afoul of other free software advocates (e.g. Debian) for documentation licensed under its GNU Free Documentation License—at least for documentation that has "invariant" sections which are required to be carried along with any derivative work. Cynical observers have pointed out that the main reason that the invariant sections exist is so that the GNU Manifesto can be more widely spread. It is difficult to argue that the invariant sections make the documentation more free, however, and they certainly make it difficult to create derivative works in the same spirit as is done with software.

As Paley points out, the creator of a work (be it software, text, photographs, video, fine art, etc.) cannot know the kinds of things that a downstream user might create with a suitably licensed work. This is an argument the free software community should be very familiar with, and would seem to be at the heart of what free software is. All Paley is trying to do is to broaden that freedom to other works in a free culture movement that seeks to remove the restrictions on at least some of the cultural works that are created by our society. Much like the FSF takes projects and other organizations to task over their "anti-freedom" moves with respect to software, Paley is essentially doing the same to the FSF. She is asking the FSF (and the much larger FOSS community) to join forces in helping to foster free culture.

Make no mistake, free culture is clearly under serious attack from the large "content" industries. Fair use is well-nigh impossible to actually exercise with organizations like the RIAA and MPAA along with media giants like Disney trying to maximize copyright in all dimensions. Without a major sea change, nothing that is under copyright today will ever come out from under it and fall into the public domain. Legislators will keep extending copyright terms so that Disney—whose success has largely been based on remixing public domain works—never loses the copyright on its iconic mouse. Without armies of expensive lawyers and lobbyists, the copyright situation is unlikely to change, but individuals certainly can participate in free culture to create a separate commons that is available to all.

Are there differences between software and other works? Of course there are, but they aren't such huge differences that the same principles cannot apply to all. In fact, a perfect example is copyright itself, which applies "equally" to a wide variety of different forms of expression. Another example is Paley's restatement of the four freedoms—it could be adopted by the FSF for software without any real loss. There is no huge chasm between technical and cultural works as some have claimed, and both kinds of works embody the opinion of their creator in one form or another.

Another part of Paley's argument should seem rather familiar to our community as well. She bemoans the dilution—perhaps distortion—of the "free culture" term by including things that are licensed in ways that aren't truly free. We have struggled with the same basic problem, most recently in the "open source" vs. "open core" debates, but, more generally, in trying to agree on what constitutes "free" (or "open") in the context of software.

The proliferation of the non-commercial (NC) versions of CC licenses on supposedly free culture works is one of the problems that Paley highlights. As she rightly points out, these are essentially "field of use" restrictions that wouldn't be accepted for free software or open source licenses. In addition, though Paley doesn't specifically mention it, NC restrictions are a murky quagmire that just make it difficult for potential users to know what's acceptable and what isn't. Can you use an NC-licensed photo on your blog if you also run Google ads to try to offset the hosting costs? Or on a commercial blog service that runs its own ads? Those are questions for lawyers, which is reason enough make folks leery of NC whether they are inclined toward free culture or not—it's simpler to just use regular copyright and decide on a case-by-case basis whether the use is suitably non-commercial.

The no-derivatives (ND) variants of CC licenses have their own set of problems as well. A strict interpretation would not allow a photo to be cropped, resized, or have text placed on it, for example. An ND text couldn't have typos fixed or an introduction added either, which seriously reduces the ability to use it in any reasonable way. NC and/or ND restrictions may be just what the creator intends, but they don't really contribute to free culture in any sensible way.

In the end, there are going to be plenty of non-free works, both software and otherwise. Whoever creates a work gets to choose the license it's available under, and no one has argued otherwise. Paley is just trying to make a fairly reasonable argument that free software and free culture should be allies, and that it's disappointing to see the FSF make fairly arbitrary distinctions between types of expression. The free culture movement is still in its infancy, more or less where free software was 20 years ago or so. If free culture can make similar inroads against the content behemoths that free software has made in the software world, we will all be better off for it. And that, in a nutshell, is what Paley is advocating.

Comments (27 posted)

VLC and unwelcome redistributors

VLC is a well-respected media player known for its multi-platform support and its ability to play almost anything the user can come up with. If one searches for VLC in Google, the project's site comes up at the top of the list. But it is likely to be accompanied by one or more paid ads from sites offering free downloads of VLC binaries. This might strike one as an interesting situation; the people behind these sites are willing to pay money for ads so that they can offer up their bandwidth for free downloads. Either their enthusiasm for VLC is so extreme that they are willing to put considerable resources into encouraging its distribution, or something else is going on.

Unsurprisingly, it seems that something else is going on. A recent blog posting by VLC developer Ludovic Fauvet names a number of these sites and complains about the business they are in:

In the best case, these distributors commercialize the software for their own objectives. In progressively worse cases, they break the program, add antifeatures, or turn it into overt spyware or malware. It is not surprising that the VLC developers are not pleased by this kind of activity. One user saying "I downloaded VLC and it infected my system" can be enough to deter many others from trying the real thing.

VLC is free software, released under the GPL. As long as these redistributors comply with the requirements of the GPL, they are within the rights that the VLC project gave to them - even if they may or may not be violating various laws regarding the distribution of malicious software. As it happens, readers will no doubt be shocked to learn that many of these companies fail to take the GPL's source availability requirements seriously. It's almost as if they actively didn't want others to see what they were doing to the program. Failure to comply with the GPL gives the VLC project one tool which can be used to shut some of these operators down; the project has evidently made use of this power at times.

What happens if a distributor corrupts VLC in some way, but complies properly with the licensing requirements? The result can easily be unhappy users and damage to the VLC project's "brand." One could say that the VLC code base reflects the developers' opinions on how a media player should work; making user-hostile changes to that code can cause those developers to be blamed for opinions which they would never have thought to express. They would like to prevent that from happening, but, it seems, the inability to restrict modified versions ties their hands.

This problem is exactly why the Mozilla project maintains such firm and uncompromising control over the use of the "Firefox" trademark. A malware version of Firefox could do untold damage to its users and, consequently, to the world's view of Firefox in general; it is not hard to imagine Firefox developers lying awake at night worrying about this scenario. A fiercely-defended trademark with a tight policy on acceptable uses gives Mozilla a means to quickly shut down Fakefoxes which behave in undesirable ways.

The trademark approach has its own problems; among other things, it makes it harder for distributors to support Firefox, especially after official support for a given release ends. Strong trademark policies often seem to run counter to the spirit of free software and free expression as well; you cannot, for example, set up a community site at FedoraFans.org (say) without encountering the Fedora trademark rules. Despite these worries, the intersection of trademarks and free software has worked reasonably well most of the time.

The VLC project is evidently working on its trademark policies, but VLC has a problem common to many development projects: it is not endowed with the sort of legal budget that Mozilla has. Enforcing trademarks in any non-trivial way requires lawyers; trademarks which are not enforced tend to go away. Organizations like the Software Freedom Law Center can help in the defense of trademarks, but resources for pro-bono work will be limited. So trademarks, even if handled well, are not a complete solution to this problem either.

That leaves the bulk of free software projects without much in the way of defense against those who would misuse their code. But, somehow, as a community, we have managed to muddle along reasonably well anyway. We have some advantages in that area: we have well established trusted distribution channels for software and a natural disinclination to run binaries from suspicious-looking third-party web sites. We also, for better or worse, have relatively few big-name programs which are sought out by users of more frequently targeted operating systems. As free software continues to grow in popularity, though, we may find ourselves confronted with unpleasant actions by sleazy people more often. Somehow we'll find a way to deal with them without compromising the freedoms that make free software what it is.

Comments (25 posted)

Semantic MediaWiki: Toward smarter wikis

Interactive Knowledge Stack (IKS) is an open source project focused on building an open and flexible technology platform for semantically enhanced Content Management Systems (CMS). It is a collaboration between academia, industry, and open source developers, co-funded with €6.58 million by the European Union. The goal is to enrich content management systems with semantic content in order to let the users benefit from more intelligent extraction and linking of their content. This could solve part of the chicken-and-egg problem for the semantic web that arises because end users don't have easy-to-use semantic web tools.

At the recent IKS workshop in Paris, one of the keynote speakers was Mark Greaves, who spoke about the possibilities of the semantic web in the wiki setting. His speech looked at the limits of traditional wikis, the promise of semantic wikis, and the birth of Semantic MediaWiki (SMW), an extension to MediaWiki, the wiki software that powers Wikipedia.

Wikis have become a powerful instrument for crowdsourcing, but they're not the only types of content management systems that tap into the potential of the crowd. Greaves, who is working as Director of Knowledge Systems at Paul Allen's asset management company Vulcan, emphasized that bulletin boards, forums, and newsgroups are the antecedents of wikis and even the beginnings of social networks. Now we have many websites that crowdsource their content from their users:

A critical property of wikis is consensus, which comes thanks to collaboration and custom policies. For instance, one of the core content policies of the Wikipedia encyclopedia is that each article should be written from a neutral point of view (NPOV). This forces authors to not write from their own point of view and that helps lead them to consensus with authors that have another point of view about the topic. The MediaWiki software also has software support to facilitate reaching consensus, such as the talk pages and change tracking.

But traditional wikis have their limits, as most knowledge is locked inside text and cannot be queried in a smart way. Wikipedia has an answer for this with thousands of lists, for instance lists of countries (which is itself a list of lists). But these are all manually maintained, each of them ordered by another property, like birth rate, literacy rate, population, income equality, and so on. So Greaves asked the logical question: "Why don't we give Wikipedia authors a way to add structure to their content?"

Semantic MediaWiki

That's where semantic wikis come in, and according to Greaves they hold a lot of promise:

One project working to add semantics to wiki systems is Semantic MediaWiki (SMW), a GPL licensed extension to MediaWiki that allows annotating semantic data within wiki pages. This means that a MediaWiki wiki that incorporates the extension is turned into a semantic wiki: content that has been enriched with semantic information can be used in specialized searches, used for aggregation of pages, displayed in alternate formats like maps, calendars, or graphs, and exported to formats like RDF (Resource Description Framework) and CSV (Comma-Separated Values).

How does this work?

Some examples will make it clear what SMW adds. For instance, on the normal Wikipedia page of France, there's a link to its capital city, Paris:

    ... the capital city is [[Paris]] ...

The [[Paris]] code is a link to a wiki page about Paris, but there's no information encoded about the specific relationship between France and Paris.

In contrast to this classical approach, the semantic web is all about interlinking data in a machine-readable way. The core technology under the hood is RDF, which is used to describe entities and their relationships. Each RDF statement comes in the form of a triple: subject - predicate - object. Each subject and predicate is identified by a URI, while an object can be represented by a URI or be a literal value such as a string or a number. So, a Semantic MediaWiki version of the sentence about Paris could be:

    ... the capital city is [[Has capital::Paris]] ...

The [[Has capital::Paris]] code not only adds a link to a wiki page about Paris, it also specifies the nature of the relationship between France and Paris: France has Paris as its capital. Or to translate it into an RDF triple: "France" (which is implicit as it is the topic of the current page) is the subject, "has capital" is the predicate, and "Paris" is the object.

This is an example where the object can be represented by a URI, but there are also other examples where the object is represented by a literal value such as a number:

    ... its population is [[Has population::65,821,885]] ...

When this code is on the page about France, it represents an RDF triple with "France" as its subject, "has population" as its predicate, and "65,821,885" as its object. These typed links (with the predicate as the type) give SMW an out-of-the-box mechanism to automatically generate lists. With SMW's inline queries feature, it's easy to re-use this structured information to generate lists and tables which are automatically updated and cached. For instance, users can easily generate a page with a list of all countries ordered by their population, or a list of all countries with a population greater than 20 million, or a table of all countries with their capitals, and so on.

Automatically-generated lists are not the only possibility when you start adding semantic links. You can also display the information in various formats, you can have different language versions of a wiki using the same data, you can integrate and mash-up your wiki's data and export it for external re-use, and more.

Ecosystem

The development of Semantic MediaWiki was initially funded by the EU project SEKT (Semantically Enabled Knowledge Technologies), and after this supported in part by the University of Karlsruhe in Germany. The first release was version 0.1 in 2005. In 2007, Vulcan started sponsoring the German company Ontoprise to develop a commercial version of the extension, Semantic MediaWiki+ (SMW+).

According to Greaves, there are 50 open source MediaWiki extensions that use the semantic information provided by SMW. For example, there's Halo, funded by Vulcan, that facilitates creation, retrieval, navigation and organization of semantic data with some intuitive graphical user interfaces, Semantic Drilldown that provides a faceted browser interface for viewing semantic data by filtering, and Semantic Result Formats that provides a large number of display formats, including maps, calendars, graphs, and charts.

If you want to install SMW on your own wiki, there's an extensive administrator manual with installation instructions and a list of the configuration options. For users who will be entering the semantic markup, the project also has a user manual

Some semantic wikis

Semantic MediaWiki is already used by over 300 public active wikis around the world. Greaves called these semantic wiki applications "the icing on the cake", as they really show the flexibility of adding semantics to a wiki. Some notable examples are Open Energy Information, a crowdsourced wiki with information about energy resources, including real-time data and visualizations, SKYbrary, a wiki created by several European aviation organizations to create a comprehensive source of aviation safety information, Familypedia, a wiki on family history and genealogy, SNPedia, a wiki investigating human genetics, Oh Internet, a wiki to track internet memes, and Ultrapedia, a search engine for OCR'd books.

Many organizations also use SMW internally, including Pfizer, Johnson & Johnson Pharmaceutical Research and Development, and the U.S. Department of Defense. Greaves added that Vulcan is eating its own dog food:

Towards a semantic Wikipedia

Some academics have already proposed using SMW on Wikipedia to tackle the problem of the many lists that have to be created manually, but according to Wikimedia Foundation Deputy Director Erik Möller it's still unclear whether SMW is up to the task of supporting a web site on the scale of Wikipedia. So while Semantic MediaWiki already powers a lot of web sites and is quite user-friendly, it remains to be seen whether it will eventually bring semantics to the ultimate wiki, Wikipedia.

The SMW project has a fairly detailed roadmap. Some of the interesting tasks are an improvement of the usability of the semantic search features (part of Google Summer of Code 2011), a light version of SMW without query capabilities, improvements for the Semantic Drilldown extension, and so on. It's already quite usable, as many of the active SMW wikis show, but to really reach the vision of the semantic web and be able to link various semantic wikis and other content management systems, Semantic MediaWiki needs to become as easy to use as Wikipedia.

Comments (5 posted)

Page editor: Jonathan Corbet

Security

Reactive vs. pro-active kernel security

Security patches are almost always a question of tradeoffs. Sometimes the protection offered outweighs the negative effects that a security-oriented fix brings—and sometimes it doesn't. In addition, pro-active security fixes often face an uphill battle to get into the kernel, especially if they cause performance or other problems, because many kernel developers are skeptical of "solutions" for theoretical problems. In many cases, these changes come under the heading of "kernel hardening", and don't correspond to a particular known security hole; instead they address a class of potential problems, which can be much harder to sell.

A good example of this can be found in Vasiliy Kulikov's recent RFC patch to implement some checks in the functions that copy data to and from user space. Copying the wrong amount of data to or from user space can lead to security problems, like code execution or disclosing the contents of kernel memory, so checking to ensure that copies are not larger than the expected data structure is certainly beneficial. But the copy_to/from_user() functions are performance-critical. In typical fashion, Linus Torvalds doesn't mince words in his reply to Kulikov:

He does go on to suggest that a cleaned up version which is configurable so that only those distributions or users who want the extra checking will pay the price for it might be acceptable. To Torvalds, the patch is more evidence of the "craziness" of the security community: "It's exactly the kind of 'crazy security people who don't care about anything BUT security' crap that I refuse to see." That, of course, is something of a recurring theme in terms of Torvalds and other kernel hackers' reactions to pro-active security fixes.

Ingo Molnar had a similar concern in the discussion of another of Kulikov's patches: an effort to remove control characters from log file output. Molnar is skeptical of the patch, partly because there are no specific threats that it addresses:

When an actual flaw is found in the kernel, especially if there are exploits for it in the wild, fixes are made quickly—no surprise. But theoretical flaws, or fixes that protect badly written user-space programs often have a tougher path into the kernel. Over the years, we have seen numerous examples of these kinds of patches, often coming from hardened kernel projects like grsecurity/PaX and Openwall. But, to some extent anyway, those projects are more concerned with security than they are with things like performance, and are willing to sacrifice performance to reduce or eliminate entire classes of security threats.

There is clearly a kernel (so to speak) of truth to Torvalds's complaint about "security crap", but there is also room for different kinds of kernels. It is reminiscent of the situation with SELinux in some ways. SELinux offers protections that can sometimes mitigate security problems before they come to light—exactly what pro-active security measures are meant to do—but SELinux is disabled by numerous administrators and by most distributions other than Red Hat's. For some, the extra protection that SELinux provides is not worth the overhead and problems that it can cause. Others may be more concerned about zero-day exploits and enable SELinux or run hardened kernels.

Another example of a fix that didn't make it into the kernel, though it would have eliminated a common security problem, is Kees Cook's attempt to disallow symbolic links in "sticky" directories—to stop /tmp symlink bugs (like this one from July 12). That particular fix was controversial, as some kernel hackers didn't think it appropriate to change core VFS code to fix buggy user-space programs. But moving the fix into a Linux Security Module (LSM)—along with a handful of other unrelated security fixes—didn't pass muster either.

There have also been various efforts to remove sources of information in /proc and elsewhere that can make it easier for exploits to function. Things like hiding kernel addresses from unprivileged processes, restricting page access to read-only and no-execute, protecting /proc/slabinfo, and lots of others have been proposed—sometimes adopted—over the last year or two. These kinds of fixes are often greeted with a level of skepticism (which is not so different from other kinds of patches really), and sometimes find their path into the mainline to be fairly difficult—sometimes impossible. That's not to say that any of those that were rejected should be in the kernel, but in most cases they do add some level of protection that very security-conscious users might be very happy to have.

The risk of keeping many of these pro-active hardening features out of the mainline is probably small, but it certainly isn't non-existent. There is a balance to be found; performance, maintainability, and less intrusiveness of patches are often more important to Torvalds and the kernel community than fixes that could, but might not, catch security exploits that aren't yet known. Essentially, making most users pay a performance penalty over and over again, potentially untold trillions of times, is too high a price. Fixing the problems that are found, when they are found, is the course that the mainline has (largely) chosen.

It is probably somewhat disheartening for Kulikov, Cook, and others to continually have their patches rejected for the mainline, but they do tend to be used elsewhere. Many of Cook's patches have been picked up in Ubuntu, where he is a member of the security team, and Kulikov is a student in the Google Summer of Code for Openwall specifically tasked with hardening both the Openwall kernel and upstream (to the extent he can anyway). Their efforts are certainly not being wasted, and security-conscious administrators may want to choose their distribution or kernel carefully to find the one that best matches their needs.

Comments (31 posted)

Brief items

Security quotes of the week

Comments (4 posted)

New vulnerabilities

apt: incorrect signature validation

Comments (none posted)

asterisk: multiple vulnerabilities

Comments (none posted)

blender: embedded code execution

Comments (none posted)

dbus: denial of service

Comments (none posted)

fabric: symlink attack

Comments (none posted)

kernel: multiple vulnerabilities

Comments (none posted)

libpng: denial of service

Comments (none posted)

libvirt: denial of service

Comments (none posted)

openoffice.org: code execution

Comments (none posted)

oprofile: privilege escalation/file overwrite

Comments (none posted)

vte: memory exhaustion

Comments (none posted)

xml-security-c: arbitrary code execution

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

Stable updates: the 2.6.32.43 and 2.6.33.16 stable kernel updates were released on July 13; 2.6.39.3 came out on July 8. All three contain the usual list of important fixes - quite a few of them, in the 2.6.39.3 case.

Comments (none posted)

Quotes of the week

Comments (1 posted)

What will that kernel be called?

Fast forward to 3.0-rc7, and the kernel is still identifying itself as 3.0.0. While hard information is relatively scarce, it seems that there are still some programs and scripts out there that will break without that final number. In a sense, the three-number versioning scheme has become a part of the kernel ABI over the years. One can argue that anything which breaks was written in a silly way to begin with, but the goal is still to avoid breaking user space. So, while no definitive answer has been given, it looks like we're going to have three-number mainline releases for some time. It would be most surprising to see the stable updates starting with anything other than 3.0.1, though.

There is one related question, though: what about programs that cannot cope with anything other than "2.6.x" as the kernel version number? Evidently such programs exist. Saying that kernel releases must start with "2.6" forevermore stretches even the firmest commitment to binary compatibility; it's not something that is going to happen. In an attempt to make life easier for people stuck with such programs, Andi Kleen has implemented a new system personality that pretends the 3.0 change never happened. If a program is run under that personality, it will think that the kernel version is 2.6.40; 3.1 will look like 2.6.41, etc.

Andi says "I know this is somewhat ugly, but I didn't find a better workaround, and compatibility to existing programs is important." As of this writing, though, the patch has not made it into the mainline.

Comments (12 posted)

Kernel development news

Who wrote 3.0 - from two points of view

This kernel release inaugurates the beginning of the 3.x series of kernels. As has been mentioned many times here, there is nothing particularly special about the 3.0 release; it has been, in many ways, a relatively boring development cycle. But it still provides a good opportunity to look back over a longer period of time. But, before doing that, we'll start with this cycle, which has, as of 3.0-rc7, seen 9,007 changesets contributed by 1,110 developers. The kernel grew 113,000 lines in this development cycle - a relatively modest figure by contemporary standards.

The most active developers during this cycle were:

Most active 3.0 developers

By changesets K. Y. Srinivasan 343 3.8% David S. Miller 176 2.0% Dan Williams 149 1.7% Jonathan Cameron 119 1.3% Takashi Iwai 108 1.2% Mark Brown 91 1.0% Johannes Berg 84 0.9% Peter Zijlstra 80 0.9% Sage Weil 79 0.9% Tejun Heo 78 0.9% Joe Perches 77 0.9% Michał Mirosław 77 0.9% Konrad Rzeszutek Wilk 76 0.8% Jamie Iles 75 0.8% Alex Deucher 71 0.8% Artem Bityutskiy 69 0.8% Steven Rostedt 66 0.7% Mike Frysinger 63 0.7% Sujith Manoharan 62 0.7% Avi Kivity 58 0.6%

By changed lines Dan Williams 82466 9.1% Larry Finger 74643 8.3% Dmitry Kravkov 38960 4.3% Vasanthakumar Thiagarajan 33618 3.7% Mauro Carvalho Chehab 26815 3.0% Bing Zhao 25576 2.8% Ralph Metzler 19933 2.2% Takahiro Hirofuchi 19318 2.1% Chaoming Li 14743 1.6% Jonathan Cameron 14574 1.6% Chris Metcalf 12144 1.3% Luis R. Rodriguez 11443 1.3% Dave Jiang 11006 1.2% Wolfram Sang 9886 1.1% K. Y. Srinivasan 9709 1.1% Mark Brown 9127 1.0% Arend van Spriel 7667 0.8% Kenji Toyama 7528 0.8% Alan Cox 7449 0.8% Takashi Iwai 7410 0.8%

K. Y. Srinivasan topped the list of changeset contributors with a massive set of cleanups to the Microsoft HV driver in the staging tree; it's impressive to see how much cleanup less than 15,000 lines of code can require. David Miller made a lot of changes in the networking subsystem; some were warning fixes and such, while others were more substantial. Dan Williams contributed Intel's "isci" storage driver, merged in 3.0-rc6. Jonathan Cameron contributed a lot of work to rationalize the industrial I/O (iio) subsystem and prepare it for an eventual merge into the mainline. Takashi Iwai continues to do large amounts of work in the ALSA sound driver subsystem.

The isci driver put Dan Williams at the top of the "lines changed" column. Larry Finger's contribution is largely negative (in line counts - not in value): he removed the rt2860sta and rt2870sta drivers from the staging tree now that the mainline driver can replace them. Dmitry Kravkov appears due to a firmware update; the bnx2x driver is one of the few which still has firmware in the mainline kernel tree. Vasanthakumar Thiagarajan also removed a lot of code, mostly through the process of eliminating duplication between Atheros wireless drivers. Mauro Carvalho Chehab removed the obsolete Micronas drx397xD driver.

A total of 184 employers (that we were able to identify) participated in the 3.0 cycle; the most active among them were:

Most active 3.0 employers

By changesets (None) 1085 12.0% Red Hat 1000 11.1% Intel 839 9.3% (Unknown) 569 6.3% Novell 441 4.9% IBM 374 4.2% Microsoft 361 4.0% Atheros Communications 241 2.7% Texas Instruments 234 2.6% Broadcom 222 2.5% Oracle 187 2.1% AMD 162 1.8% Nokia 158 1.8% Fujitsu 154 1.7% Google 129 1.4% University of Cambridge 119 1.3% Analog Devices 118 1.3% (Consultant) 113 1.3% Samsung 103 1.1% Wolfson Microelectronics 103 1.1%

By lines changed Intel 163232 18.1% (None) 152840 16.9% Broadcom 61948 6.9% Red Hat 59079 6.5% Atheros Communications 53268 5.9% Marvell 31118 3.4% (Unknown) 29261 3.2% IBM 20587 2.3% Metzler Brothers Systementwicklung GbR 19933 2.2% Novell 19578 2.2% University of Cambridge 16969 1.9% Pengutronix 16207 1.8% Realsil Microelectronics 14876 1.6% Analog Devices 12998 1.4% Tilera 12257 1.4% Freescale 11637 1.3% Microsoft 11564 1.3% Texas Instruments 10802 1.2% Wolfson Microelectronics 10051 1.1% Samsung 9784 1.1%

There are few surprises here. Microsoft at 4% of the total changes is unusual; one assumes that presence will not be permanent: even the HV drivers can only need so much cleaning up. The percentage of changes from hobbyists continues to drop; whether that's a bad thing (the kernel is becoming increasingly unapproachable to volunteer developers) or a good thing (it's impossible for anybody who can hack the kernel to remain unemployed) is still not clear.

A longer-term look

The release of 3.0 provides as good an opportunity as any to look at the entire 2.6 series. Thanks to the BitKeeper history tree put together by Thomas Gleixner, it is possible to get detailed information back almost to the beginning of the 2.5 development cycle, which can be thought of as the set of -rc kernels leading up to 2.6.0. This information is far from complete, unfortunately. The 2.5.0 through 2.5.3 releases predate the BitKeeper transition, and thus appear as big patches from Linus. Even thereafter, a lot of early changes appear to have been contributed by the maintainer they passed through instead of the actual author; it took a while to establish the infrastructure to properly credit all work. Still, there is enough data there to work with.

The history from the beginning of the 2.5 development series covers about 9.5 years of development. During this time, some 291,664 changesets were contributed by 8,078 developers; those changes added 10.5 million lines of code. Here are the most active developers over that extended period:

Most active developers since 2.5.0

By changesets Andrew Morton 7638 2.6% David S. Miller 5203 1.8% Al Viro 3828 1.3% Greg Kroah-Hartman 3309 1.1% Russell King 3226 1.1% Alan Cox 2609 0.9% Ingo Molnar 2599 0.9% Stephen Hemminger 2535 0.9% Bartlomiej Zolnierkiewicz 2485 0.9% Linus Torvalds 2479 0.8% Christoph Hellwig 2429 0.8% Takashi Iwai 2414 0.8% Adrian Bunk 2306 0.8% Tejun Heo 2205 0.8% Thomas Gleixner 2205 0.8% Paul Mundt 2113 0.7% Dave Jones 2067 0.7% Randy Dunlap 1853 0.6% Ralf Baechle 1786 0.6% Johannes Berg 1770 0.6%

By changed lines Greg Kroah-Hartman 738134 2.3% Bartlomiej Zolnierkiewicz 553077 1.7% Andrew Morton 537737 1.7% Alan Cox 432023 1.4% Jaroslav Kysela 387649 1.2% Adrian Bunk 380691 1.2% James Bottomley 367435 1.2% Linus Torvalds 325954 1.0% Ralf Baechle 319859 1.0% Paul Mackerras 279454 0.9% Sam Ravnborg 270118 0.8% David S. Miller 254574 0.8% Christoph Hellwig 238749 0.8% Mauro Carvalho Chehab 232793 0.7% Uwe Kleine-König 215560 0.7% Russell King 209362 0.7% Benjamin Herrenschmidt 195707 0.6% Jeff Garzik 190724 0.6% Paul Mundt 185781 0.6% David Howells 183872 0.6%

It should be repeated that these numbers are highly approximate. For example, while Andrew Morton was indeed a prolific code contributor in the 2.5.x and early 2.6 days, he didn't write quite that many patches; a lot of patches from others that went through him lost their authorship information on the way. That information is generally present in the changelog - somebody could try to make a new repository with proper credits given some time - but, for now, we'll have to make do with fuzzy numbers. The per-employer numbers are necessarily even fuzzier - to the point that they are most likely not worth showing here. Suffice to say that, in general form, they resemble the numbers we have been showing for the last few years.

For those who are curious about just the post-2.6.0 kernels, the numbers don't change that much. Since 2.6.0, there have been 264,706 changesets contributed by 7,725 developers adding 8.7 million lines of code.

One other exercise with this data seemed interesting: a determination of who have been the most consistent contributors over those nine years and some. After running a script to track which developers contributed to each major release, twelve developers were found who had contributed to all 41 of them. Additionally, a handful of developers have gotten code into almost every release. The most consistent developers are:

Most consistent developers 2.6.0-3.0
Developer	Releases	Missed releases
Linus Torvalds	41
David S. Miller	41
Greg Kroah-Hartman	41
Andrew Morton	41
Christoph Hellwig	41
Alan Stern	41
James Bottomley	41
Randy Dunlap	41
Russell King	41
Al Viro	41
Stephen Hemminger	41
Andi Kleen	41
Jens Axboe	40	v2.6.1
Jean Delvare	40	v2.6.4
Dave Jones	40	v2.6.35
Benjamin Herrenschmidt	40	v2.6.1
Jeff Garzik	40	v2.6.36
Ingo Molnar	39	v2.6.2 v2.6.5
Herbert Xu	39	v2.6.3 v2.6.5
Patrick McHardy	39	v2.6.2 v2.6.6
Dmitry Torokhov	38	v2.6.3 v2.6.4 v2.6.6
Rusty Russell	38	v2.6.1 v2.6.15 v2.6.39
Matthew Wilcox	38	v2.6.14 v2.6.36 v3.0
Dave Kleikamp	38	v2.6.26 v2.6.33 v2.6.37
Len Brown	38	v2.6.1 v2.6.17 v2.6.39
Oliver Neukum	38	v2.6.4 v2.6.14 v2.6.37
Wim Van Sebroeck	38	v2.6.4 v2.6.6 v3.0
Andrew Vasquez	38	v2.6.0 v2.6.1 v2.6.5
James Morris	38	v2.6.16 v2.6.37 v2.6.39
Neil Brown	37	v2.6.1 v2.6.2 v2.6.3 v2.6.6
Trond Myklebust	37	v2.6.1 v2.6.2 v2.6.8 v2.6.10
Paul Mackerras	37	v2.6.1 v2.6.3 v2.6.38 v2.6.39
Bjorn Helgaas	37	v2.6.3 v2.6.20 v2.6.39 v3.0
Tony Lindgren	37	v2.6.0 v2.6.1 v2.6.5 v2.6.20
Nicolas Pitre	37	v2.6.3 v2.6.4 v2.6.5 v2.6.23
Stephen Rothwell	37	v2.6.1 v2.6.2 v2.6.3 v2.6.7
David Howells	36	v2.6.1 v2.6.2 v2.6.3 v2.6.4 v2.6.6
Eric Sandeen	36	v2.6.1 v2.6.8 v2.6.11 v2.6.17 v3.0
Ralf Baechle	36	v2.6.1 v2.6.3 v2.6.4 v2.6.7 v2.6.38
Arjan van de Ven	36	v2.6.1 v2.6.3 v2.6.13 v2.6.14 v3.0
David Brownell	36	v2.6.33 v2.6.34 v2.6.37 v2.6.38 v3.0

Your editor, who only got changes into 32 releases during this time, knows what an accomplishment it is to consistently contribute to every release over such a long period of time.

But, then, creating the kernel and the development process we have over the course of the last 20 years is an impressive accomplishment. There are few development projects which have lasted this long, gone this far, and have been more vital than ever. It has been fun to watch. It seems likely that things will remain just as fun over the next 20 years - one could argue that we have just begun.

Comments (20 posted)

The structured logging challenge

After all these years, the main mechanism by which the kernel passes information to user space remains the lowly printk() function. It is, needless to say, a useful and flexible way of getting messages out, but it imposes almost nothing on the structure of those messages. That leads to all kinds of output like (from drivers/net/de620.c):

    printk(KERN_WARNING "%s: Thanks, I feel much better now!\n", dev->name);

or the famous message from drivers/char/lp.c:

    printk(KERN_INFO "lp%d on fire\n", minor);

System administrators should not be faulted for wondering what they should do in response to messages like these.

There have been some changes to impose structure on printk() output, starting with the addition of a marker for the severity level of each message. It still is not hard to find printk() calls without severity levels, though; actually enforcing the use of these markers has proved hard to do. A bit more structure is added by dev_printk() (and variants like dev_err()), but the use of these functions is even less universal.

The lack of structure means that there is little consistency between messages; any two network drivers will almost certainly print different things to indicate the same situation. Kernel messages can also vary over time; messages emitted by printk() are also not normally seen as part of the kernel ABI, despite the fact that changing them can break scripts that try to extract useful information from the system logs. So it's not surprising that, about one year ago, Andrew Morton said:

Various people have tried to improve the situation in spots; the user-friendly disk names, by trying to attach a consistent name to devices, is one such attempt. The netoops patch from Google is another; it helps Google figure out why machines are crashing without requiring operators to actually dig through the logs. But these changes are far from an overall framework for structured data from the kernel.

There have been a few attempts to make such frameworks over the years; all have fallen far short of making it into the kernel. It's not hard to come up with plausible reasons for this failure. The amount of work required is huge, especially if one wants to add structure to the bulk of interesting communications from kernel space. Developers like printk(); they are less likely to be enamored of some other interface which requires more work to use, is less flexible (by design) in its output, and which may well have to sit alongside the existing printk() logging. Coming up with a structured format which meets everybody's needs - and which will not have to be supplemented with a "version 2" format in the coming years - presents its own challenges.

It must also be said that kernel developers, as a whole, see little value in standardized, structured kernel logging information. It will not help them to debug their kernels. The fact that a lot of users want this sort of feature is far from irrelevant to the development community, but experience has shown that a lack of developer interest can make it much harder to get changes merged - especially if those changes are wide-ranging and disruptive.

If this problem is ever to be solved, it would seem that two things need to be found: a mechanism which looks like it could work and a motivation for kernel developers to accept it. The motivation can probably found in a combination of (1) their paychecks as customers continue to push for this capability and (2) the prospect of a continuing stream of ad hoc patches adding structure to various corners without solving the real problem. But that leaves open the problem of finding a workable solution.

Your editor has a half-baked thought on this matter based on the realization that the kernel already has a nice mechanism for passing structured data to user space. On almost any contemporary system, the /dev directory is managed by the udev daemon; udev works by receiving highly-structured messages from the kernel describing the coming and going of devices, changes in their configuration, firmware load requests, and more. It is an established protocol which enables sophisticated user-space responses to kernel events. Udev and the associated "uevent" mechanism had some early growing pains, but this code is now stable, functional, and almost universally used. Perhaps it's time for this mechanism to take on some new duties.

Uevents work because the format is simultaneously structured and flexible; it can be extended when the need arises. The generation of events is almost entirely done automatically by the driver core; most driver authors need not do anything to cause them to happen and, indeed, may not even know that this mechanism is operating underneath the hood. Driver authors don't have to make their own events; they would have to go out of their way to prevent them from happening.

Logging of other types of events will probably require explicit support in the relevant kernel code; that is the part needing some extra thought. The creation of uevents by hand is a bit of a labor-intensive business; the relevant code tends to look like:

	retval = add_uevent_var(env, "ACTION=%s", action_string);
	if (retval)
		goto exit;
	retval = add_uevent_var(env, "DEVPATH=%s", devpath);
	if (retval)
		goto exit;
	retval = add_uevent_var(env, "SUBSYSTEM=%s", subsystem);
	if (retval)
		goto exit;

Clearly, any attempt to place this kind of code in every logging location is not going to get very far. What is needed is a useful set of helper functions. These functions, for maximum utility, would probably be tied fairly tightly to the underlying subsystems. Storage drivers could have functions to report block errors, device changes, and multipath connectivity changes. Network drivers would need to report events like carrier loss, excessive checksum errors, or duplicate MAC addresses. All kernel code could benefit from helpers to log allocation failures or failed assertions. In each case, the helper would standardize the format of the reported information while allowing the addition of information specific to the call site.

The addition of a new set of logging functions would necessarily require changes to drivers to use those functions. So it would take time to achieve anything close to comprehensive coverage, and 100% coverage would never happen. But, then, we still don't have 100% coverage for the KERN_* severity markers. If this interface proved useful, one could imagine that the code paths of interest to enterprise distribution customers would be covered in relatively short order.

But, then, there are probably several things fatally wrong with this idea; the structured logging problem will likely remain unsolved for some time yet. But the problem will not go away; if anything, the need to recognize and automatically respond to system events will only increase. Someday somebody will come up with a solution that works and that can be adopted with minimal pain; until then, printk() remains the only show in town.

Comments (2 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.0-rc7 . (July 12, 2011)

• Greg KH: Linux 2.6.39.3 . (July 10, 2011)

• Greg KH: Linux 2.6.33.16 . (July 13, 2011)

• Greg KH: Linux 2.6.32.43 . (July 13, 2011)

Core kernel code

• Hans Verkuil: poll: add poll_requested_events() function . (July 8, 2011)

• Christopher Yeoh: Cross Memory Attach v3 . (July 11, 2011)

Development tools

• Hui Zhu: KGTP (Linux Kernel debugger and tracer) 20110712 release . (July 12, 2011)

• Steven Rostedt: ftrace: entend notrace to notrace called functions . (July 13, 2011)

Device drivers

• Nao Nishijima: Persistent device name using alias name . (July 8, 2011)

• Marc Zyngier: Core device subsystem . (July 8, 2011)

• Rafael J. Wysocki: PM / Domains: Add code for powering off all unused PM domains . (July 11, 2011)

• Mark Brown: regmap: Generic I2C and SPI register map library . (July 11, 2011)

• Len Brown: mrst_pmu: driver for Intel Moorestown Power Management Unit . (July 13, 2011)

• Magnus Damm: leds: Renesas TPU LED driver V2 . (July 13, 2011)

Memory management

• Dave Chinner: Per superblock cache reclaim . (July 8, 2011)

• Mel Gorman: Reduce frequency of stalls due to zone_reclaim() on NUMA . (July 11, 2011)

• Mel Gorman: Reduce filesystem writeback from page reclaim (again) . (July 13, 2011)

Architecture-specific

• Mark Brown: clk: Initial feedback for off-SoC slow bus clocks . (July 11, 2011)

• Andy Lutomirski: x86-64 vDSO changes for 3.1 . (July 13, 2011)

• Mahesh J Salgaonkar: fadump: Firmware-assisted dump support for Powerpc. . (July 13, 2011)

Security-related

• Jari Ruusu: Announce loop-AES-v3.6d file/swap crypto package . (July 11, 2011)

• Tetsuo Handa: TOMOYO 2.4: Conditional ACL support . (July 11, 2011)

• Tetsuo Handa: TOMOYO 2.4: Network socket operation restriction support. . (July 11, 2011)

Virtualization and containers

• Glauber Costa: Steal time for KVM . (July 12, 2011)

• Serge Hallyn: user namespaces: continue targetting capabilities . (July 13, 2011)

Benchmarks and bugs

• Rafael J. Wysocki: 3.0-rc6-git6: Reported regressions from 2.6.39 . (July 11, 2011)

• Rafael J. Wysocki: 3.0-rc6-git6: Reported regressions 2.6.38 -> 2.6.39 . (July 11, 2011)

Miscellaneous

• Andi Kleen: Add a personality to report 2.6.x version numbers . (July 11, 2011)

• Bart De Schuymer: ebtables v2.0.10-1 . (July 11, 2011)

• Jozsef Kadlecsik: ipset-6.8 released . (July 11, 2011)

Page editor: Jonathan Corbet

Distributions

Linux Mint beefs up its Debian-based distribution

Users of Linux Mint's Debian-based distribution are getting a couple of new options for managing updates. Instead of dealing with a continual flood of updates from Debian Testing, the Mint maintainers are going to offer monthly updates and a slightly tamer "Incoming" repository.

Linux Mint is, of course, a popular and user-friendly distribution that started as an Ubuntu derivative. Initially a GNOME-based distribution that offered a fairly light set of changes from Ubuntu — most notably pre-installed multimedia codecs and proprietary drivers — the project now offers KDE, LXDE, and Xfce-based releases as well. The Linux Mint 11 release (covered here in May) deviates even further from Ubuntu by sticking with GNOME 2.32 rather than moving to the Unity desktop. In September 2010, the project also started providing rolling release based on Debian Testing, but with the same themes, codecs, drivers, and management tools it offers with the main Mint edition.

While the rolling release model has the advantage of providing faster updates for packages as they land in Debian Testing, the model has its headaches as well. In the post announcing the Update Packs for Linux Mint Debian Edition (LMDE), Clement Lefebvre points out that LMDE requires more work and experience on the part of its users:

What's worse, users often encounter a situation that other LMDE users are not having problems with. "Because things change constantly and people don't update at the same time or as frequently, it's hard to find people with the same problem and so it's hard to talk about workarounds and find solutions." For now, Lefebvre says that users are depending on a thread in the Linux Mint forums to support one another, but says that it's "chaotic" and "far from ideal."

Problems with rolling updates might account for the LMDE release losing users. According to the stats from the Linux Mint folks, the Debian Edition has been losing popularity for the last few months. The statistics from April showed LMDE with 8.7% of Mint users (according to statistics from package updates). The numbers for May had dropped to 6.03%, and in June the Debian Edition claimed only 3.47% of users. There are no raw numbers available, so it's not entirely clear that the drop in LMDE percentages aren't a result of increased popularity in the other Mint versions. Note that you need to scroll to the bottom of the pages past the donor information to find the user statistics.

Changes afoot

To cope with the continual updates from Debian, the Mint folks are introducing two new repositories and a different version of the Linux Mint Update Manager.

For users who want rolling updates but without having to suffer system breakage, there's the new Linux Mint Debian Latest repository. This is a repository that should have monthly updates that have been tested and will be delivered to users in batches. As an example, Lefebvre points out that GNOME 3 should be entering Debian Testing soon. However, if users point their repository to the Latest repository instead of the upstream Debian Testing repository, they'll get updates "after the Linux Mint team has tested the update and gathered precious information on it." Note that this doesn't promise that the Mint team will fix broken packages or otherwise smooth over problems in Testing — they'll simply be making an effort to ensure the packages in the Latest repository won't cause problems when users run updates. If a package or set of packages has an issue of some kind, whether that's package conflicts, missing dependencies, or bugs that keep the updates from being useful, then they'll be held back.

How will they know that a package or set of packages have problems? The Mint team is also providing a Linux Mint Debian Incoming repository, which will be a speed bump between Debian's Testing repositories and the Latest repository. The team will test the Update Packs in the Incoming repository before putting them into Latest. More adventurous users can use the Incoming repository and report problems before they're released into the Latest repository for more users.

Once the team is satisfied that the repository is stable and ready for updates, it will be released as an "update pack," into the Latest repository for users to grab via the new Update Manager. Each Update Pack will come with information on the updates carried in the pack, such as potential problems with the update. For example, the current update pack has warnings that it may cause problems with Flash, and some of the packages that start Debian's transition to GNOME 3 may cause some applications to ignore a user's GTK theme. Where applicable, users are given advice on how to fix or bypass the problems — and users will still have the option of ignoring or blocking package updates.

With the update, users now have three choices of repositories for LMDE; the Latest repository, the Incoming repository, and the upstream Debian Testing repository. For users who want to stick with Debian Testing, nothing changes.

Users who want to test out the two new repositories are warned that they are brand new, and should wait a month or two for the LMDE images to be respun. The LMDE with GNOME image was last updated in December, 2010 and the image for LMDE with Xfce was last updated in April. But for the brave souls willing to try the new system before it's part of the respins, it's a fairly simple set of steps. All that's required is to install the mintupdate-debian package, and switch out the Debian Testing repository in /etc/apt/sources.list with the Incoming or Latest repository. Mint is asking users to report problems with packages in Incoming in the forums.

Note that this puts Mint in a position of hosting a much larger package archive than before, when they only needed to point users to the Debian repositories for most of the packages. The new repository is made possible by an agreement with AYKsolutions to provide an additional server with 1Gbps output to handle Linux Mint's Debian repository.

KDE and LXDE changes

In a separate post, Lefebvre announced some changes for the LXDE and KDE Mint releases. Linux Mint 11 LXDE is getting closer, but the project is replacing the LXDM login manager with GDM, due to problems loading X from the live CD with the first Mint 11 LXDE release candidate.

The KDE release of Mint may be getting a much bigger overhaul — namely, a switch to Debian away from Kubuntu as its base. Lefebvre says that "lack of performance and the amount of resources needed by the Kubuntu base" has resulted in discussions about delaying the Linux Mint 11 KDE release and switching to Debian Testing for KDE as well. "Although we're close to a release in terms of quality, a discussion is ongoing about the possibility of switching the KDE edition to LMDE. This would give it the performance it needs and make KDE a viable alternative on mid-spec computers." Depending on the outcome of those discussions, the KDE release could be out by the end of July, or users may get a release candidate for an LMDE-based KDE distribution some time in August.

It's worth noting that Mint seems to be adopting a model that's similar to openSUSE Tumbleweed. That is, both seek to provide a rolling release that provides users with software at a faster clip but without as many broken packages as they'd encounter running a rolling release tied to package updates immediately as they enter development distributions.

Though Lefebvre pointed out how the project will be able to scale technically thanks to the new server, what's unclear is if Mint has enough manpower. As was explored in the Linux Mint 11 piece in May, the Mint team is a very small operation — and Mint seems to be taking on quite a lot of new work. It will be interesting to see if the monthly update schedule boosts LMDE's popularity and smooths over the problems users have had with LMDE so far. For desktop Linux users who want quick access to newer software, this might be a good option once the Mint team gets the process nailed down.

Comments (1 posted)

Brief items

Distribution quote of the week

Comments (none posted)

CentOS 6.0 released

Full Story (comments: 4)

Debian's patent policy FAQ

Comments (15 posted)

Distribution News

Debian GNU/Linux

bits from the DPL for June 2011

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 413 (July 11)
• Fedora Weekly News Issue 280 (July 6)
• Maemo Weekly News (July 11)
• openSUSE Weekly News, Issue 183 (July 10)
• Ubuntu Weekly Newsletter, Issue #223 (July 10)

Comments (none posted)

Stories of Linux: Interview with Ian Murdock on Debian's Early Days (Linux.com)

Comments (none posted)

PCLinuxOS Review: What Does PCLinuxOS Have to Offer? (Linux.com)

Comments (none posted)

Page editor: Rebecca Sobol

Development

Making GEGL useful for applications beyond GIMP

The Generic Graphics Library (GEGL) is a non-destructive image processing framework built for raster image editing. Most people are probably familiar with it through its role as the GIMP's next-generation core editing layer, or else through critics who claim that it is taking too long to integrate. However one feels about the GIMP's pacing, though, a new branch of GEGL was just created that will allow easier use of the library by other applications — including, eventually, by non-GTK+ applications.

GEGL, graphs, and GIMP

To the casual observer, the "big news" about GEGL is generally reported to be enabling all GIMP functionality on 16-bit-per-channel and 32-bit-per-channel images. The real innovation, however, is that GEGL is non-destructive — which allows for lossless editing, rollback, and even reordering of operations. GEGL accomplishes this by storing the image as a directed acyclic graph of nodes, rather than transforming image buffers directly. Each node in the graph typically represents an operation, which can be anything from loading a buffer from a file, to applying a filter, to a matrix transform. But nodes can also be parents of sub-graphs, which enables GEGL to create nested image processing pipelines.

Ultimately, however, the "file" is simply the graph in question, so it can be stored along with any raster samples and/or vector path objects (such as selection regions) as a complete representation of the editing session — without destructively modifying the original source images. Because each node is either an operation or a sub-graph of operations, each edge in the graph represents the image that would be generated by walking the graph. GEGL's C API allows code to re-parent and rearrange nodes in the graph, so that the chain of operations can be manipulated.

The current stable branch of GIMP (2.6.x) uses GEGL for internal color operations, and allows the user to selectively use a GEGL Operation tool to apply any of two dozen or so GEGL operations to the current image. Nevertheless, the fact that GEGL has been in development since 2000 and has yet to completely replace GIMP's editing core has led a few to dismiss it as vaporware that is not ready for prime time.

There is rarely a single, simple answer as to why a particular coding task hasn't taken place, but in this case, those familiar with the code do not regard GEGL itself as incomplete. Rather, the lengthy transition period seems largely to be the result of GIMP's enormous size and relative paucity of full-time developers. Replacing the core of a large, multi-faceted application — particularly when the new core uses an entirely different model — is simply a labor-intensive task, and there seem to be too few volunteers given GIMP's size and users' expectations that it will remain stable during that kind of transition.

GTK and truly generic graphics

Reality is that any open source project could start using GEGL as an image processing library, of course, but it has not always been simple to do so. That is where this month's new branch comes into play. Jon Nordby of the MyPaint and OpenRaster projects started the GEGL-GTK branch in order to separate out some GTK+ functionality and turn GEGL into a more general library for use by other (that is, non-GIMP) applications. MyPaint is considering adopting GEGL as its own image processing system further out on its roadmap and Nordby saw this as the first step in making a hard assessment of the option, as well as being beneficial to other projects.

The first objective was the removal of a pair of GTK dependencies from within the GEGL codebase. One was gtk-display, a GEGL operation to display a graph's rendered output inside a GTK window — certainly useful for GIMP, but not a core operation. The other was the GeglView GTK widget that resided only in the examples directory; it was meant to show developers how to integrate GEGL and GTK. Both have now been pulled out and into the nascent GEGL-GTK utility library. That move immediately removes the GTK dependency from GEGL itself, but it has additional implications for application developers interested in adding GEGL support to their code.

The GeglView widget allows a GTK application to embed a live-updating view on a GEGL graph, complete with basic view operations such as zooming in and panning in any direction. The revamped gtk-display operation allows a GEGL graph to render itself onscreen, using the aforementioned GeglView widget. Nordby also added GObject introspection to GeglView (having added it to GEGL itself earlier in the year), and updated both the widget and the operation to support GTK2 and GTK3.

The result is a cleaner separation between the GEGL image processing core and the GUI toolkit, which, as Nordby explained on the GEGL mailing list, is a prerequisite for wider adoption of the library by application developers. There was some discussion in the thread about adding Qt or other toolkit-specific utilities in separate libraries in the future. For his part, GEGL maintainer Øyvind Kolås describes the changes as things that make sense to do from a GEGL point of view, too, and noted that he had created a Clutter integration library with similar hopes for making GEGL more accessible to other projects. "Having such widgets, and having them exposed should make it a lot easier for people to test getting going with interactive applications built on top of GEGL," he said.

The GTK-specific bits that seeded Nordby's GEGL-GTK have already been taken out of the mainline GEGL code, although a new release has not been rolled. Kolås also pointed out that Nordby had been actively contributing to the GEGL project in other areas, such as gardening the issue tracker and cutting into the "unhandled/lingering/patch-waiting/obsolete" bug count. The result is a bug tracker that proves more useful to project management.

Nordby says there is still a lot to be done in GEGL-GTK, starting with adding support for rotation to the GeglView widget, and adding signals to draw widget backgrounds and overlays. The latter feature is necessary to make GeglView functional as an editing component and not simply a live display — by overlaying editing tools and interface elements, for example. He would also like to add a widget property to support changing the scaling or cropping behavior, and an API to access the transformation matrix of the GEGL graph.

Further out, making the GeglView widget usable from languages other than C may be important to other developers. It is also possible that additional widgets would prove useful, such as an editable tree-view of the entire GEGL graph, or a widget that allows manipulating a node's properties. Ultimately, however, adding those features depends on developer demand. Although they would allow for low-level editing of the graph, such functionality is not how most creative graphics applications function.

GEGL all the way

One should not expect the GEGL-GTK separation to affect a large number of projects in the near term. MyPaint, like GIMP, is a GTK+ application, so it does not gain directly from separating out the GTK integration features into a distinct library (although it would benefit from the GObject introspection, GTK3 support, and new features). The other major project currently working with GEGL is the raw photo editor Darktable, which is also GTK+-based.

In the medium-to-long term, however, there is more hope. OpenRaster, the graphics interchange file format, already supports GEGL graph data. GIMP continues to migrate features to GEGL internally, with "projection" (the feature that composites selections and image layers on top of each other), scaling, the cage transform tool, and a large number of filters on the roadmap for the next stable release, 2.8.

Because raw photo editors are already expected to operate in non-destructive editing mode (the large file sizes and irreplaceable nature of the original images dictate it), they would seem to be a natural fit for GEGL. But the quickest path to increased GEGL adoption might actually come from the video realm. In May, Nordby wrote a GStreamer element called geglfilter that allows an editor to apply a GEGL filter to a GStreamer video processing pipeline.

It has not been accepted into the GStreamer plug-ins packages yet, pending some changes, but it sounds like it will eventually work its way into the upstream repository. Video, like raw photo conversion, defaults to a non-destructive editing paradigm. GStreamer-based editors such as PiTiVi could incorporate simple GEGL filters as effects, although adding a complex GEGL graph editor would be considerably more work.

As for KDE, Qt, and other GUI environments, only time will tell. Still, for a library project that has been under the purview of a single application for this long, the increased activity over the past few months is a promising turn of events. However long GEGL may have been in development (or more accurately, however long its GIMP migration may have taken), it remains the premiere, if not the only, non-destructive image editing library available in open source software. When it becomes easily accessible to other projects, the benefits could be inspiring to a wide range of creative users.

Comments (3 posted)

Brief items

Quotes of the week

- pa_log_warn("D-Bus name org.pulseaudio.Server already taken. Weird shit!");
+ pa_log_warn("D-Bus name org.pulseaudio.Server already taken. My word, "
              "this is an extraordinary occurence I dare say!");

Comments (8 posted)

Malcolm: A visualization of GCC's passes as a subway map

Full Story (comments: 5)

MELT plugin 0.8 for GCC 4.6

Full Story (comments: none)

Nettle 2.2

Full Story (comments: none)

PuTTY 0.61 released

Comments (28 posted)

SeaMonkey 2.2 released

Full Story (comments: none)

Selenium 2.0 released

Comments (none posted)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (July 12)
• LibreOffice development newsletter (July 7)
• LibreOffice development newsletter (July 12)
• Tahoe-LAFS Weekly News (July 9)

Comments (1 posted)

Page editor: Jonathan Corbet

Announcements

Articles of interest

Brockmeier: Anti-rantifesto: Why free software and free culture aren't the same

Comments (57 posted)

DeRose: Designing pro creative apps (Part 1-3)

Comments (64 posted)

Google: "Android is the Linux desktop dream come true" (der Standard)

Comments (42 posted)

Microsoft's Android Shakedown (Forbes)

Comments (18 posted)

Upcoming Events

openSUSE Conference team announces opening of registration

Full Story (comments: none)

Events: July 21, 2011 to September 19, 2011

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol