LWN.net Weekly Edition for July 8, 2011

Companies and contributions

The source for much of this speculation would appear to be the results of a survey of the Eclipse community published in June. This survey led to a number of interesting conclusions; for the curious, the results are available as a report [PDF] or in an ODS spreadsheet. One can learn a lot about the Eclipse user community there; some of its members, evidently, still develop in RPG. Most of them, though, have moved on to more current technology.

Among the conclusions drawn from this survey were that developers are increasingly using Linux as their desktop system, openJDK adoption is on the increase, git usage is on the increase (to a massive 7%; 58% of respondents still use subversion), and that open source participation "seems to be stalled." The reasoning behind that last conclusion was:

Poking holes in this conclusion is not a particularly difficult task. The survey covered Eclipse users, not the open source community or the corporate space as a whole. It asked about policies, but the respondents were mostly developers and designers who may not know what the company's policies really are. Less than 1,500 people responded to the 2009 survey, while nearly 2,000 responded to the 2010 survey, so they are not comparing the same group of people. 41% of the responses came from France and Germany, a proportion which might (or might not) be representative of the Eclipse user base, but which certainly is not representative of the community as a whole. And so on.

Looking at the survey, one could easily conclude that the Eclipse user community is growing; as it grows, it attracts companies that are relatively new to free software. These companies will naturally contribute less than those which have been comfortable with free software for some time. That conclusion is, of course, an exercise in hand waving, but it somehow seems more plausible than the idea of a wave of anti-free-software policies - not seen elsewhere - taking over the corporate world.

As a whole, our community does not appear to be in decline. We have no shortage of healthy projects with active contributor bases. Once upon a time, creating a single full-featured web browser looked like a nearly insurmountable challenge; now we have several of them. One can look at compilers, desktop suites, database managers, or point of sale systems; we have multiple vigorous competing projects in each. Sites like github can spring from nowhere and find themselves hosting millions of repositories in just a few years. In a community with this much activity, coming up with any good understanding of changes in participation rates will be hard. But one does not have to look for long to realize that things can't be in any particularly bad shape.

In this context, it is interesting to look at the last couple of press releases from the Linux Foundation; the LF has announced that LexisNexis and Toyota have become members - Toyota as a Gold member. LexisNexis also released its high-performance clustering system as free software. Neither of these new members can be thought of as a traditional information technology company, but both have found it in their interest to support the development of Linux.

That looks like the future of open source participation. Software companies have, for the most part, been working with our community for some years now. Most computing-related hardware companies are also contributing; there are signs that at least one of the big remaining holdouts will announce a change of policy before the end of the year. Companies like Volkswagen started contributing to Linux as far back as 2007. Increasingly, we are seeing interest from financial service companies, traditional manufacturers, and beyond. Being part of the free software community is not just for software companies anymore, and that is a good thing; it suggests that we'll not see a real decline in participation anytime soon.

Comments (5 posted)

A Firefox user plays with Chromium

Discussions of Chrome tend to run into confusion based on the fact that there are actually two related browsers. Chromium is an open-source (BSD-licensed) project, while Chrome is a binary-only program available for free download. Chromium is the upstream for Chrome; they differ in that Google adds a bunch of proprietary stuff (Flash player, PDF viewer, codecs), an automatic update system, and a more colorful logo to Chrome. Both browsers are available for a number of Linux distributions. Anybody wanting a fully-free system will naturally stick to Chromium.

For a user moving to Chromium from Firefox, there is, at the outset, little in the way of culture shock in store. The Chromium developers seem to have put a great deal of work into making that transition easy. Chromium will pick up a lot of information from an existing Firefox installation, including bookmarks, browsing history, passwords, and more. (As an aside, it's worth noting just how easily Chromium can get its hands on the Firefox password store; any other program can do the same). The appearance is quite similar, and many of the keyboard shortcuts are the same. After a while one begins to notice little things that are missing (the combination of shift and the scroll wheel to move through the history is at the top of your editor's list), but it mostly just works.

Firefox makes a huge variety of configuration options available to users; Chromium has a rather smaller set. Most of the important things are there, but, once again, anybody who has made extensive use of Firefox's configurability will run into annoying gaps. At the top of the "pet peeve" list here is the lack of any ability to control animated images. Your editor is an easily distracted type; text is much harder to read when there are images jumping around on the screen. The "animate once" option in Firefox has always seemed like an ideal compromise; it enables viewing of kitten animations sent by one's daughter while filtering out ongoing obnoxiousness. Chromium users have no such feature.

Also missing is any sort of mechanism for associating "helper" programs with content types. There appears to be no way, for example, to tell the browser to pass a PDF file to evince or an m3u file to the user's choice of media player. As a result, Chromium, out of the box, is totally unable to deal with PDF files; one must install an extension to be able to view them at all. (Chrome has a PDF viewer built into it). This behavior seems to be driven by the ChromeOS use case, where the concept of applications outside the browser is deemed suspicious at best. For a full desktop system, though, it is limiting.

Extensions for Chromium are not in short supply. AdBlock is there, for those who want it. On the other hand, the lack of NoScript hurts; the "NotScript" extension tries to fill that gap, but it's not the same. NotScript setup is bizarre, requiring the user to hand-edit a file named

    ~/.config/chromium/Default/Extensions/\
	odjhifogjcknibkahlpidmdajjpkkcfn/0.9.6_0/CHANGE__PASSWORD__HERE.js

and insert a password which, seemingly, is never used again. NotScript seems to break more sites than NoScript does; the Red Hat bugzilla site, for example, simply refreshes forever with scripts disabled. NotScript also breaks Chrome's PDF viewer unless scripts are enabled for the site hosting the PDF file. There is (it must be said) no direct equivalent to the Firemacs extension providing Emacs keybindings; a similar extension failed to work. Many of these features are apparently harder to implement in Chromium than they are in Firefox; it seems likely that Chromium's emphasis on sandboxing and security, along with an attempt to make extensions portable across releases, may be to blame here.

Various glitches notwithstanding, Chromium is a capable and full-featured browser. It does appear to be quite fast, though Firefox's speed has rarely been a problem in recent times. Having done the work to switch over to this browser and integrate it into his workflow, your editor does not feel any immediate need to switch back to Firefox.

Chromium is promoted as an open source project, but the community has learned that Google often sees "open source" in its own unique way. It would appear, though, that Chromium is actually run like a real open-source development project. The project's code repository contains commits from some 759 developers, most of whom have been active in the last year. Developers tend to use @chromium.org email addresses, making it hard to tell how many of them come from outside Google. The project does give commit privileges to outside developers, though - they are not limited to the submission of patches. Google must certainly maintain a certain degree of control over the direction of the project, but Chromium does truly seem to have a development community of its own.

Despite its free license and growing adoption, Chromium tends to be supported reluctantly by many distributors. The project's release cycles are unclear at best, and its practice of forking and bundling libraries does not sit well with distributors; see this posting from Tom Callaway for a long discussion on the disconnect between Chromium and distributors. Chromium has an open bug tracker entry on making the project more distributor-friendly, but it seems to have more cobwebs than contributions. For reasons that have been extensively discussed over the years, web browser projects seem to have a hard time fitting into the distributor ecosystem.

Even so, there are repositories for a number of common distributions. Some work better than others; the Fedora repository does not support Rawhide, for example. But just about anybody wanting to run Chromium without building it (a daunting process which requires a 64-bit machine just to have the address space to do the link) on Linux can do so. That said, it's probably a fair guess that an awful lot of Linux users are running the proprietary Chrome releases. One should never underestimate the allure of a working YouTube. For those who would like to take that path, there are a number of "release channels" with varying distances from the bleeding edge.

To conclude: Chromium is a capable tool which has brought an interesting new level of competition to the browser space. The project's emphasis on speed and security are certainly welcome, as is the relatively open (for Google) nature of the project itself. On the down side, one might well wonder whether it is wise to put yet another piece of web infrastructure into a single company's hands. Google's intentions seem to be good now, but, as we've often seen, companies can change alignment overnight. So while Chromium is a welcome option to have, it might be best if it does not take over. The continued existence and success of strong competitors in the free software community can only be a good thing.

Comments (126 posted)

Prey: Open source theft recovery

Chances are, you know someone who has lost a laptop or smartphone, or had one stolen outright. Although preventing the theft itself would be the most satisfying outcome, there is little a software developer can do on that front (although one might consider proximity-monitoring over Bluetooth to be a step in the right direction). Full-disk encryption and password-protection for the BIOS are common strategies, but another approach — device tracking — has never enjoyed a high level of popularity in open source. The Prey project offers one open source device tracking solution, with a small range of options — including self-hosting or using the company's online monitoring service.

Device tracking options

All software-based device tracking systems share one common element: they periodically record data about the machine and its surroundings, including location, IP address, and network settings, or perhaps even a photo taken with an on-board camera. Where they differ is in how and when that information is transmitted to the device owner, and how the application detects whether or not it has been lost or stolen. Some upload their periodically-collected data sets to a remote server on every run, others simply "phone home" to perform a status check, and only record and report information if they get the appropriate signal.

Proprietary solutions like GadgetTrak or LoJack for Laptops tend to rely on a central server run by the software provider, who waits for the device owner to report the loss or theft. The first widely-deployed open source device tracker was Adeona, which took a different approach: it uploaded its reports to a remote storage location at every snapshot period. That eliminated the need to maintain a central database of tracked devices, which is what the proprietary companies charged their recurring fees to maintain. Sadly, Adeona is currently offline, as the storage service used, OpenDHT, has been deactivated.

Prey operates more like the proprietary services. The default behavior is for the Prey script to wake up periodically and make an HTTP request to a preconfigured URL. The response code sent tells the script what to do next: a 200 (the "OK" status code) will shut the script down normally. Only a 404 "Not Found" code (which requires making a successful connection to the web server) will trigger the script's theft response. Prey can be configured only to send reports about its status and whereabouts, or to trigger local actions, such as an audible alarm, flashing an alert notification on screen, or locking the system.

Fork Ltd., the company behind Prey's development, offers two administration options. Users can configure Prey to use the "Prey Control Panel" site as the pre-shared URL, or to run in standalone mode. Control Panel users can configure device-reporting settings and actions through the web site, as well as mark a device as missing (which removes the URL requested by the script's heartbeat-checking routine) and view any reports it sends. Standalone users must manually edit the Prey configuration file, supplying their own heartbeat URL, and provide valid SMTP and email configuration details — in standalone mode, the script emails its reports rather than posting them to the Control Panel server. Fork recently started offering paid "Pro" accounts, which allow simultaneous tracking of more devices and additional configuration options, but says it will always offer free accounts.

Stalking Prey

The current release of Prey is version 0.5.3, and is available for Linux, Mac OS X, Windows, and Android. An iPhone/iPad version is presently in private beta testing. On Linux, the application itself is a collection of Bash scripts and support files (such as the audio file played as an alarm). The source bundle can be installed on any modern distribution, requiring only the set up of a cron job to run the main script at the desired interval. It depends on cURL to make its HTTP requests, the streamer utility from xawtv to capture webcam images, and Python to run a small GUI configuration tool. There is also a Debian package available on the download page, which is compatible with Debian and Ubuntu distributions.

The actual setup process is simple, but the Prey project's web site suffers from severe documentation drought. There are no step-by-step guides suitable for new users (much less for configuring Prey in standalone mode): the necessary information is split up between the README file in the code bundle, the FAQ page, the "Knowledgebase," and the mailing list. If all you are interested in doing is signing up for the free Control Panel service, you will probably muddle through without too many hiccups, but I found it difficult to track down the answers to many of the questions I would want to know before I embarked on installing it.

For example, the site's splash page, FAQ, and README all discuss adding a "valid web URL" as the means of enabling Prey in standalone mode, but the example they give is a URL on one's own domain or blog. In the prey configuration file, there are no client-side settings to control the actions (alarm, lock, etc.) performed or the data gathered when Prey sends a report — all of which are settings exposed on the Prey-hosted Control Panel site. If you read through the mailing list archives, though, you discover that the URL Prey checks for is intended not merely to return a valid HTTP response code, but to serve up an XML configuration file. There is still no documentation for the syntax and options of the file, but evidently if one first sets up an account with the Prey service, one can download and save the file to then use later in standalone mode.

I would also like to see more thorough disclosure of what machine information Prey sends to the Control Panel server. Because the Prey script makes an HTTP request, presumably a fair portion of the information could simply be gathered from the request headers, but any time an application "phones home" I prefer to see what it is going to say before I install it the first time. After initially adding a device, the Control Panel has a detailed hardware information section disclosing more about the system configuration than I generally like strangers to be able to access. For non-Pro accounts, the connection to the Control panel site is not sent over SSL, which makes it vulnerable to eavesdropping. There are scattered references to Prey using Amazon S3 to store its webcam report snapshots, rather than sending them directly to the Prey site, but here again the documentation is sparse.

From a user standpoint, though, Prey works quite well. On Linux, the Bash scripts run with root privileges, which might concern those of a high-security mindset. Prey can gather an impressive set of information about your machine for the reports it sends back if the machine goes missing. In addition to webcam snapshots and the standard geolocation and WiFi-triangulation options, it can complete a traceroute, capture a desktop screenshot, log any recently-modified files, and log running applications.

The Control Panel interface is easy to understand, with newbie-friendly explanations of the various settings and options. Even the setup tool does a good job of outlining the difference between Control Panel and standalone mode. I am not sold on the value of playing an MP3 alarm file, but the ability to lock down the currently-running session (with password-protection) is a necessity when dealing with theft. The Prey team also deserves kudos for explaining the value of encrypting the hard disk and password-protecting the BIOS; those measures may be the only means to prevent attackers from wiping a machine entirely.

If you are not patient enough to wait until someone steals your laptop, you can dry-run Prey by running the /usr/share/prey/prey.sh script with the --check option. This will test the network connection, the crontab entry, and the validity of the API key and device ID for your hardware. Testing actual report generation is harder; for non-pro accounts the only option appears to be marking your device as "missing" in the Control panel interface.

Prey forward technologies

Taking a look at the configuration file (which on Linux systems is installed at /usr/share/prey/config), there are some tantalizing options that do not yet seem to be active in the 0.5.3 code, including SSH tunneling and SCP/SFTP file transfer, both with support for RSA key-based authentication instead of passwords. There are also some experimental options for modifying the HTTP request in order to get around firewalls and other attempts to block Prey. Here, too, my hope is that we will see further details documented on the site, rather than have them buried in the comments of the configuration file alone.

One final concern for potential Prey users is whether the three-device limit imposed on non-Pro accounts is genuinely enough. Even a single individual is likely to hit the three-device limit these days, and although each member of a family could set up his or her own Control panel account, I suspect that in most families a single family member will end up bearing the burden of keeping tabs on the portable electronics. The limit will, of course, help sell Pro subscriptions — or push users into setting up their own standalone server.

To me, that makes the need for better documentation of standalone mode configuration all the more important. While an enterprising developer could craft a homemade equivalent to the Prey Control Panel and serve up a valid XML file to clients, all indications are that the Prey project wants to operate as a true open source citizen — the source code is available on GitHub and the mailing list is public. It just has a little bit further to go.

If you are looking for an open source alternative to the proprietary device tracking products on the market, Prey is worth examining. Because it is based on standard Linux system tools, it ought to run on any distribution — perhaps even including Maemo and MeeGo handsets, although they do not have Bash installed by default. You just might have some hoops to jump through if you are intent on running your own device-tracking server.

Comments (22 posted)

Page editor: Jonathan Corbet

Security

A decline in email spam?

One of the biggest internet irritants over the last decade or two clearly has to be email spam. It has collectively taken billions of hours of users' time to deal with, consumed countless terabytes of wasted disk space, burned bandwidth better spent on kitten videos, and used up vast quantities of developer time to come up with new ways to filter it out or come up with other technological fixes. So, recent reports that email spam is in decline are certainly welcome, if true, but even with the 90% decline over the last year that is being reported, the amount of spam being sent is still staggering—and likely to be with us for a long time to come.

I haven't heard friends and colleagues extolling a reduction in the amount of spam they receive but, as they say, the plural of anecdote is not data. One would think that such a precipitous drop would be noticed by email users, however. In any case, Cisco, Symantec, and others are reporting numbers like 34 billion spam emails per day for April, down from 300 billion in mid-2010. That's an enormous drop in the volume, even if 34 billion a day is still huge. Without any hard data to the contrary, some significant drop-off in spam volume is a reasonable conclusion—and one worth exploring a little bit.

Spam has always been driven by its economics. In the early days, it cost almost nothing to send out huge volumes of email, and the chances of getting caught and meaningfully punished were quite small. That led to various "spam kings" who made outrageous amounts of money by spamming the world. If sending spam is, for all intents and purposes, free, you don't need a very high response rate to the pitch in order to bring in substantial sums. But that led to a backlash.

Users quickly tired of digging through email that was 90-100% spam, ISPs got smarter about not allowing their systems to be used for spam transmission, and, eventually, governments decided to ramp up the punishment side of the equation. Spam filtering became ubiquitous, blacklists that identified sites sending spam started to pop up, prosecutions of those sending spam were successful to some extent, and so on. The cost of sending spam has risen substantially over the years.

That's not to say that there aren't some folks still making lots of money sending spam, but these days there are bigger phish (so to speak) to fry. The most lucrative schemes today don't rely on sending enormous volumes of email and are more targeted instead.

It would be nice to think that users are getting a bit more sophisticated—or just running out of body parts to enlarge. It's hard to say whether that's true or not, but, even with the growth in new internet users, one might hope that the negative publicity about internet scams is making users more wary. Unfortunately, one doesn't have to search very far to find a news item about someone taken in by email claiming to be from a foreigner who wants to send them "EIGHT BILLION DOLLARS". So, it's probably overoptimistic to attribute much of the spam volume drop to users being less likely to respond to the pitch.

Filtering has certainly gotten better over the years, and moved from something users had to fiddle with to "the cloud" (or at least their ISP). Spammers have routinely run their emails through tools like SpamAssassin to try to evade filters, but there are limits to that approach, especially when individual Bayesian filters are factored in. It's difficult for even gullible users to respond to a spam pitch they don't see, so filtering has likely done much to reduce the effectiveness of spam.

Another factor that may be at play here is that many folks have moved beyond email for much or all of their communication. Text messages, instant messaging, and the services provided by various walled gardens (e.g. Facebook, Twitter) have replaced email for a lot of people, especially those darn kids, these days. Spam has, of course, evolved to assail those media as well. That kind of spam is not reflected in these recent statistics, however.

So, while it is somewhat heartening to hear that some folks are probably receiving less email spam, it's unlikely that it's really going to change things for most people. Users will still need filtering, ISPs and governments will still need to be vigilant, and clicking on links in dodgy email will still be a bad idea. While likely mind-numbing, seven days of reading all the email you receive might also prove somewhat eye-opening.

Like it or not, spam has become part of our culture. From the origin of the "spam" name to the various terms for different kinds of spam (419 spam, phishing, etc.), spam has used and been used by internet culture. Over the years, various folks have imagined horrible demises for spammers—e.g. Rule 34—usually involving the products they pitch in some bizarre fashion. So, at least we can get a chuckle from spam now and again, even as it is an extremely annoying—sometimes dangerous—phenomenon. In fact, it would be nice if junk (snail) mail filters were even half as good as email filters are these days.

Comments (42 posted)

Brief items

Security quotes of the week

Comments (5 posted)

Vsftpd backdoor discovered in source code (The H)

Comments (37 posted)

Top 25 Most Dangerous Software Errors

Comments (none posted)

New vulnerabilities

bind: denial of service

Comments (none posted)

dokuwiki: cross-site scripting

Comments (none posted)

feh: arbitrary file overwrite

Comments (none posted)

krb5-appl: privilege escalation

Comments (none posted)

lftp: man-in-the-middle vulnerability

Comments (none posted)

libvoikko: denial of service

Comments (none posted)

NetworkManager: privilege escalation

Comments (none posted)

nfs-utils: authentication bypass

Comments (none posted)

OpenSSH: private key disclosure

Comments (1 posted)

packagekit: incorrect package signature check

Comments (none posted)

php: arbitrary file creation/overwrite

Comments (none posted)

qemu-kvm: arbitrary code execution

Comments (none posted)

qemu-kvm: privilege escalation

Comments (none posted)

rubygem-activesupport: cross-site scripting

Comments (none posted)

syslog-ng: denial of service

Comments (none posted)

tftp: buffer overflow

Comments (none posted)

weechat: man-in-the-middle attack

Comments (none posted)

wordpress: privilege escalation

Comments (none posted)

xen: privilege escalation

Comments (4 posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 3.0-rc6, released on July 4. It has a new Intel isci driver which adds a significant chunk of code, but, otherwise it's basic fixes. "It's getting to the point where I'm thinking I should just release 3.0, because it's been pretty quiet, and the fixes haven't been earth-shakingly exciting." See the full changelog for all the details.

Stable updates: no stable updates have been released in the last week, and none are in the review process as of this writing.

Comments (3 posted)

Quotes of the week

Comments (none posted)

What are they polling for?

    unsigned int (*poll) (struct file *filp, struct poll_table_struct *pt);

This function returns a value indicating whether non-blocking I/O is currently possible; it is also expected to add a wait queue to the "poll table" (pt) passed in. If no file descriptors are ready for I/O, the calling process will block on all of the accumulated wait queues.

poll() has long implemented an optimization: if an early poll() function indicates that I/O is possible, the kernel knows that it will not be blocking the calling process. So it stops accumulating wait queues; this state is indicated by passing a null pointer for pt. That all works well except in one case: what if a driver needs access to some of the information stored in the poll table?

In particular, the driver might want to know whether the caller is interested in readiness for read or write access, or whether it is looking for exceptional events. For example, if the application wants to read from the descriptor, the driver may need to fire up some device machinery to make that possible. This situation has not come up very often, but it does tend to affect Video4Linux drivers. In response, Hans Verkuil has posted a patch slightly changing the way poll() works.

With the patch, the poll table is never passed as null; instead, the "we will not be blocking" case is marked internally. So the set of events requested by the application is always available; Hans has provided a helper function to access that information:

    unsigned long poll_requested_events(const poll_table *p);

There has been little discussion of the patch; it doesn't seem like there is any real reason for it not to go in for 3.1.

Comments (none posted)

Kernel development news

Seccomp filters: No clear path

Patches to expand the functionality of seccomp ("secure computing") have been floating around for two years or more without making any real progress into the mainline. There are a number of projects that are interested in using an expanded seccomp, but the patches themselves seem to have run into a "catch-22" situation. There are conflicting visions of how the feature should be added, without a clear sense that any of the options will be acceptable to all of the maintainers involved. That leaves a useful feature without a clear path into the kernel, which is undoubtedly frustrating to some.

We first looked at seccomp sandboxing a little over two years ago, when Adam Langley posted patches that would provide a way for a process to restrict the system calls that it (and its children) could make. The idea is to allow processes to sandbox themselves by choosing which system calls are available, rather than being restricted to just the four hard-coded system calls that the existing seccomp implementation allows (read(), write(), exit(), and sigreturn()). The impetus behind Langley's patches was to provide an easier mechanism for sandboxing processes in the Chromium web browser—and to eventually remove the somewhat convoluted sandbox that Chromium currently uses on Linux.

At the time of that proposal, Ingo Molnar suggested that Ftrace-style filtering would make the expanded seccomp much more useful. That idea wasn't universally hailed at the time, and the seccomp feature went mostly dormant until it was restarted by Will Drewry back in April. Drewry took Molnar's suggestions and implemented a version of seccomp that would allow system calls to be enabled, disabled, or filtered with simple boolean expressions (e.g. sys_read: (fd == 0)).

While Molnar was pleased with the progress, he didn't think it went far enough and suggested that a perf-like interface be used instead of prctl(), which is used by the existing seccomp. He had some fairly wide-ranging ideas that using perf events in a more active way could lead to better kernel security solutions than the existing Linux Security Modules (LSM) approach provides. Once again, this idea was not universally popular. The LSM developers, in particular, were not enamored by that idea.

Nevertheless, Drewry implemented a proof of concept along the lines of what Molnar had suggested. That led to complaints from a somewhat surprising direction, as both Peter Zijlstra and Thomas Gleixner strongly objected to perf being used in an active role. Their responses didn't leave room for any middle ground, with Zijlstra, who is one of the perf maintainers along with Molnar, saying that he and Gleixner would NAK "any and all patches that extend perf/ftrace beyond the passive observing role".

All of which led Drewry, who must be feeling a bit whipsawed at this point, to return to the patchset that seemed to have the most support: using Ftrace/perf-style filters, but maintaining the prctl() interface that is currently used by seccomp. Linus Torvalds had expressed some skepticism that the feature would have any real users, but Drewry outlined how it would be used by Chromium, and several other developers spoke up in favor of expanding seccomp, saying that QEMU, Linux containers (LXC), and others would use the feature. Those endorsements, along with resolving some other technical concerns, was enough for Torvalds to remove his objection to the feature. But, as might be guessed, Molnar is still not satisfied with the approach.

When Drewry reposted the patchset toward the end of June, and asked what the next steps were, Molnar noted that his concerns were not being addressed: "You are pushing the 'filter engine' approach currently, not the (much) more unified 'event filters' approach." But Drewry is trying to find a balance between the needs of the potential users, other maintainers, and Molnar's requests, which is somewhere between difficult and impossible:

But Molnar is adamant that the "filter engine" approach is short-sighted, citing the diffstats of the various implementations as evidence:

       bitmask (2009):  6 files changed,  194 insertions(+), 22 deletions(-)
 filter engine (2010): 18 files changed, 1100 insertions(+), 21 deletions(-)
 event filters (2011):  5 files changed,   82 insertions(+), 16 deletions(-)

But, as Drewry points out, there is still a lot of work to be done to get beyond the proof-of-concept and to a fully fleshed-out solution. Given that the approach has already received several NAKs, doing all of that work has a very uncertain future. Drewry would like to see the feature be available soon, and is concerned that working on the larger problem is likely to delay that significantly, if it can ever get beyond the objections: "If all the other work is a prerequisite for system call restriction, I'll be very lucky to see anything this calendar year assuming I can even write the patches in that time."

Molnar is undeterred, however, suggesting that there is a path into the kernel through the tree that he co-maintains:

The problem, of course, is that the 5% is the piece that Drewry and others are most interested in seeing (i.e. the system call restrictions for sandboxing) in the kernel. So, what Molnar seems to be offering is a fairly sizable chunk of work that could, in the end, still leave the "interesting" part out in the cold. Molnar may be confident that he can overcome the objections from Zijlstra and Gleixner, but Drewry can hardly be as sanguine. He describes the problem as he sees it:

Both Zijlstra and Gleixner have been absent from the most recent discussion, so it's a little hard to guess what their thoughts are. In the absence of any kind of posting softening their stances, though, it would be a bad idea to believe that they have changed their minds.

It's a problem that we have seen before, where a new feature is, to some extent, held hostage to requests that a larger problem be solved. The problem was discussed at the 2009 Kernel Summit, where there was agreement that those requests should be advisory in nature, rather than demands. In this case, Molnar is not really demanding that the bigger task be done, just that he is uninterested in taking the code via the -tip tree unless it solves the larger problem.

It is unclear where things go from here. Drewry said that he would look at trying to do things Molnar's way ("but if my only chance of any form of this being ACK'd is to write it such that it shares code with perf and has a shiny new ABI, then I'll queue up the work for when I can start trying to tackle it"), but it may be a ways off. In the meantime, there are various projects interested in using the feature.

If falling back to the bitmask version of the feature solves enough of the problem for those projects, there is the possibility of trying to get that into the kernel via another tree (e.g. the security tree). There would undoubtedly be objections from Molnar, but if enough users lined up behind it, that might be a reasonable approach. It would create an ABI that would need to be maintained going forward, which is one of Molnar's objections, but it would solve problems for Chromium and others.

Steven Rostedt suggested adding the seccomp expansion as a discussion item for the Kernel Summit in October, which might provide a path forward. It's likely that most or all of the interested parties will be there (unlike the Linux Security Summit that will be held with Plumbers in September, which was suggested as an alternative). While a face-to-face discussion could be helpful, it might be a stretch to believe that the disagreement between active vs. passive perf could be resolved that way. On the other hand, it could lead to some kind of decree about the proper direction from Torvalds. That could go a long way toward resolving the issue.

Comments (1 posted)

CMA and ARM

Over the years, LWN has also looked at ARM's special memory management challenges. Recent ARM CPUs are, like those implementing other architectures, becoming more complex in order to improve performance. So ARM processors can now do speculative prefetching of memory contents in surprising ways. This prefetching works well on cached memory, but should not be used on memory that has been marked as uncached. An additional complication comes from the fact that virtual memory systems can have more than one mapping for a given range of memory, and caching is a feature of the mapping, not the memory itself. So one might well wonder what happens if different mappings have different caching attributes. On recent ARM processor designs, what happens is officially undefined; in practice, it can mean problems like corrupted memory, machine checks, or simple hangs. As it happens, kernel developers normally go out of their way to avoid that kind of behavior.

The current CMA mechanism is used as an allocator behind dma_alloc_coherent(), which creates a cache-coherent DMA buffer. In the absence of bus-snooping hardware that is able to notice when a DMA transfer changes memory, "cache-coherent" is likely to mean simply "uncached." So CMA must, on such systems, create an uncached range of memory to hand back to the requesting driver. That is easily done, and all should be well...at least, unless there happens to be another mapping to the same memory with different caching attributes.

Unfortunately, conflicting mappings can come about easily on a Linux system. One of the first things the kernel does as it boots is to create a "linear mapping" which provides kernel-space virtual addresses for most or all of the memory present in the system. The kernel cannot manipulate memory directly without such a mapping; putting as much of memory as possible into a persistent mapping thus makes sense. On a 32-bit system, just under 1GB of memory can be mapped this way (64-bit systems can always map all of memory and will be able to do so for quite some time yet). This kernel-mapped memory is called "low memory"; almost all allocations of memory for the kernel's use come from the low memory area. Naturally, low memory is mapped with caching enabled; to do otherwise would destroy the performance of the system. If a region of low memory is turned into a DMA buffer with an uncached mapping, the system will have two conflicting mappings for the same memory and will have moved into "undefined behavior" territory.

These conflicting mappings are the reason behind ARM maintainer Russell King's strong opposition to the merging of CMA in its current form. He believes that the code is unsafe on ARM systems; it should not, he says, be merged until the mapping problem has been solved. The interesting thing is that the existing DMA API has the same problem on ARM; dma_alloc_coherent() uses vanilla alloc_pages() to obtain a buffer, then changes the caching attributes before giving the buffer back to the caller. The addition of CMA does not make ARM's DMA API any more or less safe than it was before; it just perpetuates an existing problem.

Russell has a patch pending for 3.1 which addresses this problem by setting aside a chunk of memory which is never mapped into the kernel's address space. With this memory pool available, coherent DMA mappings can be set up without endangering the operation of the system. The whole reason CMA exists, though, is to provide large, contiguous buffers without the need to set aside memory; Russell's approach thus defeats the entire purpose. The pressures which have led to the creation of CMA will not go away anytime soon, so it seems that another solution is needed. Arnd Bergmann has outlined two possibilities, neither of which is entirely pleasant:

• CMA could be changed to only allocate from the high memory zone. High memory is (by definition) not in the kernel's linear mapping, so no other mappings should exist. The problem with this approach is that it forces the use of high memory on all systems; ARM-based systems are reaching the point where some of them need high memory anyway, but that need is not, yet, universal. Getting enough memory into the high memory zone to be useful could require moving the boundary and shrinking low memory; that is not desirable because low memory is often a limiting resource already. Even if that obstacle can be overcome, the ARM architecture poses unique challenges which would make a high memory implementation hard.

• Memory that has been turned into a coherent DMA buffer could simply be removed from the kernel's linear mapping until the buffer is no longer needed. This approach seems simple until one remembers that the kernel uses huge pages for the linear mapping. Splitting those huge pages into smaller pages would increase translation lookaside buffer (TLB) contention, reducing the performance of the system as a whole.

Compared to these alternatives, simply setting aside a chunk of memory at boot time might not look like such a bad idea after all. CMA developer Marek Szyprowski's plan appears to be to go with the second of those two alternatives; he thinks that it can be done without significantly hurting performance.

In truth, the best tradeoff will almost certainly differ from one platform to the next. In some situations, memory will be tight enough that a significant runtime penalty to avoid making static DMA buffers seems worthwhile; on others, setting aside a bit of memory may not be a real problem. So what may come of all this is a set of choices to be made when configuring a kernel. There does not appear to be a single solution which just works for everybody on the horizon at this time.

Comments (1 posted)

Deferred driver probing

The problem with the camera driver is a result of the fact that the "camera" is, in reality, three devices working in concert: a DMA bridge, a sensor, and an I2C bus connecting the two. The bridge (which plays the role of the overall "camera driver") must locate and identify the sensor as part of its setup routine; if the sensor does not exist, initialization will fail. But the sensor will not exist until its driver and the I2C bus driver have been loaded into the system. If all of the drivers are built into the kernel, the bridge driver's probe() function may be called first; there will be no sensor, so everything fails.

Contemporary systems - especially those of the mobile variety - are increasingly built this way. Grant gave another example:

The key point to understand is that the various components that make up a "device" may appear to be entirely unrelated at the hardware level. They can be on different buses; some of them may be subcomponents of entirely different devices. A general-purpose kernel has no real way to know what the real dependencies between devices are until all of the pieces are present and have started to recognize each other.

Grant's patch takes a simple approach to solving this problem: drivers which are unable to initialize their devices as the result of missing resources can request that the operation be retried at some point in the future. That request is a simple matter of returning -EAGAIN from the probe() function. The driver core maintains a simple linked list of drivers that have requested this sort of deferral; when the time seems right, the deferred probe() invocations are retried to see if things work any better.

One of the concerns raised with regard to this patch had to do with the determination of the right time. How might the kernel know when a failed initialization might work? The event which may change the situation is the successful addition of a new device to the system, so the current patch retries all of the deferred calls every time a new device shows up. The mechanism used for the retries (a workqueue) will tend to coalesce these attempts when a lot of devices are being registered (during system boot, for example), but it still strikes some reviewers as being inefficient. Grant has promised a revision of the patch which improves the situation.

A related question is: when can the kernel conclude that there is no longer any point in retrying a specific driver's probe() function? In today's dynamic hardware environment, there never comes a point where one can say that no more devices will show up. This question has no real answer; it could be that, in a poorly configured or broken system, the process will never terminate. The cost of a driver stuck in the deferred state should be small, though.

Others have questioned the need for this mechanism at all, but the responses have made it clear that something needs to be done to address this kind of hardware. A proper solution in the driver core seems like a better answer than a bunch of one-off hacks in specific drivers. So something will probably go in.

Someday perhaps we will see a more elegant and efficient mechanism. One could imagine an API allowing a driver to specify which resources it is looking for; that driver's probe() function would then be put on hold until those resources become available. The driver core already generates events when new devices become available; some code matching those events to waiting drivers could be the last piece. But there would be a need to come up with a language by which a driver could express a need like "a device at address 42 on this I2C bus"; getting that right could take some work.

Meanwhile, Grant's patch offers a "good enough" solution which appears capable of solving the problem most of the time. Accepting "good enough" when it's truly good enough is a key part of pragmatic programming. So chances are we'll have deferred driver initialization in the kernel sometime soon; fancier mechanisms may be rather longer in coming.

Comments (4 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.0-rc6 .. . (July 4, 2011)

Core kernel code

• Hans Verkuil: poll: add poll_requested_events() function . (July 1, 2011)

• Paul Turner: CFS Bandwidth Control v7.1 . (July 7, 2011)

Development tools

• Ankita Garg: mm,debug: VM framework to capture memory reference pattern . (July 5, 2011)

• Akinobu Mita: notifier error injection . (July 5, 2011)

Device drivers

• Mark Brown: regmap: Generic I2C and SPI register map library . (June 30, 2011)

• Sascha Hauer: implement a generic PWM framework . (June 30, 2011)

• Christopher Heiny: input/touchscreen: Synaptics RMI4 Touchscreen Driver . (July 1, 2011)

• Grant Likely: drivercore: Add driver probe deferral mechanism . (July 4, 2011)

• Manjunath Hadli: RFC for Media Controller capture driver for DM365 . (July 4, 2011)

• Alan Cox: GMA50 series update . (July 5, 2011)

• Uwe Kleine-König: media/at91sam9x5-video: new driver for the high end overlay on at91sam9x5 . (July 5, 2011)

• Rafael J. Wysocki: PM / Domains: Generic PM domains improvements . (July 7, 2011)

Filesystems and block I/O

• Konstantin Khlebnikov: cfq: request-deadline policy . (July 5, 2011)

• Allison Henderson: EXT4: Secure Delete . (July 5, 2011)

• Arne Jansen: btrfs: generic readeahead interface . (July 5, 2011)

• Li Zefan: Btrfs: initial online fsck support . (July 5, 2011)

• Theodore Ts'o: New spin of the ext4 bigalloc patches . (July 7, 2011)

Memory management

• Minchan Kim: Prevent LRU churning . (June 30, 2011)

• Marek Szyprowski: Contiguous Memory Allocator . (July 5, 2011)

• Dan Magenheimer: [PATCH v2] staging: zcache: support multiple clients, prep for KVM and RAMster . (July 5, 2011)

Architecture-specific

• Vincent Guittot: Add ARM cpu topology definition . (July 1, 2011)

• Lorenzo Pieralisi: ARM: common idle infrastructure . (July 7, 2011)

• Jonas Bonn: OpenRISC Architecture: Patch set version 2 . (July 5, 2011)

• Lin Ming: perf: Intel uncore pmu counting support . (July 5, 2011)

Security-related

• Tetsuo Handa: TOMOYO 2.4: Conditional ACL support . (July 5, 2011)

Virtualization and containers

• Glauber Costa: Steal time for KVM . (July 5, 2011)

Miscellaneous

• Stephen Hemminger: iproute2 2.6.39 . (June 30, 2011)

• Ian Kent: autofs 5.0.6 release . (June 30, 2011)

Page editor: Jonathan Corbet

Distributions

DoudouLinux: You know, for kids

The more Linux distributions that pop up, the more skeptical one can become of those that seek to "tailor" the platform for a specific subset of users because, too often, that tailoring process amounts to little more than applying the project leader's personal set of cosmetic preferences and default application choices. That is not the case for projects that attempt to craft a distribution for an entirely different demographic, though, such as children. Most "Linux for kids" distributions seem to target more or less the same space, but one, DoudouLinux, is taking a markedly different approach.

What Linux?

For starters, the distribution's name is derived from doudou, a French idiomatic word that seems to roughly mean "child's security blanket." But it may still elicit giggles from English speakers, particularly the extremely young users it sets out to serve. If you can get past that, however, DoudouLinux does offer a different take on the children's distribution niche.

There are several active kid's Linux projects today — Edubuntu, Qimo, OLPC Sugar, Foresight Kids, etc. — but most of them focus either on classroom usage or on school-age children, meaning those old enough to be comfortable using the computer to write or do general-computing tasks. There is a gray area of course, but think mid-to-late primary school at the youngest. DoudouLinux, on the other hand, is tailored towards children in the two to seven year old age range, for use at home. The project's site explains its purpose as being a home-friendly, more educational alternative to a game console plugged in to the television.

One can feel the difference in at least three ways. First, classroom-oriented distributions usually have to cope with remote management and administration issues, often using remote desktop software, and they expect the computer to be directly connected to a LAN, perhaps with access to a storage-area network. Some are even traditional "thin-client" designs. DoudouLinux dispenses with all of that. It is designed to run from live CD or live USB storage, offers minimal system configuration or management options, and the majority of its software will run without a live Internet connection.

The second difference is that "school age" kids have different application requirements. As they get older, the office suite becomes more important, and what qualifies as "educational" applications becomes more specialized, such as focusing on math or science and less on games. DoudouLinux's application offerings do not include a word processor or spreadsheet at all, but incorporate plenty of interactive and tactile-experience learning tools.

Third, extremely young children like those on the early end of DoudouLinux's audience simply do not have the fine motor skills to use a mouse or keyboard with standard-size icons, and those who are not reading yet might be lost entirely. Those requirements affect the choice of applications, but also require changes to the window system as well as the keyboard and mouse settings. DoudouLinux provides parents with a set of tools to configure the system settings to support a range of ages and mouse-finesse levels.

Course catalog

DoudouLinux's latest stable release is code-named Gondwana, from June of 2011, which you can download in CD or USB-oriented images from the project's web site. There are images in 16 languages (including two in Serbian, for both the Latin and the Cyrillic alphabets) from the stable series, and six additional languages in the "awaiting" section, which may have outstanding translation issues. There is also a PDF "Quickstart" manual in each of the stable languages. Downloading can take a while, even on a fast network connection, because the project does not have mirrors set up.

The images are built for 32-bit Intel-compatible processors only; to its credit the project does an admirable job of explaining what generations of Apple hardware this includes, as well as giving a rough overview of Linux and free software concepts in general. The principle difference between the CD and USB images are that the USB images are pre-configured to use part of the flash drive as persistent storage. DoudouLinux allows a small amount of system customization that is helpful to be persistent between sessions.

At the moment, DoudouLinux is intended primarily to run from such removable storage devices, although there is documentation for installing it to a hard drive as well. The roadmap indicates that future releases hope to add support for booting within Windows, a la Ubuntu's Wubi.

DoudouLinux is based on a minimal Debian "Lenny" (5.x) system, with a custom, tab-based user interface. At boot time, there is an "Activities Menu" displayed that offers seven session options: the full DoudouLinux system, a "Mini DoudouLinux" session with a reduced set of applications offered, and single-application sessions for several of the applications: Gamine, Pysycache, Childsplay, TuxPaint, and GCompris.

Gamine and Pysycache are both mouse-training games, aimed at different levels of experience. They hail from DoudouLinux's educational games section, which also includes language and geography titles, plus the general-purpose educational suites Childsplay and GCompris, each of which offers a variety of activities.

The other application categories include "Work" tools (a category that includes the web browser, IM software, dictionary, and calculator), "Multimedia" (which includes several music-creation applications and the Stopmotion animation tool), and general games (under "Enjoy"). The general games category includes standard puzzle and non-violent arcade fare, plus common card and board games. Each of the categories has its own tab in the DoudouLinux application launcher, which responds to single-clicks for better ease-of-use. Likewise, every application or game runs in full-screen mode, as the developers felt that window management was asking for trouble with children.

Few if any of the application offerings in DoudouLinux Gondwana are not available in other Debian-derived distributions. The real customization work involved building the activities chooser and the various configuration tools, all of which are available under the GPLv3. The configuration tools are found under the "Tune" tab in the full DoudouLinux session. They allow sound, monitor, printer, and data persistence control, as well as the separate administration tools for GCompris and Pysycache, configuration of the Activity chooser, and a bug reporting tool.

Although the configuration tools are intended for use by parents, they are simple to use — offering access to presets with one click. Mouse and monitor settings are usually foolproof on a modern Linux system, but there is documentation online for dealing with troublesome printers.

While it is not configurable from the live system, DoudouLinux runs the DansGuardian web-filtering system in the background. DansGuardian is a keyword-based content filter, not a site blacklist, so it is advertised as doing a better job of insulating children from accidental adult content delivered through search results and other side-bands. The filter is certainly restrictive in the form offered by DoudouLinux, even blocking Google Image Search results for many innocent common words to a baffling extent (for example, "cow" is permissible, but "puppy" is not). In theory, young children will probably not spend too much time in web searches, but without parent-accessible configuration tools, DansGuardian is a distinctly "when in doubt, block it" tool.

Report card time

Overall, DoudouLinux "Gondwana" is a worthy choice for families with very young children. The content that it offers is not much different than you would find in another "educational" distribution, but there is far less configuration to do, and because the system is limited in scope, it runs fast. The choice to offer it as a live-CD image only also strikes me as wise, given the inherent risks that sitting a toddler in front of a keyboard can entail. DoudouLinux cannot protect your hardware from spilled milk or the sudden disconnection of the flash drive, but then again no piece of software can.

As for the applications themselves, I am probably not a good judge of young children's learning tools, since I do not have kids myself. I am a bit skeptical that the "bright colors, cartoon characters everywhere" motif is really necessary, because I see all too many kids who are capable users of their parents' smartphones. But I do think it is commendable that the DoudouLinux team included some less-common choices in its application suite, such as the music creation tools. Too many distributions' "music" category includes only an audio player; DoudouLinux gives the kids a piano keyboard and the Hydrogen drum machine. Which probably makes the volume control tool all the more critical.

That said, I was a bit surprised not to find a typing tutor like Tux Typing in the menu, nor to find any webcam tools like GNOME's Cheese. The Empathy instant messenger client is included, which presumably would need to be configured by a parent. Video chat with grandparents is probably the intended usage, but it does stand out as needing a distinctly more complicated setup process.

The DoudouLinux project is not resting on its laurels. The roadmap includes several technical challenges: unifying the CD and USB images into a single file with user-selectable persistence, a more flexible activity chooser, and an online tool to customize the CD image. The project's issue tracker also indicates that it plans to migrate to Debian "Squeeze" (6.x), as well as the ever-present need to add more languages and more complete translations. At least one new feature is being discussed: a parental control to limit use of the computer to specific times of day. Given the project's goal to supplant game console and television vegetative-behavior, this sounds like an even more useful control than DansGuardian.

DoudouLinux's origins page says that much of the user experience design is the fruit of lead developer Jean-Michel Philippe's iterative attempts to build an interface that his own children could understand and enjoy. In that sense, it does reflect one developer's personal touch, but one based on testing and user-centric design. Better still, since the project's founding, the list of contributors has grown considerably, including two teams of university-based developers from Russia. Ultimately, the only important test of a "Linux for kids" distribution is whether or not the kids in your own house will use it, but DoudouLinux is worth a close examination for those whose children are too young to take advantage of the school-oriented builds offered by most of the competition.

Comments (4 posted)

Brief items

Distribution quotes of the week

Comments (none posted)

ConnochaetOS 0.9 RC1 released

Full Story (comments: none)

Ubuntu Oneiric Ocelot Alpha 2 Released

Full Story (comments: none)

Distribution News

Debian GNU/Linux

Debian at several conferences

Full Story (comments: none)

delegation for the backports team

Full Story (comments: none)

Fedora

Announcing FUDCon APAC 2011 in Pune

Full Story (comments: none)

Gentoo Linux

Gentoo Council Election Results 2011 to 2012

Full Story (comments: none)

Ubuntu family

Upcoming Ubuntu Classroom Events

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 412 (July 4)
• MWKN (MeeGo, July 4)
• openSUSE Weekly News, Issue 182 (July 2)

Comments (none posted)

Duffy: Anaconda Language & Keyboard Layout Selection

Comments (16 posted)

Hertzog: How to start contributing to Debian?

Comments (none posted)

Introducing Update Packs in Linux Mint Debian

Comments (1 posted)

Wolf: DebConf13 at home? Why not?

Comments (none posted)

Page editor: Rebecca Sobol

Development

A look at Gawk 4.0.0

GNU Awk (Gawk) is one of those workhorse utilities that usually doesn't make news. The 4.0.0 release, however, deserves a look. Announced on June 30th, the latest iteration of Gawk brings the first Gawk debugger, a sandbox mode for running less trusted scripts, revised internals, a number of changes to regular expressions, and IPv6 compatibility.

Gawk is one implementation of Awk. Named for the last names of its inventors (Alfred Aho, Peter Weinberger, and Brian Kernighan), Awk is a scripting language that's standard across UNIX platforms — and by standard I mean that Awk has been part of a standard UNIX (or UNIX-like) system since the beginning, as well as part of the POSIX specification from the Open Group. Even though Gawk is one of many Awks, it stands out as being one of the most widely used. What is it used for? Gawk is best-known for data extraction and reporting, though it's been used to write IRC bots, a YouTube downloader, and for AI programming.

New in 4.0.0

Gawk 4.0.0 is a fairly hefty update from the 3.1.8 release. According to the release announcement, Gawk not only packs a number of new features, bugfixes, and some updates to comply with POSIX 2008, it also has "revamped internals."

To find out what was revamped, and why, I asked Gawk maintainer Arnold Robbins by email. It turns out that the revamp has a lengthy history. Robbins says that "some years ago" John Haque took on rewriting Gawk's internals using a byte-code-style engine — and to implement a debugger in the process. Unfortunately, that work wasn't integrated and Haque moved on. In early 2010, Robbins started trying to bring Haque's code up to date. According to Robbins, the rewrite doesn't provide a huge performance boost, but it does bring a major useful feature:

Right now, dgawk is usable, but still limited. It doesn't report what an error is, but will only report "syntax error" when there's a problem. The debugger will also only work when running a program on the command line — it cannot be attached to a running Awk program. It is unlikely that the Gawk developers will focus on adding that functionality, since the Gawk manual notes that limiting debugging to programs started from within dgawk "seems reasonable for a language which is used mainly for quickly executing, short programs."

Gawk's regular expressions have undergone some changes in 4.0.0. Interval expressions are now part of the default syntax for Gawk, and no longer require the -W or --posix options. Interval expressions — where one or two numbers inside braces (such as {n} or {n,m}) tell Gawk to match a regular expression n or n through m times — were not part of the original Awk specification. Also, \s and \S have been added to match any white space character, or any character that is not white space (respectively).

While Gawk tries to be POSIX-compliant, it does have features above and beyond POSIX — and 4.0.0 introduces a few more. Gawk now supports two new patterns, BEGINFILE and ENDFILE, that can be used to perform actions before reading a file and after (respectively). These are similar to BEGIN/END rules, but are applied before and after reading individual files (since Gawk may process two or more files while running any given script). For example, Gawk programs can now test to see if a file is readable before trying to process it. In prior versions of Gawk, this was not possible — so a script would fail with a fatal error if a file passed to Gawk was not readable.

Gawk has long had the ability to work over a network connection. With the 4.0.0 release, Gawk supports IPv6 using the /inet6 special file, or /inet4 to force IPv4.

The Internet is awash with Awk/Gawk scripts that users might want to run, but worry that the scripts will do more than what's advertised. To address this, Gawk 4.0.0 has a sandbox option (--sandbox), which restricts Gawk to operating on the input data that's been specified. It does this by disabling Gawk's system() function, input redirection using getline, and output redirection using the print and printf functions.

However, Robbins cautions against being overconfident in the security it would convey.

The sandbox mode is not on by default, says Robbins, because it would break "an untold number of existing awk scripts." In short, this option may be useful, but the features disabled by the sandbox option may not be the only way a malicious script could harm a user's system.

The 4.0.0 release is the end of the line for some options and several old and unsupported operating systems. The redundant --compat, --copyleft, and --usage options are gone. The option for raw sockets has been removed, as it was not implemented. If you're still on Amiga, BeOS, Cray, NeXT, SunOS 3.x, MIPS RiscOS, or a handful of others, Gawk 3.1.8 is the final supported release. That the Gawk team has dropped those platforms is no surprise — that they've been carried so long past their expiration date is. It would be challenging indeed to find new proprietary software that's supported on BeOS or Amiga.

With the Gawk 4.0.0 release out of the way, Robbins says that the "big ticket" items for upcoming releases are to merge Gawk's three executables (gawk, pgawk for profiling, and dgawk for debugging) into one to reduce the installation footprint. He also says that Haque "has some other plans related to performance, but that's about all I can say about it in public." Robbins also says that there are plans to merge in some of the XMLgawk extensions. (XMLgawk is an extension of Gawk that has an XML parsing library based on the Expat XML parser.)

Robbins also has a few ideas listed in the Gawk roadmap on his site, which includes support for multiple-precision floating-point (MPFR) so gawk can use infinite precision numbers. He notes that it will be a "big job" and has yet to decide whether MPFR support would be on by default. Gawk is released as it's ready, so there are no dates specified as to when the features can be expected.

The Gawk team is not large, but it's got a healthy set of core contributors. Robbins says that Gawk has six people who maintain ports to different systems, one who handles testing on "a zillion different Unix systems," one contributor who helped out with documentation, and "various other people such as the xmlgawk developers, and several people from different GNU/Linux distributions." Naturally, this also includes Robbins and Haque.

Though Awk is not a particularly "sexy" language these days, it's still a go-to for system administrators and developers. It's good to see that the GNU Project is not only maintaining Gawk, but adding interesting new features that help keep it relevant.

Comments (16 posted)

Brief items

Quotes of the week

Comments (2 posted)

CERN's Open Hardware License v1.1

Comments (none posted)

Mercurial 1.9 released

Comments (12 posted)

notmuch 0.6 released

Full Story (comments: none)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (July 5)
• Mozilla Engineering Newsletter (June 29)
• Mozilla Engineering Newsletter (July 7)
• PostgreSQL Weekly News (July 3)
• Tahoe-LAFS Weekly News (July 4)

Comments (none posted)

Systemd for Developers II

Comments (7 posted)

Zeuthen: Writing a C library, part 3

Comments (none posted)

Zeuthen: Writing a C library, parts 4 and 5

Comments (none posted)

Not much in new Thunderbird 5, but roadmap looks promising (ars technica)

Comments (18 posted)

FabFi: An open source wireless network built with trash (opensource.com)

Comments (4 posted)

Interview with Lennart Poettering (LinuxFR.org)

Comments (76 posted)

PiTiVi Video Editor Now Kitten-Friendly (Linux.com)

Comments (9 posted)

Paley: Why are the Freedoms guaranteed for Free Software not guaranteed for Free Culture?

Comments (76 posted)

Brazilian government signs up to develop OpenOffice and LibreOffice (The H)

Comments (8 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

Project Harmony 1.0 and its discontents

Harmony remains controversial; see these responses by Bradley Kuhn, Richard Fontana, and Dave Neary. Quoting Richard: "Despite my admiration, respect and affection for those who have been driving Harmony, I cannot endorse the product of their work. I believe Harmony is unnecessary, confusing, and potentially hazardous to open source and free software development."

Comments (11 posted)

Nortel's patent pile sold

Comments (40 posted)

ESA Summer of Code in Space 2011

Comments (1 posted)

The 2011 survey of software usage in neuroscience research

Comments (none posted)

Articles of interest

FSFE Newsletter - July 2011

Full Story (comments: none)

Linux IT to underwrite open-source adoption (CRN)

Comments (none posted)

New Books

Designed for Use--New from Pragmatic Bookshelf

Full Story (comments: none)

Metasploit: The Penetration Tester's Guide--New from No Starch Press

Full Story (comments: none)

The Book of Ruby--New from No Starch Press

Full Story (comments: none)

Calls for Presentations

linux.conf.au 2012 CFP open

Full Story (comments: none)

Upcoming Events

Denver 2011 PG DAY (Date Moved)

Full Story (comments: none)

GR Conference 2011

Full Story (comments: none)

Events: July 14, 2011 to September 12, 2011

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol