LWN.net Weekly Edition for June 14, 2012
LinuxCon Japan: OpenRelief launches
Shane Coughlan got hands-on experience in the kinds of problems faced by disaster aid teams while helping with relief efforts after Japan's earthquake, tsunami, and nuclear accident in 2011. That experience was part of a discussion on open source technical measures to assist such efforts at LinuxCon Japan 2011, but it also led to a new project, OpenRelief, that was announced at LinuxCon Japan 2012. The project is aimed at developing inexpensive—"disposable"—drone aircraft to assist relief teams in seeing over the horizon to detect people, changes in terrain, smoke, radiation, and other conditions in places that may be difficult or dangerous for on-the-ground exploration.
![[OpenRelief and airplane]](https://static.lwn.net/images/2012/lcj-coughlan-lattimer-sm.jpg)
OpenRelief co-founders Coughlan, who is a consultant based in western Japan, and Karl Lattimer, who is UK based, came to Yokohama to announce the project and to show off the prototype drone aircraft that Lattimer built. The plane itself was an eye-catching prop for the talk, but some of the most interesting parts of OpenRelief are on the inside: open source software for route following, image acquisition and processing, and so on. Much of that code comes from existing projects, with OpenRelief integrating it into the airframe to create a mobile reconnaissance platform.
Problems and solutions
Gathering information on the state of various locales was one of the biggest problems that Coughlan saw when bringing aid from western Japan to areas affected by the earthquake and tsunami. There were locations that had supplies and doctors, but didn't know where to take them. In addition, sometimes aid arrived at locations that were already fully stocked and had no more storage, so the aid had to return back to where it came from.
The situation was "like a big fog" covering the disaster zone, he said. OpenRelief is trying to help penetrate that fog using technical measures. It would act as a supplement to existing response mechanisms. The goal is "for the responders on the ground to do their job more effectively", Coughlan said, so they don't go where they aren't needed and do go where they are. That was "quite a challenge" last year in responding to the earthquake.
"So, obviously the solution is a robot airplane", he said with a chuckle. More seriously, a robot airplane can help answer some of the questions that are hard to get answers to like "can we get there?" or "do we need to go there?". There were situations where a car couldn't get through to a particular nearby location to assess the situation, but "an airplane could have gone there to see".
Robot airplanes (or drones) have gotten a bad reputation in places like Afghanistan, Coughlan said, but they can be "immensely useful". Unlike those in various war zones, these airplanes are "full of love and peace". They are intended to provide a low-cost solution for mapping and investigating disasters.
The plane will be able to take off and land from a foot path, fly a pre-programmed route, and gather various kinds of information while traveling. Using on-board processing, it will be able to recognize roads, people, and smoke. There are also a variety of sensors that can be deployed to collect weather data, radiation levels, or other kinds of environmental conditions which can relay data via the plane.
The plane and its capabilities are "not really news", Coughlan said, as the technology has been available for some time. OpenRelief has just tied together multiple off-the-shelf and open source pieces for its use case. The technology is "phenomenal and astonishingly cheap". With that, he turned things over to "someone who can build stuff", Lattimer that is, to describe more of the technical details of the plane.
The guts of the plane
It took about a week to assemble one of the drones, Lattimer said, and few more days to finish it. The airframe has a simple design that comes mostly already constructed. It is made out of fiberglass and covered in plastic vinyl. The first that he built was "a challenging project", but the second was much easier.
The plane has an autopilot system, the Arduino-based ArduPilot, which uses a combination of GPS, airflow monitors, and air pressure sensors to fly the plane on pre-programmed flight plans. The flight plan can contain up to 600 3D waypoints that can be reprogrammed in flight from the Raspberry Pi main controller. It takes off using a standard radio controller, then the autopilot is turned on and the plane follows the flight plan.
The Raspberry Pi is "ideal" for an initial test computer, Lattimer said,
because of its low cost, but other, faster main CPUs are possible down the road.
The Raspberry Pi Another board used for testing (Samsung Orion Exynos) has a Mali graphics chip, which was reverse engineered by
his employer, Codethink, and can be used to do a variety of image
processing tasks. The main board runs Debian 6.0 ("Squeeze").
For imaging, the plane uses a CCD with a 170° fisheye lens that provides five separate images. Those feed into the Open Source Computer Vision (OpenCV) package on the Raspberry Pi for doing visual recognition of smoke, people, and roads. There are also plans to do "structure from motion" (SfM) processing to detect landscape changes (e.g. flooding, height changes) but that is fairly processor intensive and will likely require processing after the plane returns from its mission.
Lattimer also described a low-cost, ground-based radiation sensor that can be dropped or placed in areas of interest to relay its readings to the plane. He built the sensor inside of a treacle tin (for those lacking the UK context, it is a can that held a syrup not entirely unlike molasses). The sensor employs an ionization chamber that measures relative ionizing radiation, rather than absolute radiation levels like a Geiger counter would do. It uses a Nanode controller, which is an open hardware device with a long-range, low-power radio to communicate with the plane. Other types of sensors (for chemicals or weather conditions) could also be built using the Nanode.
Mission control
There is a need to tie the robot and sensors together, Coughlan said, and that is where "mission control" comes into play. Most of the technology to do that already exists, so OpenRelief has just integrated those pieces onto a laptop that runs Debian, Ubuntu, or Windows. For example, the autopilot has a sophisticated mission planning application that is written for .NET, but can be run with Mono on Linux.
The output from the mission control system is formatted to be compatible with existing disaster relief packages (e.g. Sahana Eden). The information gathered can be processed into geographic information system (GIS) formats that can be directly integrated into the maps used by those applications. Rather than trying to reinvent the disaster relief application wheel, OpenRelief is adding new capabilities to the existing systems, Coughlan said.
That is one of the keys to the OpenRelief plan: integrating mostly available open hardware and software into the existing systems so that the drones can be put to work in the near future. The system uses OpenStreetMap data for its maps and can even contribute updates to that map repository for things that have changed. "Working alongside existing processes is critical", Coughlan said. During last year's discussions there was talk of redoing much of the disaster relief infrastructure, but that is not the route that the project took; "we just want to fit in with what's there".
The project started, at least conceptually, at the disaster relief panel in 2011. After thinking about airframes, autopilots, people recognition, and the like for a bit, development started in January. The team will be testing and refining the prototype with the hope of being production-ready in December.
The equipment is relatively inexpensive, with a retail bill of materials (BoM) for the prototype at around $750. Getting it into any kind of manufacturing process will make it "ridiculously cheap", he said. The target for the final BoM is $1000 which may include a more powerful main CPU (perhaps Tegra-based) and additional capabilities.
Help wanted
The team is around 25 people currently, consisting of a variety of specialties including engineers, makers, political scientists, mathematicians, and more. The team started out with "crazy ideas", but "those ideas turn out to not be crazy at all", Coughlan said. There is still lots of work to be done, but "it is doable" and the project is looking for help to get it done. The project is hoping to find people to donate time to develop, test, and improve the hardware, but it is looking beyond that as well.
OpenRelief is "kicking off with an ecosystem", he said. Coughlan's company Opendawn along with Codethink and Nanode have all made donations of hardware and time. But there are also a lot of individuals involved. "We want you to join our ecosystem". The project is looking for "your brains, not necessarily your money" (though he admitted it wouldn't turn down monetary contributions).
The ecosystem needs technologists as well as professional and volunteer relief workers to help refine and test the platform, and to help recognize the problems that need to be solved. In addition, there is a need for commercial enterprises to "make buckets of money" by building and selling the drones. The project's focus is to help save lives, but the platform could easily be repurposed for other uses, including for farmers or local governments. While it won't be useful "for anything naughty", Coughlan said, because it is a very visible and slow plane that won't be stealthy, there is a need for this kind of technology for various uses all over the world.
He invited everyone to check out the web site (which has been translated from English into several Asian languages), mailing lists, and various social media (Facebook, Twitter, and Pinterest) for more information. The slides [PDF] from the talk also give more technical details for those interested. Much of the code (with more coming) is on Gitorius and schematics are available at solderpad.com.
So far, the plane has only been flown in radio-controlled mode, at least partly because of regulations in Japan. Lattimer hopes to test with the autopilot sometime in the next month in a free-fly zone in the UK. Regulations on autonomous aircraft vary, but will be a challenge for some time, Coughlan said. He is hopeful that the disaster relief use case, as well as the very limited threat posed by a 3kg aircraft, will help change those regulations, though it will take time.
OpenRelief is an interesting project that combines a certain "geek appeal" with a useful function for reconnaissance in a disaster area. One can certainly imagine low-cost drone aircraft being employed in a variety of situations, but one that can potentially save lives by getting aid where it is needed most is more interesting still. By supplementing existing disaster relief systems—rather than trying to supplant them—OpenRelief's drones could be a nice addition to the disaster relief toolkit. One gets the sense that the drone is just the start, however, so it will be interesting to see what else the project comes up with down the road.
[ The author would like to thank the Linux Foundation for assistance with his travel to Yokohama. ]
The accounting quest: Ledger
Last month, your editor described LWN's need for a non-proprietary accounting system and the difficulty of finding such a thing in the free software community. While there is no shortage of systems that can perform basic double-ledger accounting, it is rather harder to find a system that has everything needed to actually run a business. Since then, a number of people have mentioned a system simply called Ledger. After some experimentation, it is now time to pass on some impressions of how this system works.Ledger takes an interesting approach to the problem in a couple of ways. One is that it is a purely command-line program; there is no graphical interface of any type. The other is that Ledger does not actually maintain any sort of database of accounting data; instead, it reads that data from a plain-text file that the user manages separately. The result is a system that is quite powerful in some ways while falling short in others.
The file format is relatively simple; entries span multiple lines, with all but the first being indented. An example shipped with the program looks like this:
2004/05/27 Book Store
Expenses:Books $20.00
Liabilities:MasterCard
This transaction took place in a book store, where $20 was moved from the Liabilities:MasterCard account to Expenses:Books. The final line lacks a dollar value, so Ledger fills it in to make the transaction balance; more complex split transactions would require explicit values for each account involved. The transaction above has not been reconciled; otherwise there would be an asterisk between the date and the description fields.
Other types of entries are possible. There is a mechanism for adding "period entries," describing transactions that happen at regular intervals. These entries are not really a form of recurring transaction—ledger seems to lack support for those. Instead, period entries are used to describe budgets that can be compared against actual income and expenses. "Automated transactions" are also supported, but those aren't recurring transactions either; they are a mechanism to add splits automatically to other transactions. The syntax for both types is terse and arguably obscure (period entries start with a "~", for example); it can take a little while to understand everything that can appear in the ledger file.
What has been described thus far doesn't involve the Ledger program at
all; this file must be created and maintained via some other mechanism. In
many cases, that mechanism will be a text editor; inevitably, there is an
Emacs mode to make things easier. For those who want a more graphical
experience, Ledger can read (uncompressed) GnuCash files, but there are a
couple of caveats. One is that the Ledger developer (John Wiegley seems to
do almost all the work on Ledger)
is somewhat unenthusiastic about this approach; the manual asks: "why
would anyone use a Gnome-centric, multi-megabyte behemoth to edit their
data, and only a one megabyte binary to query it?
" The other is
that this support appears to be lacking from the in-development 3.0
release. So, while using GnuCash works for now, it's not clear that it
will continue to work in the future.
One could also imagine various automated mechanisms for generating the file from other data sources. Ledger itself never modifies the file; it is simply a consumer of the data contained therein.
As consumers go, it's a powerful one. There is a fairly long list of reports that can be generated. As one might expect, Ledger supports a complex command-line interface that can be used to filter transactions and control the format of the resulting report. The syntax is (once again) terse, but, given time, a suitably motivated user can create almost any sort of text-oriented report one might like. Pie chart aficionados will be disappointed at the outset; there is no graphical output from Ledger. With a bit of work, though, one can get Ledger output into a tool like Gnuplot and produce all kinds of pretty charts.
In summary, Ledger isn't really an accounting system; it's a report-generation utility designed to work with a specific text file format. There are both advantages and disadvantages to its approach. On the up side, Ledger could be an ideal solution for relatively small operations where the people involved are happy to use command-line utilities and do some scripting to make the pieces of a full business hold together. The plain-text ledger file is ideal for anybody wanting to use a tool like Git to track changes over time (or to merge changes done by more than one user). The tool is fast, lightweight, and will never imprison one's accounting data in a black-box binary data file.
On the other hand, it is hard to imagine scaling Ledger even to a business the size of LWN, much less something larger. The tool has clearly been written with speed in mind; it can process a file with many thousands of transactions quickly. But the file itself can only become unwieldy, especially if there are many sources of data feeding into it. Plain-text files lose some of their appeal when they get to be tens or hundreds of thousands of lines in length.
Ledger also lacks features that a typical small-business accounting program would be expected to support. These include:
- Invoicing. One can certainly enter information about an invoice as an
accounts receivable entry, but there is nothing tying it to the actual
invoice sent to a customer. Anybody wanting standard invoice formats,
summaries of outstanding invoices, or per-customer reports will have
to implement them separately.
- Check printing. Yes, it is still necessary to write checks at times.
- Integration with bank accounts. That said, LWN has to do its own
processing of files from banks before feeding the data into QuickBooks
now, so things would not necessarily change a lot in a move to Ledger.
- Recurring transactions. Again, one can certainly script such things,
but it must be done separately.
- Tracking of contractors and creation of tax data (and, in the US, 1099
forms).
- Easy execution of routine tasks like account reconciliation. Ledger can certainly provide a register view showing unreconciled items, but the editing of those items must be done elsewhere (in that big text file).
The other problem with a tool like Ledger is that it can complicate dealings with one's accountant, assuming said accountant is not comfortable with command-line tools. Generating the basic reports that an accountant would want to see will not be that hard. But a good accountant will want to dig through the entries to be sure that everything makes sense and, perhaps, add a journal entry or two to straighten them out. Ledger makes that level of interaction impossible for most accountants, and that will complicate life for many businesses trying to use it.
The bottom line, so to speak, is that Ledger is probably not the solution to LWN's accounting problems, though it might work quite well for simpler businesses with fewer customers, transactions, and accounts. For us, there are too many pieces that would have to be added, some of which would be less than straightforward. So the quest continues; stay tuned.
On mocking
LWN's coverage from LinuxCon Japan included an article on a talk by Greg Kroah-Hartman on the kernel development process and, in particular, how to avoid making kernel subsystem maintainers grumpy. The article got a number of comments, almost all of which were inspired by its last sentence, which read: "Sometimes public mocking is part of the process and can actually help instill that pride more widely". Quite a few LWN readers clearly see the mocking of contributors as a problem; some accused LWN of making the problem worse. Perhaps it is a time for a look at the role of mocking in free software development discussions.
There can be no doubt that the environment in our mailing lists, IRC channels, bug trackers, and other electronic communication channels can be intimidating and off-putting. Some developers handle such environments without trouble; others go away unhappy, and, perhaps, never return. Awareness of this problem has grown over the last decade or two, and, as a general rule, things have gotten better. Even so, one need not look too hard to find examples of harsh messages from high-profile developers. One might well wonder why such behavior persists.
One possibility is that the meritocratic nature of our community makes us
willing to tolerate any sort of behavior in the name of technical
excellence. We need top-level developers and will put up with
unpleasantness if they can produce the code; as Rusty Russell once put it: "If you didn’t run
code written by assholes, your machine wouldn’t boot
". We want our
machines to boot, so we have to accept the developers who can make that
happen. There is almost certainly some truth to this claim; it is the same
calculation that leads us to accept unpleasant politicians, doctors,
managers, and plumbers.
That said, there is evidence that, in our community, we are becoming less accepting of such behavior than we once were. The recent changes in the GNU C library project could be seen as an example.
In the conversation following the LinuxCon article, it was asserted that the kernel development community continues to grow. That was put forward as evidence that things can't be all that terribly bad—the community is doing enough things right that people still want to be a part of it. Once again, there must be some truth to that claim. But it is also hard to imagine that the community is so rich in developers that it need not worry about those who do not handle the environment well. In truth, the community does worry about those developers. Improving the environment in the mailing lists and beyond is a perennial kernel summit topic, and has been the focus of a lot of private communications as well. For all its fearsome reputation, certain counter-examples notwithstanding, linux-kernel is a far more pleasant place than it used to be.
That said, one can still run into trouble on a list like linux-kernel, or in many other places. The previous LWN conversation hinted at a plausible reason for much of the remaining hostility: the lack of reviewer bandwidth. Any project that reaches a certain size tends to discover that it has far more people posting code and asking questions than people who review that code and answer the questions. Review bandwidth tends to be the limiting resource that slows the development process as a whole. Projects can try to deal with this problem in a number of ways; they can, for example, force developers to perform reviews to get their own code merged, or they can simply slow their process to the rate their reviewers can handle. A third alternative—skipping the code review step—has also been tried, but it tends not to produce pleasing results.
In this situation, anything that wastes reviewer time and slows the process further is going to be unwelcome. Additionally, code review can be a time-consuming, tedious, and thankless task; code reviewers are, as a result, often grumpy. Mercurial maintainer Matt Mackall recently described it this way:
Code reviewers often see the same kinds of problems over and over again, either because developers don't read the documentation, don't pay attention to previous discussions, or do not listen to the advice they have been given. When this happens, it is only human nature to strike out with strong words in the hope of making the irritation go away so that some real work can get done. This kind of response can be seen on the lists for a wide variety of projects.
Of course, one could just as easily argue that it's basic human nature to club one's neighbor with a rock to get at that nice stash of saber-toothed tiger bones in his cave. Anybody who has raised children understands how long it takes to learn to function properly in human society. So the fact that it's natural to strike out against mailing-list irritations does not mean that doing so is correct. Much of the time, it's the sort of impulse that we all (hopefully) learn to resist as we grow up.
As the free software development community has grown up, it has, indeed, learned to resist many of those impulses. We have become more adult, and more professional. Most communities tolerate far less unpleasant behavior than they once did; the kernel community can certainly be included in that group. For all the talk of public mocking, it does not actually happen all that often.
In the end, though, we live in a world that is not perfect. There will be people who come into our communication channels and call down impoliteness upon themselves; our world still contains trolls, people who feel entitled to free services from others, fanboys and fanatics who do not understand what "no" means, and those who simply refuse to listen. Some communities will respond to such people by trying to flame them off the list. Others might just quietly ban them from the list—an effective solution, but not necessarily a more civilized one. Not all who are flamed or mocked deserve it, but attempts to eliminate such behavior altogether risk being a cure that is worse than the disease.
In the end, there is value in having a sense of humor about things. It is not at all hard to find examples of developers being called morons or idiots on linux-kernel, but, much of the time, those developers are directing those words toward themselves. We are all morons sometimes; occasionally, we will get caught at it in public. Our community is usually quite accepting of those who understand when they've had one of their moments; it's those who refuse to listen who get the worst of it. Yes, we can do a lot better at dealing with each other respectfully, but, in the process, we should not close our community to real discussion, expect perfection, or lose track of how well we're doing now.
Security
MySQL flaw leaves some systems wide open
For all their faults, passwords are the dominant means of authentication used by computers and applications today. That makes it a little disconcerting to see reports of various longstanding bugs in password handling recently. Obviously it's good that they are being fixed, but it does make one wonder about how much testing we are doing of this critical link in authentication.
Most of the recent password problems (e.g. the multi-package Crypt-DES vulnerability) don't rise to the level of the MySQL/MariaDB flaw reported on June 9, however. Due to an incorrect cast of the return value from memcmp(), the wrong password will be accepted for an existing account with a probability of 1/256. That means that with a fairly small number of tries, an attacker can gain access to a MySQL database server if they know a valid account name (and "root" almost always exists).
While the problem is serious, it is not as bad as it might at first appear. First, it only affects MySQL and MariaDB packages that have been built in a certain way, specifically with GCC using the SSE (Streaming SIMD Extensions) optimization. In that case, memcmp() can return values outside the range of a signed character (-128 to 127) and the MySQL code will sometimes treat that as a password match—even if it isn't. The return value from memcmp() is cast to a char, so if the value has a low-order zero byte (which happens 1/256 of the time), it is seen as a match. While it is the SSE optimization that shows the flaw, assuming that memcmp() will always return values in that range is clearly a bug.
Based on the report and an analysis by HD Moore, it would seem that it is only some Linux distributions that are affected. The official builds from the projects are not vulnerable, and only certain (mostly 64-bit) distributions are vulnerable (Ubuntu, openSUSE, Debian unstable, Fedora, and Arch Linux, according to Moore).
Any affected MySQL server is locally exploitable, but the server must be listening on an external interface for it to be remotely exploitable. Moore did a survey of exposed servers to try to determine the impact of the problem. Without actually trying to log in, it is difficult to get a full accounting, but it is clear that there are at least tens of thousands of affected systems out there listening on the internet.
Sergei Golubchik discovered the bug in MariaDB on April 4 and reported it to MySQL on April 6. It was fixed in MariaDB on April 4, and MySQL followed suit right after the report.
Oracle released a MySQL update as part of its April critical patch update, but makes no mention of the problem (it does list six other CVEs addressed), so either it was silently fixed or is not present there. The release notes for MySQL 5.5.24 and 5.1.63 do mention a security fix for bug 64884, but the bug was presumably still private at that point. MariaDB released several versions on April 6 with the fix as shown in its bug report.
Given that the code was fixed in various public repositories and released much earlier, it is unclear why the details were withheld until recently. Also, it would seem that the Linux distributions—those most affected by the bug—did not release updates in the interim. As of this writing, only Ubuntu has released a security update for the problem. That's a little puzzling as Red Hat was clearly aware of the problem and requested a CVE on April 20, though RHEL is believed to be unaffected. Fedora and other distribution updates seem like they should be coming soon.
While the PostgreSQL/PHP/BSD Crypt-DES flaw only affected users who chose to use a particular authentication scheme, this MySQL flaw is more wide-ranging. In both cases, though, some amount of password fuzz testing would have spotted the problems in short order. It would seem that kind of testing isn't being done with any frequency in some of our communities, which could lead to rather serious bugs that aren't detected for long periods of time.
One guesses that "everyone" thinks the password handling code has been shaken out since it is such an important part of the authentication path, but these bugs show that isn't always the case. This problem has existed in MySQL going back to at least 5.1 (which was released in beta in 2005) and the Crypt-DES flaw goes back further than that. It is certainly not just database systems that are affected by these kinds of flaws, one hopes that other applications and systems that use passwords are either already fuzz testing or will be doing so soon.
Brief items
Security quotes of the week
Doctorow: The Curious Case of Internet Privacy (Technology Review)
Over at Technology Review, Cory Doctorow argues that browser-makers can reclaim user privacy by snuffing out cookie-based tracking. When advertisers say the idea can't work, he says, consider that the same tactic successfully stamped out pop-ups. "When Mozilla's Firefox turned on pop-up blocking by default, it began to be wildly successful. The other browser vendors had no choice but to follow suit. Today, pop-ups are all but gone."
New vulnerabilities
asterisk: denial of service
| Package(s): | asterisk | CVE #(s): | CVE-2012-2947 | ||||||||||||||||||||
| Created: | June 11, 2012 | Updated: | June 18, 2012 | ||||||||||||||||||||
| Description: | From the CVE entry:
chan_iax2.c in the IAX2 channel driver in Certified Asterisk 1.8.11-cert before 1.8.11-cert2 and Asterisk Open Source 1.8.x before 1.8.12.1 and 10.x before 10.4.1, when a certain mohinterpret setting is enabled, allows remote attackers to cause a denial of service (daemon crash) by placing a call on hold. | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
asterisk: denial of service
| Package(s): | asterisk | CVE #(s): | CVE-2012-2948 | ||||||||
| Created: | June 13, 2012 | Updated: | June 13, 2012 | ||||||||
| Description: | From the CVE entry:
chan_skinny.c in the Skinny (aka SCCP) channel driver in Certified Asterisk 1.8.11-cert before 1.8.11-cert2 and Asterisk Open Source 1.8.x before 1.8.12.1 and 10.x before 10.4.1 allows remote authenticated users to cause a denial of service (NULL pointer dereference and daemon crash) by closing a connection in off-hook mode. | ||||||||||
| Alerts: |
| ||||||||||
flash-player: multiple vulnerabilities
| Package(s): | flash-player | CVE #(s): | CVE-2012-2034 CVE-2012-2035 CVE-2012-2036 CVE-2012-2037 CVE-2012-2038 CVE-2012-2039 CVE-2012-2040 | ||||||||||||||||
| Created: | June 11, 2012 | Updated: | June 13, 2012 | ||||||||||||||||
| Description: | From the openSUSE advisory:
Adobe Flash Player was updated to 11.2.202.236, fixing lots of bugs and critical security issues. | ||||||||||||||||||
| Alerts: |
| ||||||||||||||||||
FlightGear: multiple vulnerabilities
| Package(s): | FlightGear | CVE #(s): | CVE-2012-2090 CVE-2012-2091 | ||||||||||||||||||||||||||||||||
| Created: | June 11, 2012 | Updated: | March 14, 2016 | ||||||||||||||||||||||||||||||||
| Description: | From the Red Hat bugzilla: [1], [2]:
[1] Multiple format string flaws were reported in the way Flight Gear, the flight simulator, and SimGear, a simulation library components performed retrieval of various data chunk values from XML aircraft (FlightGear) or scene graph (SimGear) model data files. A remote attacker could provide a specially-crafted XML model file, which once opened by a local, unsuspecting user in FlightGear / in an application linked against SimGear, would lead to that particular executable crash. [2] A potential out-of stack-based buffer bounds write flaw was reported in the way Flight Gear, the flight simulator, retrieved rotor name for certain rotor models. A remote attacker could provide a specially-crafted rotor model XML data file, which once opened by a local, unsuspecting user in FlightGear would lead to 'fgfs' executable crash. | ||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||
groff: multiple vulnerabilities
| Package(s): | groff | CVE #(s): | CVE-2009-5080 CVE-2009-5081 | ||||||||||||||||||||||||||||
| Created: | June 8, 2012 | Updated: | June 13, 2012 | ||||||||||||||||||||||||||||
| Description: | From the Fedora advisory: older security fixes: - CVE-2009-5080: improper handling of failed attempts to create temporary directories in eqn2graph/pic2graph/grap2graph - CVE-2009-5081: roff2.pl and groffer.pl use easy-to-guess temporary file names | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
hostapd: insecure default permissions
| Package(s): | hostapd | CVE #(s): | CVE-2012-2389 | ||||||||||||||||||||
| Created: | June 8, 2012 | Updated: | June 19, 2012 | ||||||||||||||||||||
| Description: | From the Fedora advisory: Tighten-up default permissions for hostapd.conf (CVE-2012-2389) References: [ 1 ] Bug #826109 - CVE-2012-2389 hostapd: insecure default permissions on /etc/hostapd/hostapd.conf [fedora-all] | ||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||
java: multiple vulnerabilities
| Package(s): | java-1.6.0-openjdk | CVE #(s): | CVE-2012-1711 CVE-2012-1713 CVE-2012-1716 CVE-2012-1717 CVE-2012-1718 CVE-2012-1719 CVE-2012-1723 CVE-2012-1724 CVE-2012-1725 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 13, 2012 | Updated: | September 28, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Red Hat advisory:
Multiple flaws were discovered in the CORBA (Common Object Request Broker Architecture) implementation in Java. A malicious Java application or applet could use these flaws to bypass Java sandbox restrictions or modify immutable object data. (CVE-2012-1711, CVE-2012-1719) It was discovered that the SynthLookAndFeel class from Swing did not properly prevent access to certain UI elements from outside the current application context. A malicious Java application or applet could use this flaw to crash the Java Virtual Machine, or bypass Java sandbox restrictions. (CVE-2012-1716) Multiple flaws were discovered in the font manager's layout lookup implementation. A specially-crafted font file could cause the Java Virtual Machine to crash or, possibly, execute arbitrary code with the privileges of the user running the virtual machine. (CVE-2012-1713) Multiple flaws were found in the way the Java HotSpot Virtual Machine verified the bytecode of the class file to be executed. A specially-crafted Java application or applet could use these flaws to crash the Java Virtual Machine, or bypass Java sandbox restrictions. (CVE-2012-1723, CVE-2012-1725) It was discovered that the Java XML parser did not properly handle certain XML documents. An attacker able to make a Java application parse a specially-crafted XML file could use this flaw to make the XML parser enter an infinite loop. (CVE-2012-1724) It was discovered that the Java security classes did not properly handle Certificate Revocation Lists (CRL). CRL containing entries with duplicate certificate serial numbers could have been ignored. (CVE-2012-1718) It was discovered that various classes of the Java Runtime library could create temporary files with insecure permissions. A local attacker could use this flaw to gain access to the content of such temporary files. (CVE-2012-1717) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kernel: multiple vulnerabilities
| Package(s): | kernel | CVE #(s): | CVE-2012-2390 CVE-2012-2372 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 7, 2012 | Updated: | September 11, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Fedora advisory: The 3.4 kernel contains a large number of bug fixes * Wed May 30 2012 Josh Boyer - CVE-2012-2390 huge pages: memory leak on mmap failure (rhbz 824352 824345)
* Thu May 24 2012 Josh Boyer - CVE-2012-2372 mm: 32bit PAE pmd walk vs populate SMP race (rhbz 822821 822825)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kernel: privilege escalation
| Package(s): | kernel | CVE #(s): | CVE-2012-0217 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 12, 2012 | Updated: | July 23, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Red Hat advisory:
It was found that the Xen hypervisor implementation as shipped with Red Hat Enterprise Linux 5 did not properly restrict the syscall return addresses in the sysret return path to canonical addresses. An unprivileged user in a 64-bit para-virtualized guest, that is running on a 64-bit host that has an Intel CPU, could use this flaw to crash the host or, potentially, escalate their privileges, allowing them to execute arbitrary code at the hypervisor level. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
kernel: denial of service
| Package(s): | kernel | CVE #(s): | CVE-2012-2934 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 12, 2012 | Updated: | November 13, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Red Hat advisory:
It was found that guests could trigger a bug in earlier AMD CPUs, leading to a CPU hard lockup, when running on the Xen hypervisor implementation. An unprivileged user in a 64-bit para-virtualized guest could use this flaw to crash the host. Warning: After installing this update, hosts that are using an affected AMD CPU (refer to Red Hat Bugzilla bug #824966 for a list) will fail to boot. In order to boot such hosts, the new kernel parameter, allow_unsafe, can be used ("allow_unsafe=on"). This option should only be used with hosts that are running trusted guests, as setting it to "on" reintroduces the flaw (allowing guests to crash the host). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kernel: denial of service and possible privilege escalation
| Package(s): | kernel | CVE #(s): | CVE-2012-2383 CVE-2012-2384 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 13, 2012 | Updated: | June 13, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Ubuntu advisory:
Xi Wang discovered a flaw in the Linux kernel's i915 graphics driver handling of cliprect on 32 bit systems. An unprivileged local attacker could leverage this flaw to cause a denial of service or potentially gain root privileges. (CVE-2012-2383) Xi Wang discovered a flaw in the Linux kernel's i915 graphics driver handling of buffer_count on 32 bit systems. An unprivileged local attacker could leverage this flaw to cause a denial of service or potentially gain root privileges. (CVE-2012-2384) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
mysql: authentication bypass
| Package(s): | mysql-5.1, mysql-5.5, mysql-dfsg-5.0, mysql-dfsg-5.1 | CVE #(s): | CVE-2012-2122 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 12, 2012 | Updated: | August 13, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Ubuntu advisory:
It was discovered that certain builds of MySQL incorrectly handled password authentication on certain platforms. A remote attacker could use this issue to authenticate with an arbitrary password and establish a connection. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
nova: group policy restriction
| Package(s): | nova | CVE #(s): | CVE-2012-2654 | ||||||||||||||||
| Created: | June 7, 2012 | Updated: | June 26, 2012 | ||||||||||||||||
| Description: | From the Ubuntu advisory: It was discovered that, when defining security groups in Nova using the EC2 or OS APIs, specifying the network protocol (e.g. 'TCP') in the incorrect case would cause the security group to not be applied correctly. An attacker could use this to bypass Nova security group restrictions. | ||||||||||||||||||
| Alerts: |
| ||||||||||||||||||
nss: denial of service
| Package(s): | nss | CVE #(s): | CVE-2012-0441 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 8, 2012 | Updated: | August 21, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the Debian advisory: Kaspar Brand discovered that Mozilla's Network Security Services (NSS) library did insufficient length checking in the QuickDER decoder, allowing to crash a program using the library. For the stable distribution (squeeze), this problem has been fixed in version 3.12.8-1+squeeze5. For the testing distribution (wheezy) and unstable distribution (sid), this problem has been fixed in version 2:3.13.4-3. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
php: multiple vulnerabilities
| Package(s): | PHP5 | CVE #(s): | CVE-2012-2335 CVE-2012-2336 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Created: | June 11, 2012 | Updated: | July 5, 2012 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description: | From the CVE entries:
php-wrapper.fcgi does not properly handle command-line arguments, which allows remote attackers to bypass a protection mechanism in PHP 5.3.12 and 5.4.2 and execute arbitrary code by leveraging improper interaction between the PHP sapi/cgi/cgi_main.c component and a query string beginning with a +- sequence. (CVE-2012-2335) sapi/cgi/cgi_main.c in PHP before 5.3.13 and 5.4.x before 5.4.3, when configured as a CGI script (aka php-cgi), does not properly handle query strings that lack an = (equals sign) character, which allows remote attackers to cause a denial of service (resource consumption) by placing command-line options in the query string, related to lack of skipping a certain php_getopt for the 'T' case. NOTE: this vulnerability exists because of an incomplete fix for CVE-2012-1823. (CVE-2012-2336) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ubuntuone-client: information leak
| Package(s): | ubuntuone-client | CVE #(s): | CVE-2011-4409 | ||||||||||||
| Created: | June 6, 2012 | Updated: | June 13, 2012 | ||||||||||||
| Description: | From the Ubuntu advisory:
It was discovered that the Ubuntu One Client incorrectly validated server certificates when using HTTPS connections. If a remote attacker were able to perform a man-in-the-middle attack, this flaw could be exploited to alter or compromise confidential information. | ||||||||||||||
| Alerts: |
| ||||||||||||||
ubuntu-sso-client: information leak
| Package(s): | ubuntu-sso-client | CVE #(s): | CVE-2011-4408 | ||||
| Created: | June 6, 2012 | Updated: | June 13, 2012 | ||||
| Description: | From the Ubuntu advisory:
It was discovered that the Ubuntu Single Sign On Client incorrectly validated server certificates when using HTTPS connections. If a remote attacker were able to perform a man-in-the-middle attack, this flaw could be exploited to alter or compromise confidential information. | ||||||
| Alerts: |
| ||||||
xen: denial of service
| Package(s): | Xen | CVE #(s): | CVE-2012-0218 | ||||||||||||||||||||||||||||
| Created: | June 13, 2012 | Updated: | June 26, 2012 | ||||||||||||||||||||||||||||
| Description: | From the SUSE advisory:
A guest user could crash the guest XEN kernel due to a protection fault bounce. | ||||||||||||||||||||||||||||||
| Alerts: |
| ||||||||||||||||||||||||||||||
Page editor: Jake Edge
Kernel development
Brief items
Kernel release status
The current development kernel is 3.5-rc2, released by Linus on June 8. "I
think -rc2 is in fairly good shape, and I've been trying to fairly
aggressively revert things that caused problems (and sometimes things that
were only *suspected* to be the cause of problems).
" Since that
release he has been wandering around the world winning high-profile prizes,
so the flow of changes into the mainline has been relatively slow.
Stable updates: 3.0.34 and 3.4.2 were released on June 9; 3.2.20 came out on June 11.
Quotes of the week
"Idiot! Why the f*ck are you running i386 on your nice shiny new x86_64 machine, with Umpteen Gigs of RAM. Boot a x86_64 kernel for Christ sake and get your full potential. Your i386 userspace will still work just fine on it. You're like one of those 75 year old retirees that can finally buy a Porsche just to drive it 10 miles per hour below the speed limit with a line of cars behind them!"
Chris Mason leaving Oracle
Btrfs developer Chris Mason has announced that he is leaving Oracle and joining the growing crowd of kernel hackers at Fusion-IO. "From a Btrfs point of view, very little will change. I'll still maintain Btrfs and will continue all of my Btrfs development in the open. Oracle will still use Btrfs in their Oracle Linux products, and I'll work with all of the distros using Btrfs in production."
Alan Cox celebrates the queen
Some quotes just can't wait for the next week's edition to come about. In a discussion on ACPI, Alan Cox claimed "I spent the holiday avoiding the English queen's party and turning from a republican into a raving republican". Linus's response can only be read in its entirety.
Kernel development news
The word-at-a-time interface
While the kernel is a crucial part of almost any job one might want to do with a computer, it is rarely the kernel itself that gets the interesting work done. More often, it appears as overhead that takes system resources away from the applications the user actually wants to run. So it makes sense to optimize kernel operations to the greatest extent possible, especially when those operations are carried out frequently on performance-critical paths. The "word at a time" interface, optimized and made generic for the 3.5 release, is a good example of how far those optimization efforts can go.The kernel does a lot of string processing, especially (but not exclusively) when working with file path names. It is often necessary to know the length of a name or path component. When confronted with such a task, a C programmer would typically code a loop iterating through the string one character at a time. But, given enough strings, the per-character loop overhead starts to get expensive. It turns out that, with enough bit-level trickery, much of that overhead can be dealt with by working through string data one 32-bit or 64-bit word at a time. The "word at a time" API makes that sort of processing possible—but with a certain complexity cost.
The API
Code wanting to use this interface should include <asm/word-at-a-time.h>. A few functions are defined therein, the first being has_zero():
unsigned long has_zero(unsigned long value, unsigned long *bits,
const struct word_at_a_time *constants);
From one point of view, has_zero() is a simple boolean function that returns true if value contains a zero byte. But what are the other two parameters? Let's start with the constants value, which must simply be set to a value defined in the header file:
const struct word_at_a_time constants = WORD_AT_A_TIME_CONSTANTS;
As will be described in more detail below, this structure contains some useful constant values. The structure is small and the contents are architecture-dependent, so it was evidently deemed unnecessary to create a single, globally-accessible copy.
The bits parameter, instead, is a place where has_zero() can stash temporary data that will be useful to the remaining functions in the API. Those functions are:
unsigned long prep_zero_mask(unsigned long value, unsigned long bits,
const struct word_at_a_time *constants);
unsigned long create_zero_mask(unsigned long bits);
unsigned long find_zero(unsigned long mask);
Once has_zero() has identified a word containing a zero byte, all three of these functions must be used to determine which byte contains the zero value. The usual calling sequence looks something like this:
if (has_zero(value, &bits, &constants)) {
bits = prep_zero_mask(value, bits, &constants);
bits = create_zero_mask(bits);
zero_byte = find_zero(bits);
/* ... */
In other words, prep_zero_mask() and create_zero_mask() both take the bits value first created by has_zero() and rework it to the point that find_zero() can produce the offset of the first zero byte in the word.
This may seem like a lot of work to do, but there is a reason for it. The functionality split allows different architectures to provide optimized functions for each part of the job. But there is another interesting bit of subtlety here: it is possible to perform a logical OR of two different bits values from two calls to prep_zero_mask(). The function hash_name() in fs/namei.c uses this feature to search for either a zero byte or one containing a slash—the string it is looking at ends either with a null byte or the beginning of the next component in the path name. The kernel spends a lot of time processing path names, so this operation is worth optimizing in this way.
There is one other little detail to be kept in mind: the string might not start at the beginning of a word. Managing unaligned strings adds a bit more complexity to the task; the curious can look at lib/strnlen_user.c for one example of how these strings are handled. All told, using this interface adds enough complexity that it is probably almost never worthwhile. In that rare case where a per-character loop is too expensive, though, word-at-a-time access can help.
How it works
The x86 version of this API can be found in arch/x86/include/asm/word-at-a-time.h; one might be forgiven for thinking that parts of it came from the obfuscated C contest. It starts by defining the above-mentioned constants:
struct word_at_a_time {
const unsigned long one_bits, high_bits;
};
#define WORD_AT_A_TIME_CONSTANTS { REPEAT_BYTE(0x01), REPEAT_BYTE(0x80) }
REPEAT_BYTE() is a macro (defined in <linux/kernel.h>) that fills a word with copies of the given byte value. So, on a 32-bit machine, one_bits will be initialized to 0x01010101, and high_bits will be 0x80808080; 64-bit machines will get the same pattern, but twice as long.
After that, has_zero() is defined as:
static inline unsigned long has_zero(unsigned long a, unsigned long *bits,
const struct word_at_a_time *c)
{
unsigned long mask = ((a - c->one_bits) & ~a) & c->high_bits;
*bits = mask;
return mask;
}
In English, the code subtracts one from every byte, masks out all of the bits that were set in the original value, then masks everything but the highest bit in every byte. If one thinks of each byte as an independent value, the high bit can be thought of as the sign bit. Subtracting one from a value will only cause the sign bit to change from zero to one if the bytes's value was zero before. So this series of operations will cause the highest bit to be set in any byte whose value was zero before. (In truth, the bytes are not independent, and borrowing will cause different results after the first zero byte, but only the first one is of interest so that is unimportant).
In the x86 implementation, prep_zero_mask() does nothing and will be optimized out by the compiler. That is not true of create_zero_mask(), though:
static inline unsigned long create_zero_mask(unsigned long bits)
{
bits = (bits - 1) & ~bits;
return bits >> 7;
}
The subtraction will cause all bits up to the first set bit to be set to one; all of the other bits are then masked out and the result is right-shifted. Thereafter, all bytes prior to the first zero byte (in the original value) will be set to 0xff. All that's left, now, is to figure out how many of those fully-populated bytes there are. The code that does this is not entirely straightforward; it is the result of a request Linus posted on Google+ in March. For 32-bit machines, find_zero() comes down to this code:
long a = (0x0ff0001+mask) >> 23;
/* Fix the 1 for 00 case */
return a & mask;
On 64-bit systems, instead, it looks like:
return mask*0x0001020304050608ul >> 56;
Either way, the effect is to produce a number that is the byte offset of the first zero.
This API is relatively new, having been first added (for x86 only) in the 3.4 development cycle. In 3.5, it was substantially reworked and made more widely applicable. There are specific implementations for x86 and powerpc (the latter uses a "count leading zeroes" instruction to speed things up); there is also a "generic" version that really only works properly for big-endian architectures. That is enough, though, for a number of architectures to make use of the capability. The resulting microsecond-sized time savings may not seem like much, but, multiplied across all of the string operations the kernel does, it adds up to a significant improvement.
LinuxCon Japan: One zImage to rule them all
An elusive, but highly sought after goal for Linux on ARM is to have a "single" kernel image that can boot on all ARM devices. The full goal is likely not attainable, but reducing the number of different ARM kernels is, and progress is being made. Linaro kernel tech lead Deepak Saxena reported on the motivations behind the consolidation effort as well as progress toward the goal at LinuxCon Japan 2012.
The holy grail
Saxena started with a problem statement. He noted that those who have worked with ARM for a while may not really see the problem, but it is a problem that every ARM processor needs its own kernel. For example, a laptop vendor that makes a few small hardware changes will almost certainly need a separate kernel for each revision. The "holy grail" is a single kernel binary that will boot on any ARM device, he said.
![[Deepak Saxena]](https://static.lwn.net/images/2012/lcj-saxena-sm.jpg)
Requiring separate kernels "creates a lot of extra work" for developers. If they are building a driver for multiple platforms—or even three revisions of the same platform—they will probably need to build and test a kernel for each. Sometimes there is a register that can be read to determine the hardware revision, thus reducing the number of different kernels, but that often is not the case.
For distributions, supporting ARM is difficult right now. Debian, for example, has different kernels for the BeagleBoard, PandaBoard, and others. In the consumer electronics and mobile space, the problem is similar, and those companies want to reduce the amount of testing that needs to be done. Right now, we think of cell phones more than anything else, but the Saxena pointed to the video projector and camera as devices that are also likely ARM-powered. Beyond that, there will soon be ARM servers.
The model in the server/enterprise world is very different than mobile. Typically, a mobile device comes with a full software stack, and updates come the same way, but that is not the case for servers. You may or may not get the distribution that you want on servers and, in fact, may get Windows on those systems. That means that distributions need to have installation media that can work on a wide variety of server platforms.
"The distros have spoken", Saxena said, and they need one kernel image that can be built, booted, and tested on all of the different platforms. There are thousands of different x86-based laptops today, and you don't need a different kernel for each. The Linux ARM community wants to get to that same model. Beyond that, cloud and hyperscale computing also need to be able to deploy on lots of different platforms without requiring separate kernels.
"How did we get here?", he asked; you can boot a single Ubuntu or Fedora install disk on all x86 systems, but that is not true for ARM. Part of the problem is the diversity in the ARM world. There is also a lot of fragmentation in how code has been written to support Linux on ARM. There are multiple implementations for the same IP block, for example. Functionality has been duplicated at least partly because of a closed-door policy by the various ARM vendors.
In addition, Linux ARM maintainer Russell King got overloaded, which led ARM platform and system-on-chip (SoC) developers to start pushing their code to Linus Torvalds directly. Saxena said that he may have been the first to do that, for ixp4xx, but now he sees the problems that caused and apologized for doing so. It led to an inability for anyone to track the "big picture" in Linux ARM support, he said.
Fixes needed
It is now a multi-faceted problem that requires several different avenues of attack to clean up. Saxena identified four areas that are being pursued. Cleaning up and consolidating the header files within the various ARM machine and platform trees is one, while consolidating ARM drivers is another. In addition, device tree will provide a way to specify the differences between ARM SoCs at runtime. Finally, doing active maintenance of the ARM tree, keeping in mind the big picture, will also help. No one of those fixes the problem, but all of them together get us closer to the holy grail, he said.
To start with, there are various header file collisions in the ARM tree. There are a bunch of arm/arch/mach-* directories, one for each of the machine types. Each of those has an include/mach directory that maps to the top-level include/mach directory at build time. In order to build for multiple machine types, those header files need to be consolidated in one place so that the remapping doesn't need to happen.
In the 3.0 kernel tree, which was around the time the consolidation effort started, there were 64 different machine types in the tree. Some of those machine types are similar and could be consolidated. For the others, there are lots of overlapping symbols that need to be dealt with. The goal is to get rid of as many of those as possible either by making them generic for all ARMs or by moving platform-specific symbols to non-generic header files.
There were also 577 occurrences of #include <mach/*> in the drivers directory. Unfortunately, ARM has a lot of driver-specific definitions in the architecture headers, which means that drivers depend on arch/arm header files. Basically, it is prevalent for ARM drivers to require definitions from both the driver directories and the architecture headers, which makes it difficult to build multi-platform kernels.
Linaro and the ARM community started working on these problems last year. They met in August and thought they could have a solution in relatively short order, but that proved not to be the case. Some changes require coordination between multiple maintainers and others are awaiting agreement between maintainers on how to proceed. The problem "may seem trivial at first", Saxena said, but it actually fairly complicated. The hope is to have a single zImage for multiple systems by the end of 2012.
Beyond the header file issue, there is a need to cleanup and consolidate the drivers in the tree. The problem is not directly related to creating a single kernel, but fixing it will help to get there, he said. There are lots of implementations of drivers for the same hardware in the tree, sometimes with a different API. That can cause problems with overlapping symbols and code bloat.
The clock management code is the "epitome of code duplication and fragmentation" in the ARM kernels, Saxena said. The clk.h file, which declares a struct clk, was introduced in 2.6.16 back in 2006. Since then, 27 different struct clk implementations have been added to the tree, each with its own API and semantics. Over the last two years or more, work has been done to create a common definition and API, though the job is not done yet, as there is ongoing discussion on certain parts.
Pinmux is another example of duplication. It is a subsystem to manage the pins on SoCs and there were multiple implementations of that functionality. The problem is not as bad as struct clk, he said, but there was still a need to consolidate. After six months of work, a common pinmux implementation is now upstream, though there are still discussions about certain parts of the implementation and API.
Another piece of the solution is device tree. Before device tree, very small, simple changes to the hardware would require a kernel rebuild because the information about IRQ lines, memory maps, and so forth were statically defined. Device tree makes it so that much of this information can be probed at boot time.
Using device tree means creating a source file using a "simple markup language" that defines the hardware. It can specify where the registers are or what the interrupts are for a particular device. That means that the same kernel can be used on a new SoC once a device tree file has been created for that SoC. Since the kernel will not have to change, it makes hardware revisions and testing multiple devices that much easier.
Status and plans
Currently, a single kernel can be built for the four Linaro member platforms (imx, omap2, ux500 and vexpress), though only the Versatile Express board boots at this point. The goal is to have all four booting by the end of the year. Linaro is focused on its member platforms, but would like to get other platforms supported as well. That is even more reason for SoC vendors to get their code upstream, Saxena said, as it will be more difficult to participate in the multi-platform kernel effort with code that is out of the mainline. The ARM SoC tree has been used as the base, with Arnd Bergmann maintaining a branch for the multi-platform work.
There are, of course, things still left to do. USB drivers need to be consolidated as there are some problems building multiple host controllers into one kernel at this point. Finishing the device tree conversion is another piece of the puzzle; the infrastructure is there, but there are lots of drivers and board files to convert. At the moment there is something of a hack around the "driver #include madness", which needs to be cleaned up for real.
While the holy grail has not been reached, things will be better than they are today, Saxena said. Due to micro-architecture and page table differences, four kernels are envisioned: ARM v6/7 with large physical address extensions (lpae), ARM v6/7 non-lpae, ARM v4/5, and, eventually, ARM v8. It still means multiple kernel binaries, but that could be treated in the same way that the distributions handle various CPU extensions in the x86 world. The idea of "one zImage to rule them all" turns out to not be practical, but we will end up with far fewer ARM kernel binaries in the near future.
[ The author would like to thank the Linux Foundation for assisting with travel to Yokohama for LinuxCon Japan 2012. ]
A big.LITTLE scheduler update
ARM's big.LITTLE architecture is an example of asymmetric multiprocessing where all CPUs are instruction-set compatible, but where different CPUs have very different performance and energy-efficiency characteristics. In the case of big.LITTLE, the big CPUs are Cortex-A15 CPUs with deep pipelines and numerous functional units, providing maximal performance. In contrast, the LITTLE CPUs are Cortex-A7 with short pipelines and few functional units, which optimizes for energy efficiency. Linaro is working on two methods of supporting big.LITTLE systems.
The first method pairs up each big CPU with a LITTLE CPU, so that user applications are aware of only one CPU corresponding to each such pair. A given application thread can then be transparently migrated between the big and LITTLE CPUs making up the pair, essentially providing extreme dynamic voltage/frequency scaling. Instead of merely varying the voltage and frequency, at some point the application thread is migrated between the big and LITTLE CPUs making up the corresponding pair, as was described in an earlier LWN article by Nicolas Pitre. This approach is termed “big.LITTLE Switcher.”
The other way to support big.LITTLE systems is to have all CPUs, both big and LITTLE, visible in a multiprocessor configuration. This approach offers greater flexibility, but also poses special challenges for the Linux kernel. For example, the scheduler assumes that CPUs are interchangeable, which is anything but the case on big.LITTLE systems. These big.LITTLE systems were therefore the subject of a scheduler minisummit at last February's Linaro Connect in the Bay Area which was reported on at LWN.
This article summarizes progress since then, both on the relevant mailing lists and as reported during the Linaro Connect sessions in Hong Kong, and is divided into the following topics:
- Emulate asymmetric computation on commodity systems
- Document system configuration for big.LITTLE work
- Define mobile workload and create simulator
- Address kthread performance issues
- Wean CPU hotplug from stopping all CPUs
- Add minimal support to scheduler for asymmetric cores
- Conclusions
Following this are the inevitable Answers to Quick Quizzes.
Emulating asymmetric computation on commodity systems
The need to develop software concurrently with the corresponding hardware has led to heavy reliance on various forms of emulation, and ARM's asymmetric big.LITTLE systems are no exception. Unfortunately, full emulation is quite slow, especially given that core kernel code and user-level code really only needs a reasonable approximation to big.LITTLE's performance characteristics. What we need instead is some way to cause commodity systems to roughly mimic big.LITTLE, for example, by artificially slowing down the CPUs designated as LITTLE CPUs.
There are a number of ways of slowing down CPUs, the first three of which were discussed at the February scheduler minisummit:
- Assign a high-priority
SCHED_FIFOcycle-stealing thread for each designated LITTLE CPU, which consumes a predetermined fraction of that CPU's bandwidth. - Constrain clock frequencies of the LITTLE CPUs.
- Make use of Intel's T-state capability.
- Use perf to place a load on the designated-LITTLE CPUs.
Rob Lee presented the results of experiments comparing the cycle-stealing and clock-frequency approaches. Morten Rasmussen proposed the perf-based approach, in which he configured perf to interrupt the designated LITTLE CPUs every few thousand cycles. Each of these approaches has advantages and disadvantages, as laid out in the following table:
| Type | Transparent to Scheduler? | Portability? | Calibration? | Scope? |
|---|---|---|---|---|
| Cycle stealing | SCHED_FIFO threads visible |
Portable | Requires calibration for duty cycles less than about 10 milliseconds | Process-level only |
| Clock Frequency | Transparent | Requires per-CPU clock domains | Auto-calibrating, but limited number of settings | All execution |
| T-State | Transparent | Intel only | Auto-calibrating | All execution |
| perf | Transparent | Requires fine-grained perf | Requires calibration | All code subject to perf exceptions |
As can be seen from the table, none of these mechanisms is perfect, for example, many embedded systems-on-a-chip (SoCs) have multiple CPUs (often all of them) in a given clock domain, which limits the applicability of the clock-frequency approach. Rob has placed scripts for the cycle-stealing and clock-frequency approaches in a git tree located at git://git.linaro.org/people/rob_lee/asymm_cpu_emu.git; he plans to add Morten's perf-based approach as well.
At this point, it seems likely that more than one of these will be required in order to cover the full range of development hardware and workloads.
Documenting system configuration for big.LITTLE work
The Linux kernel as it is today can handle big.LITTLE systems, give or take hardware bring-up issues. However, the current kernel does need some help to get good results from a big.LITTLE system. Chris Redpath's presentation is a first step towards determining the best division of labor between current kernels and the application.
Chris described an experiment running an Android workload on emulated
big.LITTLE hardware.
He made use of Android's bg_non_interactive cpuset,
which holds low-priority threads (ANDROID_PRIORITY_BACKGROUND)
and is limited to 10% of CPU usage.
Chris further constrained this cpuset to run only on CPU 0, which on his
system is a LITTLE CPU.
Chris then created two new cpusets, default and
fg_boost.
The default cpuset is constrained to the LITTLE CPUs,
and contains all non-background tasks that are not SCHED_BATCH.
The SCHED_BATCH tasks remain in the bg_non_interactive
cpuset called out above.
The new fg_boost cpuset contains tasks that are
SCHED_NORMAL and that have a priority higher than
ANDROID_PRIORITY_NORMAL.
Chris used a combination of sysbench and cyclictest as a rough-and-ready mobile-like workload, where sysbench mimics high-CPU-consumption mobile tasks (e.g., rendering a complex web page) and cyclictest mimics interactive workloads (e.g., user input and communications). Chris's configuration uses four sysbench compute threads and eight cyclictest threads. The sysbench threads all run for 15 seconds and then block for 10 seconds, and repeat this on a 25-second cycle. The eight cyclictest threads run throughout the full benchmark run. Without Chris's additional cgroups, the scheduler scattered the threads across the system, slowing execution of the sysbench threads and wasting power. With the additional cgroups, the sysbench threads were confined to the big CPUs, and the cyclictest threads to a single LITTLE CPU.
In short, use of cpusets to constrain whether given threads run on big or LITTLE CPUs works quite well, at least as long as we know a priori which threads should run where.
Chris also tested sched_mc, which can be set up to keep
short-lived tasks off of the big CPUs.
Although sched_mc was able to keep an additional CPU free
of work, it was unable to help the remaining CPUs to reach deeper sleep
states, and was nowhere near as effective as use of cpusets.
These big.LITTLE results support the decision taken at the February Linaro
Connect to remove sched_mc from the kernel.
Finally, Chris tested userspace tools that dynamically migrate tasks among the CPUs in response to Android's priority boosting of foreground user-interface (UI) threads. In contrast, Chris's first approach statically assigned the threads. This approach required minor changes to the Android software stack. Although this approach was effective in getting high-consumption UI threads running on big CPUs, the migration latency rivaled the bursts of UI CPU-bound activity, which in many cases defeats the purpose. The migration latency would need to be decreased by about a factor of two for this approach to be worthwhile, though your mileage may vary on real hardware. These results underscore the importance of planned scheduler changes for dynamic general-purpose workloads.
Several other potential approaches were discussed:
- Use a modified
cpufreqgovernor to switch between the big and LITTLE CPUs. One potential issue with this approach is that the governor currently runs at full frequency most of the time, in accordance with its race-to-idle strategy. Some tweaking would therefore be required for workloads for which race-to-idle is inappropriate. - Treat specific applications specially, for example, make big.LITTLE
scheduling decisions based on the
->commname. This is likely to work well in some cases, but is unable to optimally handle applications that have different threads that want to run on different CPUs. - Rather than migrating threads from one cpuset to another, add and remove CPUs to given cpusets. This should reduce the sysfs overhead in some cases.
In short, it is possible to get decent performance out of big.LITTLE systems on current Linux kernels for at least some workloads. As more capabilities are added to the scheduler, we can expect more workloads will run well on big.LITTLE systems, and also that it will become easier to tune workloads that can already be made to run well on big.LITTLE.
Define mobile workload and create simulator
An easy-to-use mobile-workload simulator is needed to allow people without access to the relevant hardware, emulators, and SDKs to evaluate the effects of changes on mobile workloads. The good news is that the work presented at Linaro Connect used a number of synthetic workloads, but the not-so-good news is that the setups are not yet generally useful. Shortcomings include: (1) Much work is required to interpret the output of the workloads, (2) The figures of merit are not constant, but instead different figures of merit are chosen in different circumstances, and of course (3) They are not (yet) nicely packaged. Hopefully we will see additional progress going forward.
Address kthread performance issues
At the February Linaro Connect, Vincent Guittot presented work showing that the bulk of CPU-hotplug latency was due to creation, teardown, and migration of the per-CPU kthreads that carry out housekeeping tasks for various kernel subsystems. Further discussion indicated that any mechanism that successfully clears all current and future work from a given CPU will likely incur similar latencies.
In an ideal world, these per-CPU kthreads would automatically quiesce when the corresponding CPU went offline, and then automatically spring back into action when the CPU came back online, but without the overheads of kthread creation, teardown, or migration. Unfortunately, the scheduler does not take kindly to runnable tasks whose affinity masks only allow them to run on CPUs that are currently offline. However, Thomas Gleixner noticed that there is one exception to this rule, namely a set of per-CPU tasks that have exactly this property: the idle tasks. This key insight led Thomas to propose that the per-CPU kthreads should have this same property, which should completely eliminate the overhead of per-kthread creation, teardown, and migration.
The idle tasks are currently created by each individual architecture,
so Thomas posted a
patchset that
moves idle-task creation to common code.
This patchset has been well received thus far, and went into mainline
during the 3.5 merge window.
Thomas has since followed up with a new
patch
that allows kthreads to be parked and unparked.
The new kthread_create_on_cpu(),
kthread_should_park(),
kthread_park(), and
kthread_unpark() APIs can be applied to the per-CPU
kthreads that are now created and destroyed on each CPU-hotplug cycle.
At the Q2 Linaro Connect in Hong Kong, Nicolas Pitre suggested a different way to leverage the properties of idle tasks, namely to provide a set of high-priority per-CPU idle tasks that would normally be blocked. When it was time to quiesce a given CPU, its high-priority idle task would become runnable, preempting all of the rest of that CPU's tasks. The high-priority idle task could then power down the CPU. This approach can be thought of as a variant of idle-cycle injection. There are some limitations to this approach, but it might work quite well in situations where the CPU is to be quiesced for short periods of time.
In the meantime, Vincent Guittot has been experimenting with various userspace approaches to reduce the per-kthread overhead. One promising approach is to use cgroups to ensure that kthreadd (the parent of all other kernel threads) is guaranteed the CPU bandwidth needed during the hotplug operation. Preliminary results indicate large improvements in latency. Although this approach cannot achieve a five-millisecond CPU-hotplug operation (a target adopted at the February meeting), it is nevertheless likely to be useful for projects that are required to use current kernels with the existing slow CPU-hotplug code.
Wean CPU hotplug from stopping all CPUs
Another source of CPU-hotplug slowness—and of real-time
latency degradation—is its use of __stop_machine().
The problem is that __stop_machine() function causes
each CPU to run a special per-CPU __stop_machine()
kthread, which brings all processing on the system to a grinding
halt for the duration.
This system-wide grinding halt is bad for performance and fatal to real-time
response.
Given that other systems have managed to remove CPUs from service
without requiring such a grinding halt, it should be possible to
wean Linux's CPU-hotplug process from its __stop_machine()
habit.
Quick Quiz 4: Why bother to fix CPU hotplug? Wouldn't it be better to just rip it out and replace it with something better? Answer
This is a large change, and will take considerable time and effort. However, it is likely to provide good improvements in both performance and robustness of CPU hotplug.
Add minimal support to scheduler for asymmetric cores
There has been great progress in a number of areas.
First, Paul Turner posted a new version of his
entity load-tracking patchset.
This patchset should allow the scheduler to make better
(and more power-aware) task-placement and load-balancing decisions.
Second, Morten Rasmussen ran some experiments (including experimental
patches) on top of Paul Turner's patchset.
See below for more information.
Third, Peter Zijlstra posted a
pair of
patches
removing sched_mc
and also posted an
RFC email
proposing increased scheduler awareness of hardware topology.
This should allow the scheduler to better handle asymmetric
architectures such as ARM's big.LITTLE architecture.
Finally, Juri Lelli posted an
updated version
of a prototype
SCHED_DEADLINE patchset,
which Juri has taken over from Dario Faggioli.
Please see below for a discussion of how
SCHED_DEADLINE might be used for energy efficiency on
big.LITTLE systems.
Morten Rasmussen presented prototype enhancements to the Linux scheduler
that better accommodate big.LITTLE systems.
As noted above, these enhancements are based on Paul Turner's entity
load-tracking patchset.
The key point behind Morten's enhancements is that work should be
distributed unevenly on big.LITTLE systems in order to maximize
power efficiency with minimal performance degradation.
Unlike SMP systems, on big.LITTLE systems it matters deeply where
a given task is run, not just when a task is run.
Morten therefore created a pair of cgroups, one for the big CPUs
and another for the LITTLE CPUs, but with no load balancing between
them.
The select_task_rq_fair() function checks task load history
in order to make the correct big.LITTLE selection when a given task
starts running.
High-priority tasks that have tended to be CPU bound are placed on big CPUs,
and all other tasks are placed on LITTLE CPUs.
Of course, a high-priority task that has historically been I/O bound might enter a CPU-bound phase. Therefore, Morten's patchset also periodically migrates tasks via load balancing. There are of course issues with global load balancing, and addressing these issues is important future work. Nevertheless, when running the Bbench browser benchmark on an emulated big.LITTLE Android system, Morten's patches provided response time very nearly that of an SMP system with all big CPUs, both when running on SMP hardware emulating big.LITTLE and when running on Paul Turner's LinSched. This is clearly an impressive achievement. In contrast, big.LITTLE response time on a vanilla kernel is 10-20% slower with much worse variability in response times.
Obviously, considerably more performance evaluation is required, but preliminary results are quite promising. In addition, more work will be required to handle thermal issues and also to do load balancing of tasks back onto LITTLE CPUs when all the big CPUs are saturated.
Juri Lelli's SCHED_DEADLINE is quite important for some
types of real-time computing, but some thought is required to see possible
application to non-real-time big.LITTLE systems.
The key thing about SCHED_DEADLINE is that it provides the
scheduler with information about the likely future needs of the various
applications running on the system.
This information should allow the scheduler to make better big.LITTLE
task placement decisions, and also potentially better frequency-selection
decisions.
Of course, this begs the question of where the deadlines and CPU requirements come from. In fact, one reason that deadline schedulers remain the exception rather than the rule even for real-time applications is that it is difficult to generate the needed information for a given real-time application, especially given the non-deterministic nature of modern hardware and variations in rendering time for different media—or even for different segments of a given item being played back. One way to solve this problem for some mobile workloads might be to use feedback-directed profiling techniques, where the application's CPU requirements are measured rather than derived.
Regardless of how things turn out for the use of deadline scheduling for mobile workloads, there is clearly some important and innovative work ahead for power-aware scheduling of mobile workloads.
Finally, Vincent Guittot led a discussion on additional inputs to the scheduler, which centered mainly around accurate load tracking (e.g., Paul Turner's patchset), which CPUs share a clock (and thus must all run at the same frequency), which CPUs share idle state, which CPUs share each level of cache, and thermal constraints. There was also a call to keep all this simple, which is clearly extremely important—and also will likely be extremely difficult. But if keeping things simple was easy, where would be the challenge?
Conclusions
Although there is still a great deal of work remaining before mobile workloads fully exploit the power and power efficiency of big.LITTLE, the past few months have seen some impressive progress in that direction. With appropriate user-mode help, some important workloads can make good use of big.LITTLE systems even given the existing scheduler, and some modest modifications (in conjunction with Paul Turner's entity load-tracking patchset) greatly improves the scheduler's ability to support big.LITTLE with minimal user-mode help. A promising design for streamlining CPU hotplug has been identified, and work in that direction has started.
This work gives some hope that the future will be bright for asymmetric multiprocessing in general and ARM's big.LITTLE architecture in particular.
Acknowledgments
We all owe a debt of gratitude to Srivatsa Bhat, Juri Lelli, Daniel Lezcano, Nicolas Pitre, Christian Daudt, Roger Teague, and George Grey for helping to make this article human readable. I owe thanks to Dave Rusling and Jim Wasko for their support of this effort.
Answers to quick quizzes
Quick Quiz 1: How can you tell whether multiple CPUs on your system are in the same clock domain?
Answer:
Look at the contents of this sysfs file:
/sys/devices/system/cpu/cpu0/cpufreq/affected_cpus
This file lists all of the CPUs that are in the same clock domain as CPU 0. Similar files are present for the rest of the CPUs.
Quick Quiz 2: Given that these emulation approaches are never going to provide a perfect emulation of big.LITTLE systems, why bother? Wouldn't it be simpler to just wait until real hardware is available?
Answer: Indeed, none of these emulation methods will perfectly emulate big.LITTLE systems. Then again, the various big.LITTLE implementations will not be exactly the same as each other anyway, so significant emulation error is quite acceptable. More importantly, these emulation approaches allow big.LITTLE software work to proceed long before real big.LITTLE hardware is available.
Quick Quiz 3: This sounds way simpler than CPU hotplug, so why not just get rid of CPU hotplug and replace it with this mechanism?
Answer: First, CPU hotplug is still needed to isolate failing hardware, which some expect to become more prevalent as transistors continue to shrink. Second, there are a number of limitations of Nicolas's approach compared to CPU hotplug:
- Interrupts must be directed away from the CPU.
- Timers must be migrated off of the CPU.
- If a CPU is left in this state for more than two minutes,
the softlockup subsystem will start reporting errors
if there are runnable tasks affinitied to this CPU.
One workaround is to migrate tasks away from this
CPU, but that approach starts looking a lot like CPU
hotplug.
- If one of the preempted threads is hard-affinitied to the
CPU and is holding a mutex, then that mutex cannot be
acquired on any other CPU.
If the CPU remains in this state very long, a system hang
could result.
- The per-CPU kthreads would need to be informed of this transition if it persists for very long.
In other words, as noted above, Nicolas's approach is appropriate only for cases where the CPU is to be quiesced for a short time.
Quick Quiz 4: Why bother to fix CPU hotplug? Wouldn't it be better to just rip it out and replace it with something better?
Answer: Although there is always a strong urge to rip and replace, the fact is that any mechanism that successfully removes all current and future work from a CPU will really need to remove all current and future work from that CPU. And it is the removal of all such work that makes CPU hotplug difficult, not the actual powering down of the electronics. Of course, this suggests the possibility of removing all work without actually powering the CPU down, a possibility that is likely to be supported by Thomas's ongoing CPU-hotplug rework.
Patches and updates
Kernel trees
Architecture-specific
Core kernel code
Development tools
Device drivers
Filesystems and block I/O
Memory management
Security-related
Virtualization and containers
Miscellaneous
Page editor: Jonathan Corbet
Distributions
Guacamayo media player distribution
Guacamayo is a new entrant in the field of Linux-based operating systems for multimedia devices. Most such distributions (e.g. LinHES or Mythbuntu) start with a desktop distribution and add media management, playback, and other features on top, but Guacamayo's creators Tomas Frydrych and Ross Burton are taking the opposite approach, starting with a minimal embedded system and adding only the software necessary to stream multimedia content. The result, they hope, will be a more stable base for building set-top boxes, network-connected speakers, and other nontraditional media platforms.
Frydrych said the impetus for the project was the popular Raspberry Pi device, which offers a low-cost board capable of HDMI output. Despite that inspiration, however, he builds Guacamayo against a different test platform, the pricier (but still lightweight) ZBox, a roughly WiFi-router-sized fanless desktop unit. Frydrych and Burton are both veterans of OpenedHand and Intel (which acquired the former in 2008), and Guacamayo builds on many of the projects created by OpenedHand over the years. It is based on the Yocto embedded Linux platform, uses the Clutter library for its GUI layer, and runs Media Explorer (MEX) as its "shell." For now, the emphasis is on Universal Plug-and-Play (UPnP) and Digital Living Network Alliance (DLNA, which is a consumer electronics compliance organization built on top of UPnP) compatibility through the Rygel library.
The first public release is version 0.2, which was code-named "See No Evil, Hear All You Want!" because it supported audio playback only (MEX support has been added subsequently). Burton posted a blog entry about the release on May 22, saying he used it successfully with a UPnP media storage device. In that configuration, Rygel presents the Guacamayo front-end as a UPnP "Media Renderer" that can be remotely controlled by any compliant UPnP or DLNA "Media Controller." In the same post, Burton said that Frydrych had tested the release running on a BeagleBoard being controlled over the network with a Android DLNA control application.
Sound off
Audio playback alone might not sound particularly exciting, but the next release should add video playback, which is where the system begins to get more interesting. The audio decoding and playback are handled by PulseAudio and GStreamer, both of which are standard system components on desktop Linux. But Guacamayo's video stack is intended to dispense with X Windows, window managers, and desktops entirely. MEX is a simple media browser and player built specifically for Clutter, but Guacamayo's Clutter will run with an OpenGL back-end. That will enable smooth graphics on Raspberry Pi and other such low-power boxes that have hardware-accelerated OpenGL support.
![[MEX on Guacamayo]](https://static.lwn.net/images/2012/06-guacamayo.png)
According to Burton, many of the GPUs in this generation of device are also supported (at least partially) by the video acceleration API (VA-API). GStreamer supports VA-API, therefore Clutter and MEX should be able to take advantage of it as well. On the other hand, Guacamayo can be built for a broad spectrum of hardware devices, some of which might be able to take full advantage of VA-API and some of which might not. Burton noted that his old ThinkPad can decode MPEG2 in hardware, but it can do the job just as easily in software. In contrast, a more recent Cedar Trail Atom netbook can decode the more complex H.264 codec at 1080p in hardware, but is incapable of decoding the same format in software fast enough to be watchable.
You can get another taste of the flexibility of the Guacamayo code from the build instructions. The project uses the Poky build system, and at the moment three build options are supported: the headless audio-only release, the video-enabled MEX target that runs directly in OpenGL, and an interim option that runs MEX on X11.
Despite how clean and simple the setup is — and there can be
little denying that the stripped-down media playback environment of
Guacamayo is simple — a few challenges remain. The first is video
drivers. Both Burton and Frydrych observed that open source drivers
for Intel video chips offer good support for OpenGL (and in particular
the EGL interface used by
Guacamayo), but that other chipmakers are spottier. Frydrych said
that the BeagleBoard drivers "have issues
" and
"took far more effort than should be necessary
", while
Burton said that he thought the Nouveau drivers for Nvidia hardware
worked well. Still, Frydrych said that he felt that "
the
OpenGL(ES) route is good enough to bank on
", since the
chipmakers are aware that they need working OpenGL ES drivers in order
to compete in the marketplace. He has yet to see exactly how well the
Raspberry Pi delivers on this point, he added, but if they "
get
it right
" it could have a significant impact on the
competition.
Another challenge is support for input devices and remote controls. The X-less builds use the kernel's evdev interface to capture input events, but without X there is no XInput2 subsystem to enable multi-touch, "multimedia keyboards" or other flashy input devices. Although Rygel gives Guacamayo automatic support for UPnP/DLNA remote controls, infrared is still the dominant flavor of remote control. Burton commented that DLNA and IR remotes are "orthogonal," but that Guacamayo needed to support them both.
Platforms versus products
I asked whether Guacamayo was focused squarely at do-it-yourself media devices, or if its creators hoped to see it powering consumer products some day. Frydrych replied that he would rather see Guacamayo grow into a Yocto-based platform that others found useful, and which could be tailored to more than one use case. In that regard, he approaches the project as a set of add-ons that enable Yocto to work on media-centric devices that it otherwise could not, rather than as a separate distribution. The key distinction, he said, is establishing a reliable QA process that makes the distribution maintainable. That is an area where Yocto excels, so it makes for the ideal base platform.
Where Guacamayo heads in the future is still up in the air; there is a
general roadmap
on the project's GitHub wiki, but Frydrych emphasizes that he is more
interested enabling others to build a user experience than in
dictating one of his own. "Yocto makes it possible to create a
tailored Linux device in a very short period of time, taking care of
the boring and tedious; Guacamayo should do the same for the
multi-media device category.
"
For now the Rygel-driven streaming media interface of MEX is the only application on the roadmap. That may prove popular enough with users to spawn a following; after all the consumer device space is filled with streaming-only set-top boxes. But each generation of set-top box also adds more and more features (e.g., Boxee adding live TV tuning, or Roku adding USB storage and games). It will be interesting to see if future Guacamayo releases entice third-party developers to try something new, and how the project adapts. The X-less OpenGL environment is certainly different. Clutter has an OpenGL GUI toolkit called Mx (not to be confused with MEX), but Burton cautioned that Mx was lightweight in design and was not intended to provide a full application framework along the lines of Qt or GTK+. Both developers mentioned that if OpenGL alone proves not to be enough, they will target Wayland rather than X in future releases.
The truly interesting match-up will be to see how video-enabled Guacamayo stacks up against a Tizen set-top box. Despite both projects' interest in media playback devices, they approach the task from starkly different angles. Tizen is concerned (on the set-top front) with providing a complete development environment for consumer electronics manufacturers; the platform offers compatibility with desktop Linux and with the other flavors of Tizen device, and it wants to attract all sorts of independent application vendors. Guacamayo (at least at the moment) is targeting DIY devices first, with a limited application set on a minimal, embedded Linux base. Neither one is necessarily right or wrong, but that's what makes watching them entertaining.
Brief items
Distribution quotes of the week
Of course the actions by Microsoft are against anti-trust law in the US and in Europe grossly violate the rule against tying of software and hardware. And claiming "Why you could pirouette and do a handspring backwards, and if Microsoft agrees, then you can install Fedora, so there is no extra bar to installation." is incorrect. Before now we did not have to do the pirouette and handspring. Before the New Microsoft Regime of Booting, we did not have to beg Microsoft to sign our keys.
(1) Live webcast of the candidates playing rock paper scissors lizard spock. This will last at most 17 rounds. If the candidates both stubbornly refuse to budge from spock for all 17 rounds we go to step 2.
(2) Live webcast of the candidates in a 17 minute long hotdog eating contest. Again, we allow for ties and repeat this up to 17 times. If 289 minutes of stuffing themselves with hotdogs does not resolve the matter I think they have both earned a seat on the Board.
Distribution News
Debian GNU/Linux
bits from the DPL: May 2012
Debian Project Leader Stefano Zacchiroli reports on his May activities, starting with the wheezy release. Also covered is the discussion on Debian multimedia, sprints, expenses, and a few other topics.Bit from the Release Team: armhf and s390x
The Debian release team reports that armhf and s390x architectures have release architecture status for wheezy. Bugs specific to these architectures may be release critical and could delay the release of wheezy (Debian 7.0).
Fedora
Election Results for Fedora Board, FAmSCo, and FESCo seats
Fedora project leader Robyn Bergeron has posted results from this week's Fedora Board, Fedora Ambassadors Steering Committee, and Fedora Engineering Steering Committee elections. Bergeron notes that "apparently, we like to keep things interesting around here; the improbable situation of having a tie for a seat has occurred. Read on for details!" There will be a run-off election to break the tie.
Other distributions
CentOS: MirrorList.centos.org is now available on ipv6
The CentOS project has announced that mirrorlist.centos.org is available on ipv6. "This is a full featured service, using geoip on the server end to hand out local-to-user IPv6 mirrors. Every url handed out is verified at regular intervals and only the freshest ones used."
Scientific Linux 4 EOL and archived
Scientific Linux 4 has received no updates since February, but it's remained available in the main repositories. Now it's time for it to move. "On June 12 2012 we will be moving all instances of Scientific Linux 4 into the 'obsolete' directory under /linux/scientific/ . This is the same place that Scientific Linux 3 has been since it was archived. They will be at /linux/scientific/obsolete/". This move will break all yum update operations that use the old path. If you wish to use our archived copies you may update your /etc/yum.repos.d/ files to point to this archived location they can still function while the archive is available.
Newsletters and articles of interest
Distribution newsletters
- Debian Project News (June 11)
- DistroWatch Weekly, Issue 460 (June 11)
- Maemo Weekly News (June 11)
- Ubuntu Weekly Newsletter Issue 269 (June 10)
CentOS Project Release Times
Johnny Hughes explains the steps the project has taken to address lagging point releases. "We now have corporate sponsors who sponsor 2 CentOS Developers to work on the CentOS Project full time. That means that we now have 80 paid hours per week of CentOS Project time where we get do nothing but CentOS Project related work. The sponsors do not ask for anything in return, just faster CentOS updates by the current CentOS developers who get to make the CentOS Project their daily work priority. This should be huge in preventing future delays."
Interview with Patrick Volkerding of Slackware (LinuxQuestions)
LinuxQuestions.org members had some questions for Patrick Volkerding, the founder of Slackware Linux (the oldest Linux distribution still in existence). "I try very hard with Slackware to defer to the wishes of the upstream developers. This isn't the place to be patching in new features, or packaging beta versions (if that can be helped). Lao Tzu said that in order to lead, one must follow. I have my own ideas about how things should work, of course, but I try not to impose them in a selfish manner. The goal is to move forward without abandoning the project's roots. I have nothing against change, but at the same time I try not to let things get juggled around simply for the sake of making them different. People who come back to Slackware after a time tend to be pleasantly surprised that they don't need to relearn how to do everything. This has given us quite a loyal following, for which I am grateful. I guess we're doing something right."
Page editor: Rebecca Sobol
Development
printerd modularizes Linux printing
In May, Red Hat's Tim Waugh and Richard Hughes announced the printerd project; a new print spooler implemented as a D-Bus system service. Printerd is deemed experimental by its creators, but it offers a different approach from the CUPS-based Linux printing system that has dominated distributions for years.
CUPS, of course, includes a number of modules that connect together to form the life-cycle of a print job. There is the spooler, which schedules print jobs initiated by applications, followed by a series of filters that transform the data (including scaling, format changes, color-to-grayscale conversions, and N-up placement), plus back-end drivers for specific printers. Printerd is an effort to write a new scheduler from scratch — one that can be integrated directly with the system services of a modern Linux-based OS.
System services
At a basic level, that design decision means two things. First, the printerd spooler interacts with applications over D-Bus, and second, the spooler uses PolicyKit to enforce security. The CUPS API is based on the Internet Printing Protocol (IPP), which allows client applications to connect to the print server and request information (such as a list of available printers or their capabilities) or submit and check on print jobs. Policy, such as which users are allowed to access a printer, is spelled out in the cupsd.conf configuration file. It allows administrators to control access on the basis of function (such as sending printer jobs or canceling jobs), IP address, username, or group, and it covers general settings like authentication method.
Printerd's D-Bus API is modeled on IPP, but it is not intended to
be an IPP implementation, and thus has some minor differences. Waugh
said in the initial announcement that "the aim is to be able to
implement an IPP server on top of the D-Bus API as a separate
process.
" This separates the IPP server from the local print
spooler, allowing computers with only un-shared, local printers to
dispense with managing an IPP service. By using D-Bus, printerd also
allows client applications to access the print spooler asynchronously,
which is not possible with CUPS. This would be useful for shared
printers, but also for programs that need to print automatically. As
it is today, printing through CUPS either blocks the application or
forces it to start a new thread.
PolicyKit integration does not offer different security features than CUPS's own security framework, but it does make it easier to deploy a system-wide policy across user accounts. As Waugh explains in a May 23 blog post, there is a CUPS plugin called cups-pk-helper that uses PolicyKit to enforce a security policy, but it works by bypassing the cupsd.conf settings altogether. Thus, to restrict access to a certain function (e.g., canceling all print jobs on a shared printer), one would need to specify the security rule in both CUPS and PolicyKit.
Print pipelines and the future
Another modernizing choice in printerd design is that it supports only PDF as its print job format. Waugh's rationale is that basing the workflow on PDF dramatically simplifies the job assembly process; PDF has support for selecting and arranging individual pages built in to the format itself (which is necessary, for example, in automatic duplex printing). The other main contender for a job format is PostScript, but although there are vendor-specific conventions for page-selection and layout manipulation, they are not consistently implemented.
Nevertheless, commenters on the blog took issue with Waugh's assertion
that "generally people print from applications that generate
PDF.
" While it might be true that most documents can be
rendered to PDF for printing, not all can, and not all applications are
designed to produce PDF. Plain text files, for example, are handled
automatically by CUPS filters, and would require a special filter to
be printed with printerd. Many specialized printers, such as those
designed for printing labels or photos, need lower-level output
formats than the standard file types embedded in a PDF,
although in practice they tend to use special drivers (such as those
maintained by the Gutenprint project) as
well, so they are already exceptions to the standard pipeline.
In spite of the alarm expressed by some over printerd's PDF-only policy, it is important to remember that PDF print queues are already the plan settled upon by the vendor-neutral OpenPrinting workgroup at the Linux Foundation. GTK+ and Qt, the two most widely-used application toolkits, already support PDF printing pipelines, as do LibreOffice and Mozilla, two large projects which, together, arguably generate as much of the printed matter on a typical Linux desktop as the toolkits. Ubuntu formally became a PDF-pipeline distribution with the 11.10 release, and the IEEE's Printer Working Group seems poised to adopt PDF as its standard page format, too.
That said, it is interesting to note that when we last examined PDF print queues in March
2012, the issue at hand was that Apple, the corporate parent of CUPS,
had begun making changes to the stack that adversely affected
non-Apple systems. The changes in CUPS 1.6 introduced some additional
work for Linux printing system maintainers, but the bigger fear was
that those changes were just the tip of the iceberg, and that future
releases would make CUPS less and less relevant to Linux. At that
time, Waugh commented
that a Linux-centric CUPS replacement had been discussed, but that it
was not in the works "for the time being
".
Waugh still describes printerd as "experimental" and cautions readers of his blog that it is an attempt to test and develop print pipeline ideas. Among his interests he lists several outstanding problems he would like to solve, including the lag time between canceling a print job and having the printer stop the paper feed, and separating the filter pipeline into a separate process from the driver backend, in order to better more efficiently multiplex multiple simultaneous print jobs.
Technically, applying a filter chain to a print job is a different part of the pipeline than the spooler, so that work would probably not be part of printerd itself. But it seems clear that Waugh is envisioning a post-CUPS printing landscape. Furthermore, regardless of how long it takes to arrive or whether or not printerd is ultimately part of the final product, he seems intent on splitting the individual steps in the print pipeline into separate tools. That is probably a good approach; today it is difficult to explain what CUPS is (at least succinctly) because it bundles so many discrete steps into a single, monolithic package. Modularizing the tools could make it simpler to maintain quality printer drivers, plus cleanly separate local and IPP printing. There is no telling if or when such a modular printing stack would prove preferable to CUPS, but for now printerd and related experiments may at least offer the opportunity to find out.
Brief items
Quotes of the week
Collabora and Fluendo launch GStreamer SDK
Collabora and Fluendo, the two primary development companies behind the open source GStreamer media framework, have created a cross-platform software development kit (SDK) for building GStreamer-based applications. The announcement says the SDK will build applications "to be functionally identical on Windows XP, Windows Vista, Windows 7, Mac OS X (version 10.5 or later on Intel) and all supported Linux platforms". The kit appears to be shell-based, and include a lengthy set of tutorials along with build and packaging tools.
Emacs 24.1 released
GNU Emacs 24.1 is out. New features include (finally) a mechanism for dealing with package installation and repositories, bidirectional text support, an improved completion system, and more. See the NEWS file for vast amounts of detail.MPlayer 1.1 released
It turns out that the long wait for MPlayer 1.0 was in vain - the project went straight to a 1.1 release. "We gave up on 1.0 After a long pause, we decided that it might be a good idea to make a new release. While we had our fun with the naming scheme with lots of 'pre' and 'rc' it seemed time to move on and with everyone incrementing major versions between weekly and monthly we hope to be forgiven for jumping ahead to 1.1." There's a lot of new stuff in this audio and video player release, much of which comes from the recent FFmpeg 0.11 release.
Postgres-XC 1.0.0
Postgres-XC is "an open source project to provide a write-scalable, synchronous multi-master, transparent PostgreSQL cluster solution. It is a collection if tightly coupled database components which can be installed in more than one hardware or virtual machines.". Version 1.0.0—the project's first stable release—is now available; see the documentation page for more information.
GNU Radio 3.6.1
Version 3.6.1 of the GNU Radio software-defined radio package is out. "This release contains numerous bug fixes, some new signal processing blocks, a new Python API documentation system, and the first steps in the code reorganization that will become complete in the 3.7 API release." See the changelog for details.
Vignatti: X on Wayland
Tiago Vignatti describes the mechanism by which X clients will be supported on a Wayland display server. "All X windows created from now on will be redirected to offscreen pixmap and stored on a DRM buffer (via the xwayland video driver); that’s how compositing works on Wayland. The idea is that a X client will behave very likely as a regular Wayland client. Therefore, there’s no protocol calls or any major task involved on xwayland and all happens seamlessly, with the protocol 'conversion' penalty close to nil."
X11R7.7 released
The X.org project has released X11R7.7, which includes "both new features and stability and correctness fixes, including support for reporting multi-touch events from touchpads and touchscreens which can report input from more than one finger at a time, smoother scrolling from scroll wheels, better cross referencing & formatting of the documentation, pointer barriers to control cursor movement, and synchronization fences to coordinate between X and other rendering engines such as OpenGL."
Newsletters and articles
Development newsletters from the last week
- Caml Weekly News (June 12)
- What's cooking in git.git (June 6)
- Perl Weekly (June 11)
- PostgreSQL Weekly news (June 10)
- Ruby Weekly (June 7)
- Tahoe-LAFS Weekly News (June 11)
Open crowdsourcing arrives with PyBossa (The H)
The H looks at PyBossa, a Python framework developed by the Open Knowledge Foundation to build crowdsourcing features. "Pulling together volunteers to work on tasks can be a difficult logistical task, especially if the tasks are small like classifying the contents of a picture or transcribing some text from an existing document. For paid work like that, solutions like Amazon's Mechanical Turk divide up the load and then make the tasks available to a pool of workers, but for open data, open source or other open projects, there haven't been many options until the arrival of PyBossa."
Page editor: Jonathan Corbet
Announcements
Brief items
2012 Linux Scholarship Program Open Now
The Linux Foundation has announced its 2012 Linux Training Scholarship Program. It also announced a new Enterprise Linux Training program aimed at "preparing the next generation of enterprise architects". The deadline for scholarship applications is July 11, 2012.
Articles of interest
IFOSSLR 4.1 available
Volume 4, Issue 1 of the International Free and Open Source Software Law Review is out. Featured topics this time include open licensing and databases, an exploration of what "distribution" of software really means under US copyright law, open hardware licensing, and more.Linus Torvalds: Linux succeeded thanks to selfishness and trust (BBC)
The BBC interviews Linus Torvalds. "So I think that in order to make it in a consumer market, you really do need to be pre-installed. And as Android has shown, Linux really can be very much a consumer product. So it's not that the consumer market itself would necessarily be a fundamentally hard nut to crack, but the 'you need to come preinstalled' thing is a big thing. And on the laptop and desktop market, we just haven't ever had any company making that kind of play. And don't get me wrong - it's not an easy play to make."
The BBC also reports that Linus has co-won the Millennium Technology Prize.
Famous judge spikes Apple-Google case (GigaOm)
GigaOm looks at a surprising turn in the Apple-Google case, and in the patent wars in general. "In his remarkable ruling, U.S. Circuit Judge Richard Posner stated that there was no point in holding a trial because it was apparent that neither side could show they had been harmed by the other’s patent infringement. He said he was inclined to dismiss the case with prejudice — meaning the parties can’t come back to fight over the same patents — and that he would enter a more formal opinion confirming this next week."
US Navy buys Linux to guide drone fleet (The Register)
The Register reports that the US Navy has signed off on an almost $28 million contract from military contractor Raytheon to install Linux ground control software for its fleet of vertical take-off and landing (VTOL) drones. "As for those worried over GPL licensing, the US Department of Defense is well ahead of you. The DOD has already issued guidelines on the use of open source code in its systems, and says the matter is in hand. "The US government can directly combine GPL and proprietary/classified software into a single program arbitrarily, as long as the result is never conveyed outside the U.S. government, but this approach should not be taken lightly," it states. "When taking this approach, contractors hired to modify the software must not retain copyright or other rights to the result (else the software would be conveyed outside the US government.)" (Thanks to Phil Endecott)
New Books
Maintainable JavaScript--New from O'Reilly Media
O'Reilly Media has released "Maintainable JavaScript" by Nicholas Zakas.
Calls for Presentations
Call for Presentations: Postgres Open 2012
Postgres Open will take place September 17-19, 2012 in Chicago, Illinois. The call for talks and workshops is open until June 26. "Presentations should be oriented towards the business or development user of PostgreSQL, and should have substantial technical content."
PyCon Finland 2012 - Call for Proposals
PyCon Finland 2012 will take place October 22-23 in Espoo, Finland. The call for proposals deadline is August 1. They are looking for proposals for both talks and sprints.
Upcoming Events
Events: June 14, 2012 to August 13, 2012
The following event listing is taken from the LWN.net Calendar.
| Date(s) | Event | Location |
|---|---|---|
| June 11 June 16 |
Programming Language Design and Implementation | Beijing, China |
| June 11 June 15 |
YAPC North America | Madison, Wisconsin, USA |
| June 13 June 15 |
2012 USENIX Annual Technical Conference | Boston, MA, USA |
| June 13 June 14 |
HotStorage '12: 4th USENIX Workshop on Hot Topics in Storage and File Systems | Boston, MA, USA |
| June 14 June 17 |
FUDCon LATAM 2012 Margarita | Margarita, Venezuela |
| June 14 June 15 |
TaPP '12: 4th USENIX Workshop on the Theory and Practice of Provenance | Boston, MA, USA |
| June 15 | NSDR '12: 6th USENIX/ACM Workshop on Networked Systems for Developing Regions | Boston, MA, USA |
| June 15 June 16 |
Nordic Ruby | Stockholm, Sweden |
| June 15 June 16 |
Devaamo summit | Tampere, Finland |
| June 16 | Debrpm Linux Packaging Workshop in the Netherlands | The Hague, Netherlands |
| June 19 June 21 |
Solutions Linux Open Source | Paris, France |
| June 20 June 21 |
Open Source Summit (NASA, State Dept, VA) | College Park, MD, USA |
| June 26 June 29 |
Open Source Bridge: The conference for open source citizens | Portland, Oregon, USA |
| June 26 July 2 |
GNOME & Mono Festival of Love 2012 | Boston, MA, USA |
| June 30 July 6 |
Akademy (KDE conference) 2012 | Tallinn, Estonia |
| June 30 July 1 |
Quack And Hack 2012 | Paoli, PA, USA |
| July 1 July 7 |
DebConf 2012 | Managua, Nicaragua |
| July 2 July 8 |
EuroPython 2012 | Florence, Italy |
| July 5 | London Lua user group | London, UK |
| July 6 July 8 |
3. Braunschweiger Atari & Amiga Meeting | Braunschweig, Germany |
| July 7 July 12 |
Libre Software Meeting / Rencontres Mondiales du Logiciel Libre | Geneva, Switzerland |
| July 7 July 8 |
10th European Tcl/Tk User Meeting | Munich, Germany |
| July 8 July 14 |
DebConf12 | Managua, Nicaragua |
| July 9 July 11 |
GNU Tools Cauldron 2012 | Prague, Czech Republic |
| July 10 July 15 |
Wikimania | Washington, DC, USA |
| July 10 July 11 |
AdaCamp Washington, DC | Washington, DC, USA |
| July 11 | PuppetCamp Geneva @RMLL/LSM | Geneva, Switzerland |
| July 11 July 13 |
Linux Symposium | Ottawa, Canada |
| July 14 July 15 |
Community Leadership Summit 2012 | Portland, OR, USA |
| July 16 July 20 |
OSCON | Portland, OR, USA |
| July 26 July 29 |
GNOME Users And Developers European Conference | A Coruña, Spain |
| August 3 August 4 |
Texas Linux Fest | San Antonio, TX, USA |
| August 8 August 10 |
21st USENIX Security Symposium | Bellevue, WA, USA |
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol