User: Password:
Subscribe / Log in / New account Weekly Edition for October 3, 2013

The history of Mesa

By Jake Edge
October 2, 2013
X.Org Developers Conference

The Mesa project (aka "Mesa 3D") is "almost old enough to drink" (21 in the US), said Ian Romanick when introducing Brian Paul's X.Org Developers Conference (XDC) talk. Paul is "the father of Mesa", Romanick said; he was presenting a look back at the 20 (or so) years of development on the open source OpenGL graphics language implementation.

Paul noted that there is some question about when development on Mesa actually began. He has been doing some archeology to try to figure it out, but he doesn't think the August 1993 date noted on the Mesa web site is correct. He thinks he may have gotten started in 1992 and hopes to eventually find the Amiga floppies (with timestamps) to determine that.

[Brian Paul]

He was following the newsgroup in 1992 when Silicon Graphics (SGI) posted an announcement of the OpenGL language. At the time, he was working at the University of Wisconsin on a visualization package using SGI's IRIS GL, which was "terrible". There were other contenders for a replacement (e.g. PHIGS), but SGI was pretty dominant in the graphics world. After the announcement, he read all the documentation "in a couple of days" and "jumped pretty quickly" into OpenGL. It was "beautiful and slick", he said, which is an interesting comparison to OpenGL today, which has 1000 functions and "tons of ways to do the same thing".


Before all of that, though, he got started with computers (and graphics) as a freshman in high school on the TRS-80 Model III. It had 128x48 monochrome graphics (with non-square pixels) and he wrote a paint program for it in BASIC. That's what got him "hooked on graphics", he said.

His first computer was an Atari 800XL, which was a "quantum leap" as it was the first color graphics he had access to (160x192 with four colors, or 320x192 monochrome). Using that, he wrote a 3D modeling program, a 3D renderer using Painter's algorithm, and a ray tracing program. The ultimate output from the ray tracer was a ten-frame animation that took two-and-a-half days to render.

By the time he got to college, he had an Amiga 500, which had 640x400 resolution with 4K colors. Even though it had lots more RAM (1M vs. 64K on the Atari), it still lacked the memory for a real Z-buffer. That didn't prevent him from writing a scan-line rasterizer with lighting and dithering as well as another ray tracer. That was all written in Modula-2 because that compiler was cheaper than the Amiga C compiler, he said with a laugh. He did eventually switch to Matt Dillon's C compiler when that became available.

In the early 90s, he was working on visualization software on "really expensive" workstations (like $180K). The price of those workstations made them prohibitive for most, but there were other barriers too. Each vendor had its own API for graphics, so graphics programs had to have multiple backends to support each. But SGI and IRIS GL were pretty dominant, and that was most of what he was doing then.

There were lots of 2D workstations around at the time, it was 3D hardware that was lacking. The Vis5D 3D visualization package for atmospheric science that he developed needed to be able to run on 2D workstations. He found the VOGL library that was a small subset of IRIS GL. It didn't have triangle rendering or hidden surface removal, both of which he hacked in. "It worked but it was ugly", he said.

When OpenGL was announced, he immediately knew it was the direction he wanted to go. SGI targeted it to replace all of the other alternatives, and some industry players (DEC and IBM) were supportive of that, but others (Sun and HP) were not. But it took a surprisingly long time before vendors had OpenGL implementations. It was 1994 or 1995 before SGI shipped anything, he said, and it wasn't until 1996 that it was available across all SGI models.

Starting from scratch

He began work on Mesa (though it didn't yet have that name) in either 1992 or 1993. He was working on the Amiga at home and on Unix systems at work, using floppy disks to move the code back and forth. He developed the device driver interface to accommodate the two different drawing interfaces (Amiga and Xlib).

He had a "lot more spare time then", he said, so he had a mostly complete OpenGL implementation in November 1994. It was still missing some features, but he was using it at work and thought it might be useful to others. So he started talking with SGI about open-sourcing the code.

SGI was "pretty reasonable", he said, contrasting that reaction to what he thinks Microsoft would have said to a similar request. He and SGI went back and forth a bit, but eventually the company said "go for it". SGI said he shouldn't use the letters "GL" in the name and that it needed a disclaimer saying it was not a "real OpenGL". The irony is that today Mesa is a real OpenGL "and we have the certification to prove it", as an audience member noted. The name, Mesa, "just popped into my head one day", Paul said.

He thinks that SGI was receptive to the idea of releasing Mesa because it wanted to get the interface out in front of people as much as possible. Open source would "push that goal forward". Folks that didn't have OpenGL for their workstation could get Mesa instead, so it filled a hole in the ecosystem.

He got permission to release the code in early 1995 and announced Mesa 1.0 beta on on February 3, 1995. When he did that, he had no feel for how much interest there would be, but within days it became clear that there was lots. He started receiving dozens of emails about the project, ranging from "wow" or "thanks" to patches.

To support the project, he wrote his first page for the web, which was just getting traction at that point. He also set up a mailing list. He only had email access at work, so he had to try to deal with the mailing list before work or on weekends. As the project quickly ramped up, his boss, Bill Hibbard, was quite supportive and allowed him to use part of his workday for Mesa. The project owes a lot to Hibbard, he said.

Early days

In the early days, the project faced a number of issues. Supporting multiple different systems (IRIX, SunOS, HPUX, ...) was sometimes difficult due to compiler incompatibilities. Some of those kinds of problems are still seen today. The displays could be problematic as well. 24-bit displays were expensive, so most displays were 8 or 16-bit. That meant dealing with color maps and dithering. He remembers spending "days and weeks" on dithering patterns, which just "seems crazy" today.

Handling lots of patches and rolling out new releases without using revision control was another problem area. He's not sure why he didn't use SCCS or RCS, but he didn't have any revision control until Mesa moved to a SourceForge predecessor in the late 1990s, he said. Performance was another issue. PCs had 16MHz 386 processors. Workstations were faster, but "still not super fast". Calling Mesa "interactive 3D graphics" was sometimes an overstatement, he said.

But OpenGL was "pretty small back then", so he could still keep all of the Mesa code in his head in those days. That "was a lot of fun", but with a million lines of code today, Mesa is "pretty overwhelming".

Late 90s

In the late 1990s, 3D was really starting to take off. There was consumer 3D hardware available from companies like NVIDIA, ATI, Matrox, 3dfx, and others, with price points that were fairly reasonable. Those cards used Direct3D, Glide (for 3D effects), or some one-off vendor API, and there was little OpenGL support.

SGI did open up its GLX client/sever code so that programs could use OpenGL in X windows. The Utah GLX project, which was an XFree86 server-side driver that provided indirect rendering, also got started. Paul said he was not directly involved in Utah, but it attracted some talented developers to Mesa. That included Keith Whitwell who had done a lot of work to increase the performance of Mesa. Around that time, John Carmack of id Software donated $10,000 to Mesa, which funded more of Whitwell's work.

In addition, Precision Insight was formed as a company to work on X and other related technology in 1998. Paul met the founders of the company at the SIGGRAPH 99 conference and joined Precision Insight in September 1999. That meant that he could finally work on Mesa, Direct Rendering Infrastructure (DRI), and GLX full time. Prior to that, most of his work on Mesa had been done in his spare time.

In the 2000s, DRI became well established. Precision Insight was acquired by VA Linux, but then the dot-com bubble popped "and we all got laid off", so he helped form Tungsten Graphics with other former Precision Insight employees in 2001. There were multiple releases of Mesa (4.x–7.x) over that decade with "lots of hardware drivers". The Gallium3D project was started by Whitwell in 2008 to simplify creating new 3D drivers. The existing device driver interface was not a good match to what the graphics hardware was doing, so Gallium3D provided a different interface that would help reduce the code duplication in Mesa drivers, he said.


Paul said that he never expected Mesa to be so successful, nor to last as long as it has. He would have guessed that vendors would put out their own OpenGL libraries so that there would be no need for Mesa. But people still need software rendering, using llvmpipe and other drivers. It's also important to have an open source implementation available that people can adapt and use for new projects, he said.

There have been as many as 1000 contributors to Mesa over the years, which is far beyond what he expected. The project has grown in size and complexity beyond his expectations as well. Mesa is in all of the Linux distributions and used by thousands (or millions?) of users every day. By his measure, it is a "very successful project".

Paul outlined what he saw as the keys to that success. Mesa was "the right project at the right time". He happened to have a job that caused him to follow, which led him to start the project. If he hadn't, though, someone else would have around the same time, he said. He also has always liked the perspective that Fred Brooks has espoused about computer scientists as "tool makers". He took that idea to heart and made a tool that others use to do their job. In the early days, that job was visualization, today it is entertainment, which is "not as noble" as it once was, "but still pretty cool".

As project lead, he let people "smarter than me" take control of areas they were interested in. Beyond that, the Mesa development community has always been "pretty civilized". There are no insults or screaming in the mailing list. He has seen other open source leaders who do that and he doesn't like it at all, he said. But perhaps the most significant reason for Mesa's success is that it was following a specification. "We are the carpenters" who are building a "house that someone else designed".

He closed by thanking the developers who contributed "amazing work" to the project over the years. Mesa is now his calling card and the project has greatly exceeded his expectations. In addition, he met some of his closest friends through the project.

At the start of the talk, Paul said it has been around five years since he last attended an XDC. He is now with VMware (which bought Tungsten Graphics along the way). He said that he was quite impressed to see all of the new faces of those involved with X today. Keith Packard likely summed up the feelings of many in the room (and beyond) when he thanked Paul for his longtime work on an important project that is available as open source so that it can be used by all.

[I would like to thank the X.Org Foundation for travel assistance to Portland for XDC.]

Comments (8 posted)

TT-RSS makes a comeback

By Nathan Willis
October 2, 2013

Veterans of The Great Google Reader Shutdown scattered in a number of different directions once the search giant's RSS- and Atom-feed reading service closed its doors in July 2013. Quite a few proprietary web services stepped up in an attempt to attract former Google Reader users, but free software projects gained prominence as well. The ownCloud project added a feed reading application, for example, but Tiny Tiny RSS (TT-RSS) had a considerable head start. The project recently released its latest update, the latest from a post–Google Reader surge in development.

TT-RSS has been in development for more than eight years, but prior to the March announcement of the Google Reader shutdown, the pace of development had slowed. The project still made regular updates every few months, but there was not a steady influx of new developers and the feature set grew in small increments with each new release. The uptick in interest after the Reader announcement was dramatic; one need only look at the Ohloh graphs for contributor count and commits to see the difference. From a low of 54 commits by two committers in January, activity shot up to 564 commits by 31 committers in March. Most of those commits came from lead developer Andrew Dolgov, but it would be fair to say that the Reader deactivation reinvigorated interest in the project.

The first new major release following this flurry of activity was version 1.8 in June. That update brought updates to feed parsing (including dropping the Simplepie parsing library in favor of a native TT-RSS parser), moved several optional plugins out of the core codebase into a separate contrib repository, and reworked the user interface. The moved plugins constitute a significant number of lines of code, including several authentication modules (such as IMAP, LDAP, and RADIUS authentication) and connections to a variety of third-party web services. Those web-service plugins include the ability to share items from TT-RSS via Twitter,, Google+, Pinterest, ownCloud, and several other well-known sites.

Version 1.9 followed in late July. Several parser fixes were included, as were translation updates and a refresh of the icon set. Arguably the most visible changes were the addition of a plugin that serves a feed of the user's shared articles and the migration of TT-RSS from GPLv2 to GPLv3. Version 1.10 arrived on September 21, containing only minor changes and bug fixes. [TT-RSS]

There are a glut of feed-reading options available these days, and many look the same on the surface. The TT-RSS interface will feel familiar to anyone who has used Google Reader, Feedly, Digg Reader, NewsBlur, or the like. A scrollable list on the left-hand side shows all of the subscribed feeds by category (plus special categories for saved and starred items) and how many unread entries are available, while the remainder of the screen is occupied by split-pane news reading interface: one can view feeds by headline or by full text, mark or unmark them as read, and so on.

But TT-RSS still offers a handful of features that the majority of the competition does not—even though those features may not jump out of the interface. For one thing, it is one of the few feed readers that supports subscribing to username/password-protected feeds. There are not a lot of feeds out there that require authentication, but that is something of a chicken-and-egg problem: lack of reader support decreases the demand for authenticated feeds, and vice versa.

Another example of TT-RSS's feature set is its support for automatically detecting duplicate articles and removing the duplicates from the unread item interface. This, too, is not a common problem at the global scale, perhaps, but it is one that many free software fans are likely to run into. Planet software aggregates multiple blogs, but in doing so a single person's blog may get carried by multiple planets if that person is involved in multiple projects. Having reader software like TT-RSS filter out the duplicates that come from extra-active bloggers is surely a better solution than trying to persuade those bloggers to participate in fewer projects.

As for the changes in the latest releases, TT-RSS does feel snappier than it used to. That may be attributable to the new, lightweight feed parsing code, to the reduction in the number of plugins installed, or even to the accumulated improvements of all of those new contributors and Dolgov's surge of development in March. Whatever the cause, I last used TT-RSS more than two years ago and frequently found it pausing momentarily when switching to a new folder or when loading large posts; I have noticed no such hangs with version 1.10.

A resurgence in interest is welcome news for any free software project. The lone dark cloud in the recent uptick in TT-RSS's popularity has been the project's hostility toward forum users who bring suggestions or questions that the team does not like. The FAQ directs users with Google Reader–related requests to an antagonistic forum thread where one such user is ridiculed, and the same FAQ page defends Dolgov's right to "either mock or not deal with people" who ask "stupid questions again and again because new people gracing tt-rss community with their presence largely can't be bothered to read anything." The justification given for this approach is that community management is too time-consuming.

No doubt community management is a task that requires a significant time investment, but it is hardly an unsolved problem—and one would think the ten-fold increase in volunteers seen last March could have provided some person-power that could be directed toward community building. Prickly project leads are not a new problem, but TT-RSS is quality code and it was in essence handed a ready-made stable of millions of potential users when Google Reader closed; it would be a shame to drive them away.

Comments (7 posted)

Some tidbits from XDC

By Jake Edge
October 2, 2013
X.Org Developers Conference

The X.Org Developers Conference (XDC) is a three-day, one-track event with presentations covering many different parts of the graphics stack. This year it was held September 23–25 at Portland State University in Portland, Oregon—home of Voodoo Doughnuts, which were provided daily. XDC is an intense experience and this year's edition will lead to a few more articles in the coming weeks. There were also a few shorter sessions with some news and plans that seem worth reporting on here.

Non-profit status

X.Org Foundation board member Peter Hutterer reported on the state of the foundation. The most recent news about the foundation was that it had lost its 501(c)(3) (US) non-profit status in August. Hutterer was happy to report that had all been reversed. With help from the Software Freedom Law Center (SFLC), which helped the foundation become a non-profit in 2012, the foundation was able to regain its status. The paperwork was "still in transit", he said, but the non-profit status was restored.

[Peter Hutterer]

Part of the problem that led to the revocation of the non-profit status was that it is a lot of work to maintain a 501(c)(3) organization. The members of the foundation board are not lawyers or accountants, he said, and some of them are not even living in the US, which makes it that much more difficult. So the board decided to look into organizations that manage free software projects, foundations, and the like. Umbrella organizations like the Software Freedom Conservancy (SFC), Apache Software Foundation (ASF), and Software in the Public Interest (SPI) are set up to handle much of the paperwork for various types of projects.

The board has voted that SPI is the right umbrella organization for the X.Org Foundation. That switch is not finalized as it may require a vote of the X.Org membership, Hutterer said. The board will be consulting with the SFLC and looking at the by-laws to determine that. Assuming the change is made, SPI would take 5% of any donations made to the foundation for the work that it does, which "seems fair", he said.

The foundation has "a slab of money" that remains from a number of years ago, when it was getting donations of $100,000 or so per year. It uses that money to put on XDC and to sponsor the travel of several participants (four this year, including a Google Summer of Code student and an LWN editor). It also funds GSoC students and participants in the X.Org Endless Vacation of Code program. The pile of money is enough to last for another four or five years, Hutterer said, before the foundation needs to consider doing some fundraising—something that's never been done since he became involved.

[Meeting room]

The foundation is also moving banks after HSBC closed its account for unclear reasons. "Banks are fun", Hutterer said with a laugh. The current plan is to move to Bank of America, he said.

The Board of Directors consists of eight people and four of those seats turn over every year. There are 78 members of the foundation, which is "lots better than it was a couple of years ago". Hutterer strongly encouraged those present to think about joining, which allows voting in elections and has a few other benefits.

X server 1.15 planning

Keith Packard took the floor on the first day to discuss plans for the X server 1.15 release. It was supposed to have been released the week of XDC, but in August he and others realized that the release itself was "really boring". He asked the assembled developers if there were any features due in 1.15 that they were "desperate to have". Hutterer mentioned that having touch input working would be nice, but Packard noted that those changes had been backported to 1.14. [Keith Packard]

In fact, as far as he knows, all of the security, stability, and usability bug fixes have been added to 1.14. The 1.14 release manager has been making minor releases with those changes, which are, of course, ABI-compatible with 1.14—unlike 1.15. At this point, Packard said, 1.15 looks like "1.14 plus an ABI change".

There are, however, many features that are awaiting the release of 1.15 before they get merged. So, an idea that had been batted around on IRC was to delay the release of 1.15 until it had some features of note. Those features might include a rewrite of the Xephyr nested X server (that deleted "many thousands of lines of code, which is what we do best at Xorg", Packard said, pronouncing the last word as "zorg"), Packard's own DRI3 and Present extensions which are getting close to being ready to merge, some XWayland changes, Adam Jackson's GLX rewrite (which removes around 40,000 lines of code), and possibly others.

Packard would talk about DRI3 and Present later in the conference, as would Jackson about the GLX rewrite, so the final decision would be made after those discussions. All of the proposed features seemed like they would plausibly be ready in time for a code freeze at the end of October. The normal pattern would be for a two-month stabilization period after that, putting the release of 1.15 at the end of the year. "A Christmas present", Jackson suggested.

An informal straw poll of those in the room found all in favor of the proposed change, but there wasn't any real emotion one way or the other. "Consensus by apathy", one developer suggested, which is a "hallmark of X.Org" added another—to chuckles around the room. Packard encouraged anyone with additional features they would like to see in 1.15 to "let us know". In the end, the X.Org calendar shows a final 1.15 release scheduled for December 25.

Links to the slides and videos for most of the sessions can be found from the XDC schedule page.

[I would like to thank the X.Org Foundation for travel assistance to Portland for XDC.]

Comments (4 posted)

Page editor: Jonathan Corbet


Integrity and embedded devices

By Jake Edge
October 2, 2013
Linux Security Summit

David Safford's talk for the 2013 Linux Security Summit was in two parts—with two separate sets of slides. That's because the US Department of Homeland Security (DHS), which sponsored IBM's work on hardware roots of trust for embedded devices—part one of the talk—was quite clear that it didn't want to be associated with any kind of device cracking. So part two, which concerned circumventing "verified boot" on a Samsung ARM Chromebook, had to be a completely separate talk. The DHS's misgivings notwithstanding, the two topics are clearly related; understanding both leads to a clearer picture of the security of our devices.

The DHS is interested in what can be done to verify the integrity of the code that is running on all types of systems. For servers, desktops, and mobile devices, there are a variety of existing solutions: crypto hardware, secure boot, Trusted Platform Module (TPM) hardware, and so on. But for low-cost embedded devices like home routers, there is no support for integrity checking. The DHS is also interested in integrity checking for even lower cost sensors, Safford said, but those don't run Linux so they weren't part of his investigations.

Because routers and the like are such low-cost and low-margin systems, the researchers focused on what could be done for "zero cost" to the manufacturer. The team looked at four devices: the venerable Linksys WRT54G router (which spawned projects like OpenWrt and DD-WRT), the Pogoplug local network attached storage (NAS) cache, and two other router devices, the TP-Link MR3020 and D-Link DIR-505. All of those typically retail for around $50.

The MR3020, which was the focus for much of the talk, retails for $30. It has very limited space in its 4M flash chip, so the challenge to adding integrity features is "not so much technical" as it is in "squeezing things in" to the flash. For example, the MR3020 only has 64K of space for U-Boot, 1M for the kernel, and 2.8M for the root filesystem.

Safford gave a handful of examples of router vulnerabilities over the last few years. Beyond just theoretical examples, he noted that 4.5 million home routers were actually compromised in Brazil in 2011. While the vulnerabilities he listed were at the higher levels (typically the web interface), they do show that these embedded devices are most certainly targets.

So, without increasing the cost of these devices, what can be done to ensure that the firmware is what it is expected to be? In the supply chain, the wrong firmware could be added when the system is built or changed somewhere along the way. Safford said that IBM had some (unnamed) customers who had run into just this kind of problem.

So there needs to be a way to "measure" the firmware's integrity in hardware and then to lock the firmware down so that rootkits or other malware cannot modify it. In addition, these devices typically do not have support for signed updates, so malicious update files can be distributed and installed. There is also no ability for the system to refuse to boot if the code has been changed (i.e. secure or trusted boot).

Providing those capabilities was the goal for the project, he said. He showed a table (also present in his slides [PDF] and associated paper [PDF]) outlining the abilities of each of the devices in four separate integrity categories: "Measure BIOS?", "Lock BIOS?", "Secure local updates?", and "Secure Boot?". All of the boxes were "No", except that both the Pogoplug and WRT54G had a way to measure—verify—the firmware (by reading it using SATA via an immutable boot ROM and JTAG, respectively). By the end of the talk, those boxes had all been changed to "Yes" by the changes Safford and his team had made.

The traditional approaches for integrity revolve around either attestation (e.g. trusted boot) or a trusted chain of signed code as in UEFI secure boot. Attestation means that a system uses a TPM to measure everything read and executed, then sends that information to a trusted system for verification before being allowed to continue. There are several National Institute of Standards and Technology (NIST) standards that govern parts of the integrity puzzle, including trusted boot, but there are none, at least yet, that govern secure boot. Safford is working with NIST to get that process started.

Since a TPM chip is "expensive" ($0.75), it violates the zero-cost constraint. But in order to verify the firmware, it must be read somehow. The firmware itself cannot be involved in that step as it may lie about its contents to avoid malware detection. The Serial Peripheral Interface (SPI) bus provides a mechanism to read the contents of the flash for devices lacking other means (e.g. JTAG). That bus can be shared if it has proper buffering, but both the MR3020 and DIR-505 lack the resistors needed.

[Bus Pirate on MR3020]

Enter the Bus Pirate—a device that can be used to read the SPI bus. Using it requires adding three buffering resistors to the Atheros System-on-Chip (SoC) used by the devices, but that adds less than $0.01 to the cost of the device, which is close enough to zero cost that device makers can probably be convinced. That means that users (or device makers) can verify the contents of the flash fairly inexpensively (a Bus Pirate costs around $30).

Once the contents of the flash are verified, there needs to be a way to lock it down so that it can only be modified by those verified to be physically present. The SPI flash chips used by all of the devices have a status register that governs which addresses in the flash can be written, along with an overall write-disable bit. That register can be locked from any updates by holding the chip's write-protect (!WP) pin low. Physical presence can be proved by holding down a button at boot to drive !WP high.

Safford showed the modifications made to the MR3020 and DIR-505 to support the physical presence test. The WPS (Wireless Protected Setup) button was repurposed on the MR3020, while an unused sliding switch position was used on the DIR-505. The paper indicates that similar changes were made on the other two devices. Both the slides and paper have pictures of the modifications made to the devices. In addition, U-Boot was modified so that it locks the entire flash on each boot, but if !WP is held high when power is applied, U-Boot will unlock the flash.

Adding secure boot support to U-Boot was the next step. Once the root of trust is extended into the kernel, the kernel's integrity subsystem can take over to handle integrity verification from there. So it is a matter of verifying the kernel itself. The modified U-Boot will use a public key that is stored at the end of its partition to verify the signature of the kernel. That signature is stored at the end of the kernel partition.

As mentioned earlier, the trick is in getting that code (and key) to fit into the 64K U-Boot partition. Using code derived from PolarSSL, with everything unneeded removed, the modified U-Boot weighed in at 62K. Though Safford was never very specific, the U-Boot modifications must also provide RSA signature checking for updates to the firmware. Support for signed updates is one of the integrity requirements that were successfully tackled by the project.

Through some effectively zero-cost modifications, and some changes to the firmware, the team was able to achieve its integrity goals. All of the devices now support all four of the integrity requirements they set out to fulfill.

Breaking verified boot

Moving on to the DHS-unapproved portion of the talk, Safford showed how one can take control of a Samsung ARM Chromebook. The work on that was done in his spare time, he said, but many of the tools used for adding secure boot for embedded devices are the same as those for removing and altering a system with secure boot. The Chromebook is a "very secure" system, but the verified boot (VB) mechanism does not allow users to take control of the boot process.

However, a fairly simple hardware modification (removing a washer to change the !WP signal) will allow the owner to take control of the device, as Safford found. Beyond the hardware change, it also requires some scripts and instructions [tar.gz] that Safford wrote. Unlike the embedded devices described above, there is a full 4M flash just for U-Boot on the Chromebook, so there is "an embarrassment of riches" for adding code on those systems. VB has been added to the U-Boot upstream code, incidentally, but it is way too large (700K) for use in routers, he said.

In the normal VB operation, there is no way to write to the upper half of the SPI flash, which contains a copy of U-Boot and Google's root key. That key is used to verify two keys (firmware and kernel) stored in the read-write half of the SPI flash. The firmware key is used to verify another copy of U-Boot that lives in the modifiable portion of the flash. That U-Boot is responsible for verifying the kernel (which actually lives in a separate MMC flash) before booting it.

Holding down ESC and "refresh" while powering on the system will boot whatever kernel is installed, without checking the signatures. That is the "developer mode" for the system, but it circumvents secure boot, which is not what Safford set out to do. He wants to use secure boot but control the keys himself. In addition, developer mode must be enabled each time the system boots and you get a "scary screen" that says "OS verification is OFF".

A less scary approach is to use a non-verifying U-Boot that gets installed in place of the kernel. That U-Boot is signed, but does no verification of the kernel (installed in a different part of the MMC flash) before booting it. That way you don't have to invoke developer mode, nor do you get the scary screen, but you still don't get secure boot.

[ARM Chromebook washer]

Removing the washer is the way forward as it allows read-write access to the entire SPI flash. Once that is done, Safford has a set of scripts that can be run from a developer-mode kernel to create new key pairs, sign the read-write U-Boot, sign the kernel, and verify all of the signatures. If any of that goes wrong, one may end up at the "Chrome OS is missing or damaged" screen, which actually means the device hasn't been "bricked" and can be restored from a USB device. Even in the event of bricking, one can recover the device using Bus Pirate as the SPI flash is properly buffered, he said (seemingly from a fair amount of experience).

As part of his demo, he wanted an easy way to show that he had gained control of the low-level boot code in the SPI flash. He decided to change the "chrome" text in the upper left of the "scary screen" to "DaveOS", which actually turned out to be one of the harder ways to demonstrate it. Because of the format of the logo and where it was stored in the flash, it turned out to be rather painful to change, he said with a chuckle.

As Kees Cook pointed out, the washer removal trick was a deliberate choice in the design of the system. Google and Samsung wanted people to be able to take control of the keys for the device, but didn't want an attacker to be able to do so quickly while the user's attention was momentarily distracted. Safford agreed that it was a reasonable compromise, but that it is important for users to be able to set their own keys.

The slides [PDF] for the second half of the talk are instructive as well, with a number of pictures of the infamous washer, scary and DaveOS screens, the Bus Pirate in action, and so on. Seeing the problem from both angles, adding and subtracting secure boot functionality, was useful to help better understand integrity verification. Techniques like secure boot certainly can be used in user-unfriendly ways to lock down devices, but it can also provide some amount of peace of mind. As long as users can provide their own keys—or disable the feature entirely—secure boot is likely be a boon for many.

[I would like to thank LWN subscribers for travel assistance to New Orleans for LSS.]

Comments (8 posted)

Brief items

Security quotes of the week

Now think it through. If Verisign is owned by the NSA, and a Russian CA is owned by FSB, and a Chinese CA is owned by that government, and all three of these compromised CAs agree on a cert, what does it mean?

It means the cert is probably accurate, or about as accurate as you can possibly get, without going over to the server certing it yourself. If those three parties are conspiring to disrupt your Amazon order, then I'm afraid you're not going to get your package, no matter what crypto you use. :-)

— Slashdot user Sloppy (Thanks to Zenaan Harkness.)

The public now has to think about the fact that anybody in public life, or person in a position of influence in government, business or bureaucracy, now is thinking about what the NSA knows about them. So how can we trust that the decisions that they make are objective and that they aren't changing the decisions that they make to protect their career? That strikes at any system of representative government.
— Former Microsoft privacy adviser Caspar Bowden (Thanks to Michael Kerrisk.)

Comments (10 posted)

New vulnerabilities

chicken: code execution

Package(s):chicken CVE #(s):CVE-2013-4385
Created:September 30, 2013 Updated:February 10, 2014
Description: From the Red Hat bugzilla:

Chicken, a compiler for the Scheme programming language, is found to have a buffer overrrun flaw due to the read-string! procedure from the "extras" unit, when used in a particular way.

It was found that there was a missing check for the situation when NUM was at #f (the scheme value for false) in the buffer as the buffer size, then it will read beyond the buffer until the input port is exhausted. This may result in a DoS or a remote code execution.

Though currently all stable releases are vulnerable to this flaw, there is a simple workaround to be used in code that uses read-string!: simply convert all (read-string! #f buf ...) invocations to (read-string! (string-length buf) buf ...) or, if possible, use the non-destructive read-string procedure from the same unit.

Fedora FEDORA-2013-22130 chicken 2014-02-09
Fedora FEDORA-2013-17912 chicken 2013-11-26
Fedora FEDORA-2013-17868 chicken 2013-09-30
Gentoo 201612-54 chicken 2016-12-31

Comments (none posted)

davfs2: privilege escalation

Package(s):davfs2 CVE #(s):CVE-2013-4362
Created:September 27, 2013 Updated:December 2, 2016

From the Debian advisory:

Davfs2, a filesystem client for WebDAV, calls the function system() insecurely while is setuid root. This might allow a privilege escalation.

Debian DSA-2765-2 davfs2 2014-07-16
Fedora FEDORA-2013-17836 davfs2 2013-12-19
Fedora FEDORA-2013-17853 davfs2 2013-12-19
Mageia MGASA-2013-0304 davfs2 2013-10-11
Mandriva MDVSA-2013:244 davfs2 2013-09-30
Debian DSA-2765-1 davfs2 2013-09-26
Gentoo 201612-02 davfs2 2016-12-02

Comments (none posted)

firefox: denial of service

Package(s):firefox CVE #(s):CVE-2013-1723
Created:September 27, 2013 Updated:October 2, 2013

From the CVE entry:

The NativeKey widget in Mozilla Firefox before 24.0, Thunderbird before 24.0, and SeaMonkey before 2.21 processes key messages after destruction by a dispatched event listener, which allows remote attackers to cause a denial of service (application crash) by leveraging incorrect event usage after widget-memory reallocation.

openSUSE openSUSE-SU-2014:1100-1 Firefox 2014-09-09
Mageia MGASA-2013-0329 iceape 2013-11-20
openSUSE openSUSE-SU-2013:1499-1 Mozilla 2013-09-29
Gentoo 201309-23 firefox 2013-09-27
openSUSE openSUSE-SU-2013:1495-1 thunderbird 2013-09-27
openSUSE openSUSE-SU-2013:1491-1 seamonkey 2013-09-27
openSUSE openSUSE-SU-2013:1493-1 firefox 2013-09-27

Comments (none posted)

glibc: multiple vulnerabilities

Package(s):glibc CVE #(s):CVE-2013-4788 CVE-2013-4332
Created:September 30, 2013 Updated:October 11, 2013
Description: From the OpenWall advisories [1; 2]:

I recently discovered three integer overflow issues in the glibc memory allocator functions pvalloc, valloc and posix_memalign/memalign/aligned_alloc. These issues cause a large allocation size to wrap around and cause a wrong sized allocation and heap corruption. (CVE-2013-4332)

This bug was discovered in March 2013 while we were developing the RAF SSP technique. The glibc bug makes it easy to take advantage of common errors such as buffer overflows allows in these cases redirect the execution flow and potentially execute arbitrary code.

All statically linked applications compiled with glibc and eglibc are affected, independent of the operating system distribution. Note that this problem is not solved by only patching the eglibc, but it is also necessary to recompile all static executables. As far I know there are a lot of routers, embedded systems etc., which use static linked applications. Since the bug is from the beginning of the PTR_MANGLE implementations (years 2005-2006) there are a ton of vulnerable devices. (CVE-2013-4788)

Debian-LTS DLA-165-1 eglibc 2015-03-06
Gentoo 201503-04 glibc 2015-03-08
SUSE SUSE-SU-2014:1128-1 glibc 2014-09-15
SUSE SUSE-SU-2014:1122-1 glibc 2014-09-12
Scientific Linux SLSA-2013:1605-2 glibc 2013-12-03
Mageia MGASA-2013-0340 glibc 2013-11-22
Oracle ELSA-2013-1605 glibc 2013-11-26
Mandriva MDVSA-2013:284 glibc 2013-11-25
Mandriva MDVSA-2013:283 glibc 2013-11-25
Red Hat RHSA-2013:1605-02 glibc 2013-11-21
Ubuntu USN-1991-1 eglibc 2013-10-21
Scientific Linux SLSA-2013:1411-1 glibc 2013-10-10
Oracle ELSA-2013-1411 glibc 2013-10-09
Red Hat RHSA-2013:1411-01 glibc 2013-10-08
openSUSE openSUSE-SU-2013:1510-1 glibc 2013-09-30
Fedora FEDORA-2013-17475 glibc 2013-09-28

Comments (1 posted)

gpg2: information disclosure

Package(s):gpg2 CVE #(s):CVE-2013-4351
Created:September 27, 2013 Updated:November 13, 2013

From the openSUSE bug report:

RFC 4880 permits OpenPGP keyholders to mark their primary keys and subkeys with a "key flags" packet that indicates the capabilities of the key [0]. These are represented as a set of binary flags, including things like "This key may be used to encrypt communications."

If a key or subkey has this "key flags" subpacket attached with all bits cleared (off), GnuPG currently treats the key as having all bits set (on). While keys with this sort of marker are very rare in the wild, GnuPG's misinterpretation of this subpacket could lead to a breach of confidentiality or a mistaken identity verification.

Gentoo 201402-24 gnupg 2014-02-21
Fedora FEDORA-2013-18647 gnupg 2013-11-13
Scientific Linux SLSA-2013:1459-1 gnupg2 2013-10-24
Scientific Linux SLSA-2013:1458-1 gnupg 2013-10-24
Oracle ELSA-2013-1459 gnupg2 2013-10-24
Oracle ELSA-2013-1459 gnupg2 2013-10-24
Oracle ELSA-2013-1458 gnupg 2013-10-24
CentOS CESA-2013:1459 gnupg2 2013-10-25
CentOS CESA-2013:1459 gnupg2 2013-10-24
CentOS CESA-2013:1458 gnupg 2013-10-25
Red Hat RHSA-2013:1459-01 gnupg2 2013-10-24
Red Hat RHSA-2013:1458-01 gnupg 2013-10-24
Fedora FEDORA-2013-18676 gnupg 2013-10-12
Debian DSA-2774-1 gnupg2 2013-10-10
Debian DSA-2773-1 gnupg 2013-10-10
Ubuntu USN-1987-1 gnupg, gnupg2 2013-10-09
Mandriva MDVSA-2013:247 gnupg 2013-10-10
Mageia MGASA-2013-0299 gnupg2 2013-10-10
openSUSE openSUSE-SU-2013:1532-1 gpg2 2013-10-08
openSUSE openSUSE-SU-2013:1526-1 gpg2 2013-10-06
openSUSE openSUSE-SU-2013:1494-1 gpg2 2013-09-27

Comments (none posted)

kernel: denial of service

Package(s):kernel CVE #(s):CVE-2013-4205
Created:September 27, 2013 Updated:October 2, 2013

From the Ubuntu advisory:

A memory leak was discovered in the user namespace facility of the Linux kernel. A local user could cause a denial of service (memory consumption) via the CLONE_NEWUSER unshare call.

Ubuntu USN-1971-1 linux-lts-raring 2013-09-27
Ubuntu USN-1974-1 kernel 2013-09-27

Comments (none posted)

kernel: information leak

Package(s):linux-2.6 CVE #(s):CVE-2013-2239
Created:September 30, 2013 Updated:October 2, 2013
Description: From the Debian advisory:

Jonathan Salwan discovered multiple memory leaks in the openvz kernel flavor. Local users could gain access to sensitive kernel memory.

Debian DSA-2766-1 linux-2.6 2013-09-27

Comments (none posted)

kernel: off by one error

Package(s):kernel CVE #(s):CVE-2013-4345
Created:October 1, 2013 Updated:October 23, 2013
Description: From the Red Hat bugzilla:

A flaw was found in the way ansi cprng implementation in the Linux kernel processed non-block size aligned requests. If several small requests are made that are less than the instances block size, the remainder for loop code doesn't increment rand_data_valid in the last iteration, meaning that the last bytes in the rand_data buffer gets reused on the subsequent smaller-than-a-block request for random data.

openSUSE openSUSE-SU-2014:0766-1 Evergreen 2014-06-06
Debian DSA-2906-1 linux-2.6 2014-04-24
Ubuntu USN-2158-1 linux-lts-raring 2014-03-31
Ubuntu USN-2110-1 linux-ti-omap4 2014-02-18
Ubuntu USN-2109-1 kernel 2014-02-18
Ubuntu USN-2074-1 linux-ti-omap4 2014-01-03
Ubuntu USN-2076-1 linux-ti-omap4 2014-01-03
Ubuntu USN-2072-1 linux-ti-omap4 2014-01-03
Ubuntu USN-2070-1 linux-lts-saucy 2014-01-03
Ubuntu USN-2068-1 linux-lts-quantal 2014-01-03
Ubuntu USN-2071-1 kernel 2014-01-03
Ubuntu USN-2075-1 kernel 2014-01-03
Ubuntu USN-2064-1 kernel 2014-01-03
Ubuntu USN-2065-1 EC2 kernel 2014-01-03
Oracle ELSA-2014-3002 kernel 2014-02-12
Scientific Linux SLSA-2013:1645-2 kernel 2013-12-16
Oracle ELSA-2013-2584 kernel 2013-11-28
Oracle ELSA-2013-2584 kernel 2013-11-28
Oracle ELSA-2013-2585 kernel 2013-11-28
Oracle ELSA-2013-2585 kernel 2013-11-28
Oracle ELSA-2013-2583 kernel 2013-11-28
Red Hat RHSA-2013:1645-02 kernel 2013-11-21
Red Hat RHSA-2013:1490-01 kernel-rt 2013-10-31
Oracle ELSA-2013-1645 kernel 2013-11-26
Scientific Linux SLSA-2013:1449-1 kernel 2013-10-23
Oracle ELSA-2013-1449 kernel 2013-10-23
CentOS CESA-2013:1449 kernel 2013-10-23
Red Hat RHSA-2013:1449-01 kernel 2013-10-22
Fedora FEDORA-2013-17942 kernel 2013-10-03
Fedora FEDORA-2013-17865 kernel 2013-10-01

Comments (none posted)

monkeyd: multiple vulnerabilities

Package(s):monkeyd CVE #(s):CVE-2013-2163 CVE-2013-3724 CVE-2013-3843
Created:September 26, 2013 Updated:October 2, 2013

From the CVE entry:

The mk_request_header_process function in mk_request.c in Monkey 1.1.1 allows remote attackers to cause a denial of service (thread crash and service outage) via a '\0' character in an HTTP request. (CVE-2013-3724)

From the Gentoo bug report:

The vulnerability is caused due to a signedness error in the "mk_request_header_process()" function (src/mk_request.c) when parsing the request and can be exploited to cause a stack-based buffer overflow. (CVE-2013-3843)

From the monkeyd bug report:

The ranges parser did not validate properly the maximum offset allowed, so if a requester set limit offset equal to file size it continue processing, internally the sendfile(2) did not failed returning always zero, this condition was not handled and for hence that connections keeps running without ending, it could lead to a DoS. (CVE-2013-2163)

Gentoo 201309-17 monkeyd 2013-09-25

Comments (none posted)

mozilla: privilege escalation

Package(s):firefox, thunderbird, seamonkey CVE #(s):CVE-2013-1726
Created:September 30, 2013 Updated:October 2, 2013
Description: From the CVE entry:

Mozilla Updater in Mozilla Firefox before 24.0, Firefox ESR 17.x before 17.0.9, Thunderbird before 24.0, Thunderbird ESR 17.x before 17.0.9, and SeaMonkey before 2.21 does not ensure exclusive access to a MAR file, which allows local users to gain privileges by creating a Trojan horse file after MAR signature verification but before MAR use.

SUSE SUSE-SU-2013:1497-1 firefox 2013-09-27
Gentoo 201309-23 firefox 2013-09-27

Comments (none posted)

MRG Grid: denial of service

Package(s):MRG Grid CVE #(s):CVE-2013-4284
Created:October 2, 2013 Updated:October 2, 2013
Description: From the Red Hat advisory:

A denial of service flaw was found in the way cumin, a web management console for MRG, processed certain Ajax update queries. A remote attacker could use this flaw to issue a specially crafted HTTP request, causing excessive use of CPU time and memory on the system.

Red Hat RHSA-2013:1295-01 MRG Grid 2013-10-01
Red Hat RHSA-2013:1294-01 MRG Grid 2013-10-01

Comments (none posted)

nas: multiple vulnerabilities

Package(s):nas CVE #(s):CVE-2013-4256 CVE-2013-4257 CVE-2013-4258
Created:September 27, 2013 Updated:June 26, 2014

From the Fedora advisory:

CVE-2013-4258 (formatting string for syslog call)

CVE-2013-4256 (parsing display number)

CVE-2013-4257 (heap overflow when processing AUDIOHOST variable)

Gentoo 201406-22 nas 2014-06-26
Mageia MGASA-2013-0298 nas 2013-10-10
Debian DSA-2771-1 nas 2013-10-09
Ubuntu USN-1986-1 nas 2013-10-01
Fedora FEDORA-2013-16989 nas 2013-09-27
Fedora FEDORA-2013-17036 nas 2013-09-27

Comments (none posted)

openstack-keystone: incorrect token revocation

Package(s):openstack-keystone CVE #(s):CVE-2013-4294
Created:September 26, 2013 Updated:November 8, 2013

From the Red Hat advisory:

It was found that Keystone did not correctly handle revoked PKI tokens, allowing users with revoked tokens to retain access to resources they should no longer be able to access. This issue only affected systems using PKI tokens with the memcache or KVS token back ends.

Fedora FEDORA-2013-20373 openstack-keystone 2013-11-08
Ubuntu USN-2002-1 keystone 2013-10-23
Red Hat RHSA-2013:1285-01 openstack-keystone 2013-09-25

Comments (none posted)

squid: information disclosure

Package(s):squid CVE #(s):CVE-2009-0801
Created:September 27, 2013 Updated:October 2, 2013

From the CVE entry:

Squid, when transparent interception mode is enabled, uses the HTTP Host header to determine the remote endpoint, which allows remote attackers to bypass access controls for Flash, Java, Silverlight, and probably other technologies, and possibly communicate with restricted intranet sites, via a crafted web page that causes a client to send HTTP requests with a modified Host header.

Gentoo 201309-22 squid 2013-09-27
Red Hat RHSA-2016:1139-01 squid 2016-05-31
Red Hat RHSA-2016:1140-01 squid34 2016-05-31
CentOS CESA-2016:1139 squid 2016-05-31
CentOS CESA-2016:1140 squid34 2016-05-31
Scientific Linux SLSA-2016:1139-1 squid 2016-06-07
Scientific Linux SLSA-2016:1140-1 squid34 2016-06-16

Comments (none posted)

sudo: privilege escalation

Package(s):sudo CVE #(s):CVE-2013-2776
Created:October 1, 2013 Updated:October 2, 2013
Description: From the CVE entry:

sudo 1.3.5 through 1.7.10p5 and 1.8.0 through 1.8.6p6, when running on systems without /proc or the sysctl function with the tty_tickets option enabled, does not properly validate the controlling terminal device, which allows local users with sudo permissions to hijack the authorization of another terminal via vectors related to connecting to a standard input, output, and error file descriptors of another terminal. NOTE: this is one of three closely-related vulnerabilities that were originally assigned CVE-2013-1776, but they have been SPLIT because of different affected versions.

Gentoo 201401-23 sudo 2014-01-21
Scientific Linux SLSA-2013:1701-2 sudo 2013-12-09
Oracle ELSA-2013-1701 sudo 2013-11-26
Red Hat RHSA-2013:1701-02 sudo 2013-11-21
Scientific Linux SLSA-2013:1353-1 sudo 2013-10-10
Oracle ELSA-2013-1353 sudo 2013-10-02
Red Hat RHSA-2013:1353-01 sudo 2013-09-30

Comments (none posted)

tpp: code execution

Package(s):tpp CVE #(s):CVE-2013-2208
Created:September 26, 2013 Updated:February 12, 2014

From the Gentoo advisory:

TPP templates may contain a --exec clause, the contents of which are automatically executed without confirmation from the user.

A remote attacker could entice a user to open a specially crafted file using TPP, possibly resulting in execution of arbitrary code with the privileges of the user.

Fedora FEDORA-2014-1955 tpp 2014-02-11
Fedora FEDORA-2014-1935 tpp 2014-02-11
Gentoo 201309-19 tpp 2013-09-25

Comments (none posted)

txt2man: file overwrite

Package(s):txt2man CVE #(s):CVE-2013-1444
Created:October 1, 2013 Updated:October 2, 2013
Description: From the CVE entry:

A certain Debian patch for txt2man 1.5.5, as used in txt2man 1.5.5-2, 1.5.5-4, and others, allows local users to overwrite arbitrary files via a symlink attack on /tmp/2222.

Ubuntu USN-1979-1 txt2man 2013-09-30

Comments (none posted)

vino: denial of service

Package(s):vino CVE #(s):CVE-2013-5745
Created:October 1, 2013 Updated:November 7, 2013
Description: From the Ubuntu advisory:

Jonathan Claudius discovered that Vino incorrectly handled closing invalid connections. A remote attacker could use this issue to cause Vino to consume resources, resulting in a denial of service.

SUSE SUSE-SU-2013:1631-2 vino 2013-11-06
SUSE SUSE-SU-2013:1631-1 vino 2013-11-06
Fedora FEDORA-2013-17121 vino 2013-10-25
Scientific Linux SLSA-2013:1452-1 vino 2013-10-23
Oracle ELSA-2013-1452 vino 2013-10-22
Oracle ELSA-2013-1452 vino 2013-10-22
CentOS CESA-2013:1452 vino 2013-10-23
CentOS CESA-2013:1452 vino 2013-10-23
Red Hat RHSA-2013:1452-01 vino 2013-10-22
Mageia MGASA-2013-0300 vino 2013-10-10
Ubuntu USN-1980-1 vino 2013-09-30

Comments (none posted)

zabbix: man-in-the-middle attacks

Package(s):zabbix CVE #(s):CVE-2012-6086
Created:September 30, 2013 Updated:October 15, 2013
Description: From the Red Hat bugzilla:

A security flaw was found in the way Zabbix, an open-source monitoring solution for IT infrastructure, used (lib)cURL's CURLOPT_SSL_VERIFYHOST variable, when doing certificate validation (value of '1' meaning only check for the existence of a common name was used instead of value '2' - which also checks if the particular common name matches the requested hostname of the server). A rogue service could use this flaw to conduct man-in-the-middle (MiTM) attacks.

Fedora FEDORA-2014-5551 zabbix 2014-05-01
Fedora FEDORA-2013-18351 zabbix 2013-10-14
Fedora FEDORA-2013-18348 zabbix 2013-10-14
Fedora FEDORA-2013-14029 zabbix 2013-09-30

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 3.12-rc3, released on September 29. Linus's summary was: "On the whole, nothing really appears very scary. Go forth and test."

Stable updates: 3.11.2, 3.10.13, 3.4.63, and 3.0.97 were released on September 26; they were followed by 3.11.3, 3.10.14, 3.4.64, and 3.0.98 on October 1.

Comments (none posted)

Quotes of the week

It's also probably the first time that code entered on an ordinary cell phone has gets into the Linux kernel, so it's probably a new Linux milestone, in a twisted, sick way.

Ingo Molnar

We could put the following comment *below* the function in arch/x86/lib/misc.c:

     * Sent from my mobile phone.  Please pardon brevity and lack of formatting.
Borislav Petkov

Comments (11 posted)

Exchanging two files

By Jonathan Corbet
October 2, 2013
The renameat() system call changes the name of the file given as an argument, possibly replacing an existing file in the process. This operation is atomic; the view of the filesystem from user space will reflect the situation before or after the renameat() call, but it will never expose an intermediate state. Things work well when one file is involved, but what happens when multiple rename operations need to be run as a single atomic operation? That is a big problem, but, thanks to a patch from Miklos Szeredi, we might have a solution to a smaller subset.

The problem Miklos is trying to solve is the problem of exchanging two files — both files continue to exist, but their names have been swapped. To achieve this, he has posted a patch set adding a new renameat2() system call:

    int renameat2(int olddir, const char *oldname, 
		  int newdir, const char *newname, unsigned int flags);

This system call differs from renameat() in that it has the new flags argument; if flags is zero, renameat2() behaves exactly as renameat(). If, instead, flags contains RENAME_EXCHANGE, an existing file at newname will not be deleted; instead, it will be renamed to oldname. Thus, with this flag, renameat2() can be used to atomically exchange two files. The main use case for renameat2() is to support union filesystems, where it is often desirable to atomically replace files or directories with "whiteouts" indicating that they have been deleted. One could imagine other possibilities as well; Miklos suggests atomically replacing a directory with a symbolic link as one of them.

No review comments have been posted as of this writing.

Comments (16 posted)

Kernel development news

Transactional memory in the dentry cache

By Jonathan Corbet
October 2, 2013
LWN recently described the "lockref" mechanism merged into the 3.12 kernel. Lockrefs aim to reduce locking overhead in situations where all that needs to happen is the modification of a reference count, but the data structure as a whole must be held invariant while that count is changed. This new locking primitive has helped to reduce the overhead of locking in the system's dentry cache — the extensive data structure that caches mappings between file names and inodes. But dentry cache overhead strongly affects the performance of the system as a whole, so it is not surprising that there is a desire to improve things even more; that appears to be especially true in the 3.12 development cycle, which has seen a lot of attention paid to the reduction of core locking overhead.

One might think that it would be hard to improve on the performance of a lockref in this area, but there is one technology that might just help in this regard: transactional memory. For some years now, transactional memory has been one of those hardware features that was destined to solve a number of our scalability problems. As is often the case, that future has been slow to arrive in the real world. But some manufacturers are now shipping CPUs with some transactional memory support built into them, and software is starting to take advantage of this feature. See, for example, this article describing the work done to add transactional memory support to the GNU C library.

Linus recently got an itch to upgrade to a newer, faster desktop computer; when he chose his new processor, he made a point of getting one that provided transactional memory support. So, he decided, the time had come to try to use that support to further speed dentry cache locking. The result was a short, proof-of-concept patch that shows how things could work. The core part of the patch is worth showing directly:

    asm goto("xbegin %l[repeat]": : :"memory","ax":repeat);
    if (unlikely(d_unhashed(dentry)))
    	goto xabort;
    if (unlikely(!is_simple_dput(dentry)))
    	goto xabort;
    if (unlikely(!arch_spin_value_unlocked(dentry->d_lock.rlock.raw_lock)))
    	goto xabort;
    asm volatile("xend");

    asm volatile("xabort $0");

    /* Fallback to lockref code */

The first line adds the xbegin instruction that starts the transaction. This instruction behaves a little strangely, in that its influence extends over the code that follows. Execution will continue with the code after the xbegin, but, should the transaction abort for any reason, all changes made during the transaction will be rolled back and control will jump to the address provided to xbegin (the label repeat in this case).

What follows is a set of tests to determine whether it is truly safe to decrement the reference count without holding the dentry spinlock. In particular, the dentry must be hashed (have a name associated with it, essentially), be on the LRU list, and not be locked. Reading these fields of the dentry structure adds them to the transaction; should some other code make a change to one of them, the transaction will be aborted by the processor. If any of the tests show that the dentry is not in a suitable state for a quick reference count decrement, the code uses the xabort instruction to make the transaction fail. Otherwise, the reference count will be decremented and the xend instruction will bring the transaction to a successful conclusion. The reference count, too, will become part of the transaction; should any other processor also try to change it, the transaction will abort.

The code as written is clearly not intended for merging; one does not put late-model x86 assembly code directly into generic filesystem code, after all. But it is enough to get a sense for how well transactional memory works in this situation. According to Linus, "profiles with this look beautiful," with the locking cost associated with the dput() operation reduced essentially to zero. So there may well be a place for transactional memory within the dentry cache.

Whether this technique will see much wider use in the kernel remains to be seen, though. Andi Kleen posted a patch using transactional memory for most kernel locks back in March, but that work did not go very far, mostly because he could not provide any benchmark numbers showing what kind of performance improvements could be expected. In his patch, Linus made it clear that he suspects those improvements will be small, saying "I doubt the intel TSX patches are going to be useful (if they were, Intel would be crowing about performance numbers now that the CPU's are out, and they aren't)." Instead, he has suggested that this feature will only be useful in a few places:

Quite frankly, from all I've seen so far, the kernel is not going to have very good luck with things like lock elision, because we're really fine-grained already, and at least the Intel lock-elision (don't know about POWER8) basically requires software to do prediction on whether the transaction will succeed or not, dynamically based on aborts etc. And quite frankly, by the time you have to do things like that, you've already lost. We're better off just using our normal locks.

So as far as I'm concerned, transactional memory is going to be useful - *if* it is useful - only for specialized code. Some of that might be architecture-internal lock implementations, other things might be exactly the dput() kind of situation.

The end result is that Andi's patches may be a good starting point for transactional memory support in a more architecture-independent way, but we may not see transactional memory used as a general locking mechanism (or lock-elision mechanism) in the kernel.

Along the lines of architecture-independence, it is worth noting that Intel was not the first to ship a processor with transactional memory support; the PowerPC architecture has had similar functionality for a couple of years. Naturally, the feature works a little differently in that architecture, but the basic functionality is the same. So it should be possible to create some sort of wrappers around transactional memory that can be used in common code. That is, if kernel-space transactional memory can be made to work at all in an efficient manner on PowerPC; there are some challenges in the way of getting that functionality working reliably.

What one might conclude is that, while transactional memory is a useful technology for speeding up specific bits of high-profile kernel code, it does not look like a general solution to the locking problem at this point. That could change in the future, of course, as hardware transactional memory gets faster and more capable. But, for now, transactional memory looks mostly like a useful trick to squeeze a little more performance out of a few highly optimized code paths.

Comments (9 posted)

How much memory power management is useful?

By Jonathan Corbet
October 2, 2013
In most contemporary systems, the easiest ways to reduce power consumption have long since been taken advantage of. But any system component that uses power is worth looking at to see whether it can be made to use less. In the case of main memory, the potential power savings are said to be significant, but they come at the expense of complicating the (already complicated) memory management subsystem. Now, with the latest in a long series of memory power management patches, some developers are questioning whether all of that complexity is called for.

Srivatsa S. Bhat's memory power management patch set was last covered here in April 2013. Many of the core principles remain the same, but some of the details have changed significantly.

The patch set is still based on the notion of dividing main memory into "regions" that match the power-management boundaries of the underlying hardware. At the hardware level, a region is a range of memory that can be powered up or down as a unit. The fundamental goal of the memory power management patches is to keep as many of these regions free as possible, enabling them to be kept in a low-power state.

Regions have some similarity to the "zones" maintained by the memory management subsystem. But zone boundaries are defined by the kernel, and have little to do with the hardware architecture (though zones will not span multiple NUMA nodes). Regions are, instead, a hardware concept. Since there is little commonality between regions and zones, the patch set ends up defining regions as an independent structure, almost entirely distinct from zones.

The first step is then simple: when allocating pages, do so from the lowest-numbered region with available pages. That will tend to concentrate allocations at one end of the memory range, keeping the higher-numbered regions free. To facilitate allocation in this mode, the page allocator's free list is modified to keep free pages sorted by region; that allows pages from the appropriate regions to be found quickly, with no need to scan through the list. The allocation logic is also changed a bit. Currently, the kernel will try to avoid splitting up large contiguous areas to satisfy an allocation request if a suitable (and smaller) chunk of memory is available. The memory power management patches sacrifice that heuristic to the extent that large chunks in low-numbered zones will be split rather than allocating from a smaller chunk in a higher-numbered zone.

The next step is somewhat more invasive: a "region allocator" is added to the memory management subsystem; it manages memory in large blocks corresponding to hardware regions. In current kernels, the page allocator works directly with the system's available memory; in the new scheme, instead, the page allocator allocates memory from regions of memory obtained from the region allocator. In other words, the region allocator has been placed between the page allocator and physical memory. If the page allocator needs more memory, it will request a new region from the region allocator. If, instead, the page allocator realizes that it has a full region on the free list, that region can be returned to the region allocator.

A determined effort is made to ensure that all allocations from any given region have the same "migration type." A page's migration type describes how easily the contents of that page could be moved elsewhere. Anonymous pages for user-space use are relatively easy to move; all that is required is to copy the data and change any page table entries that point to the page. Pages used in the kernel's slab allocators, instead, are firmly nailed into place; moving such a page would require finding and changing all of the kernel-space pointers referring to objects allocated from that page — not an easily solved problem. The purpose here is straightforward enough: it only takes one non-movable page to keep an entire region powered up. If all of those unmovable pages can be kept separate from pages that are more easily relocated, it will be easier to free regions of memory.

The final part of the patch set takes advantage of movable pages to actively free zones of memory when the conditions are right. For example, if a region containing movable pages is mostly empty and free pages are available elsewhere, the kernel will attempt to relocate data out of that region and, once the region is empty, return it to the region allocator. In other cases (clean page-cache pages, for example), the pages can simply be reclaimed. In this way, it is hoped, the inevitable fragmentation that occurs over time can be cleaned up, keeping the maximum number of regions free.

Most of this 40-part patch set is relatively uncontroversial, but there are some worries about the active defragmentation measures. If they are not applied carefully, they could end up increasing power consumption rather than decreasing it: moving pages around takes power, as does replacing data in the page cache that was reclaimed prematurely. So there needs to be some clear evidence that the active measures help to reduce power consumption; thus far, that evidence does not exist, since no power savings benchmark results have been posted for some time.

More to the point, there are concerns that the active measures may reflect a bit of a mismatch between the design of the patch set and how memory power management actually happens on current hardware. The core idea is to keep as many memory regions entirely free of data as possible; that will allow those regions to be powered down — losing all data stored therein — without causing problems. In this model, a single active page can keep a region from being powered down, so regions must be completely evacuated for the power savings to be realized.

But, as Arjan van de Ven explained, memory power management, on current Intel hardware at least, works a bit differently. It has a number of reduced-power modes, most of which are entirely transparent to the operating system and preserve the data stored in memory. Transitions to and from the reduced-power states are quite fast, to the point that latency problems are not a real concern. For this type of power management to kick in, all that is required is that the relevant region of memory be idle most of the time. As Arjan put it:

To get the power savings, my deep suspicion (based on some rudimentary experiments done internally to Intel earlier this year) is that it is more than enough to have "statistical" level of "binding", to get 95%+ of the max theoretical power savings....

In other words, if a significant number of regions can be kept mostly empty, it doesn't matter that much if the last few pages are still in occasional use. That suggests that the parts of Srivatsa's patch set that control the allocation patterns are all that's needed to reap most of the power-savings rewards — on Intel hardware at least. The active clearing of regions may turn out to be a futile effort to gain the last few percent of the available savings. Of course, not all hardware is made by Intel; results for other processors and architectures may vary. Even on Intel systems, there may well be situations where it makes sense to support full, content-destroying memory power-down. But the fact that most of the benefits can apparently be obtained without actively moving pages around will argue for, at least, that feature being turned off by default.

Note the use of the word "apparently" above, though. What is lacking from the discussion at this point is any sort of comprehensive measurements of actual power use with and without these patches. Real measurements, which generally trump more theoretical discussions, should be the deciding factor when the memory management maintainers decide how much of this work to merge and whether the active defragmentation functionality should be enabled by default or not. Either way, the bulk of the patch set seems to be relatively uncontroversial, so we should see some sort of movement on memory power management, finally, in the near future.

Comments (5 posted)

NUMA scheduling progress

By Jonathan Corbet
October 1, 2013

NUMA balancing was a topic of fierce debate through much of 2012; that discussion culminated with the merging of Mel Gorman's NUMA balancing infrastructure patch set into the 3.8 kernel. Those patches provided the basic structure upon which a NUMA balancing solution could be built, but did not attempt to solve the problem in a comprehensive way. Since then, one might be forgiven for thinking that the developers involved have lost interest; not much NUMA-related code has found its way into the mainline. But, as can be seen in Mel's basic scheduler support for NUMA balancing patch set, which weighs in at 63 individual changesets, quite a bit of work has been happening in this area.

The NUMA balancing problem is one of overcoming a fundamental characteristic of NUMA hardware: accesses to memory local to any given NUMA node will be much faster than accesses to remote memory. If processes and their memory live on the same NUMA node, things will run quickly; otherwise performance will be significantly worse. NUMA performance on Linux has long been deemed to be worse than it should be, so there is clearly some room for improvement in this area.

The kernel has, for years, attempted to allocate memory on the same node that the allocating process is running on, but that turns out to be inadequate. Over time, it seems to be inevitable that processes and memory will move around the system, leading to poor performance. Recent efforts to improve the situation have thus been focused on cleaning things up as the system runs, moving memory to the nodes that are actually using it and moving processes to be closer to their memory. This particular patch set, as suggested by the "scheduler support" name, is mainly concerned with the latter task — ensuring that the scheduler runs processes on the nodes that host the bulk of their memory.

Locating a process's memory

If the scheduler is to keep processes near their memory, it must, clearly, have a sense for where that memory is placed. This information is not as straightforward to come by as one might expect, especially if one is concerned mostly with the memory that is actually being used, as opposed to memory which has been allocated but is not in active use. The kernel needs to continually observe a process's memory access patterns to be able to optimize its placement in the system.

One of the core changes in the current NUMA patch set is to enable this observation, using the virtual memory mechanism. The scheduler will periodically scan through each process's address space, revoking all access permissions to the pages that are currently resident in RAM. The next time the affected process tries to access that memory, a page fault will result. The scheduler will trap that fault and restore access to the page in question; it will also increment an access counter in a per-process array indexed by the NUMA node number. Over time, these counts (which are maintained as a sort of running average) will provide a picture of where the process's memory accesses are going. At that point, it is a simple matter of looping through the array to find the node with the most accesses; that becomes the "preferred node" for the process.

If the process is running on a node other than the preferred one, the scheduler will attempt to migrate it to the node where its memory lives. That should result in more node-local memory accesses and, thus, better performance. Of course, the picture is a bit more complicated than this for a number of reasons, starting with the fact that the migration of the process could fail for a number of reasons. So one of the patches in the series will periodically retry the attempt to migrate a process to its preferred node if things didn't work out the first time. Even if the initial migration attempt fails, the process should eventually end up on the preferred node.

This process-follow-memory migration can also run counter to another one of the scheduler's core goals: distributing the load across the system to make the best use of the available CPU time. If a process that has been migrated closer to its memory cannot actually run because the destination NUMA node is overloaded, the goal of improved performance is unlikely to be achieved. So it may be necessary to migrate other processes off the overloaded node. That complicates the picture somewhat. The patch set does not try address the full problem, but it does take on one specific subproblem: cases where swapping two processes between two CPUs will make both processes run more efficiently. Even that task is fraught with hazards: moving two processes at once is more complicated than migrating a single process. To make it work with a minimum of fuss, the patch set adds a variant of stop_machine() that effectively halts work on the two processors involved while the exchange is made.

It also became necessary to avoid tracking NUMA access information for shared executable pages. Including those pages will tend to pull processes toward the nodes where pages holding shared library code can be found. But there is relatively little value in migrating processes near their executable pages, as it turns out, because much of the needed data is in the CPU caches much of the time anyway. So patch 35 in the series avoids tracking faults from those pages.

NUMA groups

As described thus far, the patch set adds a basic mechanism for putting a process near its (exclusively owned) pages. But the concept of "a process's memory" is not necessarily simple either; processes often share pages with each other. That is especially true of threaded applications where all of the threads run in the same address space and have access to the same memory. Threaded applications written with NUMA performance in mind will partition their memory accesses, but they can still end up sharing pages, especially if a feature like transparent huge pages is in use. In other cases, entirely separate processes may still share a large memory area; neither "owns" it, but both can benefit by being placed close to it. Either way, there can be performance benefits from placing such processes on the same node whenever possible.

To improve performance in the shared memory case, the patch set adds a mechanism that tries to detect when processes are sharing memory with each other — and, importantly, when they are accessing the same pages. Doing so requires tracking which processes are accessing each page. Andrea Arcangeli's AutoNUMA patch set, one of the contenders in the 2012 discussions, added an extensive tracking infrastructure for this purpose, but the resulting memory overhead was deemed to be too high. So, as is so often the case, this patch set tries to track this data by shoehorning the data into struct page instead.

In particular, the data is put into the flags field, which is already rather more complex than a simple set of flags. The ID of the NUMA node containing each page is stored there in current kernels; this patch set changes that to the CPU ID, and adds the process ID of the last process to access the page as well. Needless to say, the full process ID does not fit into a subfield of flags, so some compromises need to be made. In this case, only the bottom eight bits of the process ID are used, with the understanding that some collisions will be unavoidable. Whenever the system handles a page fault, this "cpupid" number is stored in the relevant page structure.

With that infrastructure in place, the scheduler can respond to a NUMA scanning fault by checking whether the process currently accessing the page is the same as the last process to do so. If not, the scheduler will consider putting the two processes into a "NUMA group." But first, it must find the other process that accessed the page — not a trivial matter, given that the full process ID for that process is not available. This problem is handled by looking at whichever process is currently running on the CPU that last accessed the page; if the relevant part of the process ID matches, the scheduler assumes (after applying a few sanity tests) that it is the same process as the one that accessed the page. In this case, the processes are placed into a NUMA group to indicate that they are sharing pages; if both processes are already in groups, those two groups will be coalesced into one larger group.

Once again, shared library pages threw a wrench into the works. Since every process shares access to, for example, the C library, all processes in the system tended to get pulled together into a single NUMA group. Deeming that to be less than fully useful, Peter Zijlstra tossed in a patch avoiding group tracking for read-only pages.

Once a process is part of a NUMA group, page faults for shared pages will be tracked at the group level. That gives the scheduler two somewhat independent sets of statistics from which to make decisions: accesses to private pages, and accesses to shared pages. If both sets of numbers agree on the preferred node, the decision is easy. If, instead, there is a disagreement, the preferred node will be chosen by iterating through the arrays and picking the node where the sum of private and shared faults is the highest. That sum is distorted, though, by weighting shared faults a bit more heavily in an attempt to push the system toward decisions that put processes with shared pages on the same node.

There is a lot more than this to the patch set, including a number of algorithmic tweaks and fixes, but the above text describes the most significant themes. Mel has included quite a bit of benchmark data with the patch set; there is a lot of information there, but the bottom line, as he described it, was: "some reports indicate that the performance is getting close to manual bindings for some workloads but your mileage will vary." A number of details are yet to be addressed; in particular, support for CPU hotplugging is not yet there. Even so, Mel thinks that the code is getting ready for movement into the "tip" tree, and, thus, linux-next. That suggests it could be ready for merging in the 3.13 or (perhaps more likely) 3.14 development cycle.

Comments (13 posted)

Patches and updates

Kernel trees


Core kernel code

Development tools

Device drivers

Filesystems and block I/O

Memory management




Page editor: Jonathan Corbet


Fedora and bug tracking

By Nathan Willis
October 2, 2013

Bug reports are a blessing and a curse: not enough and the software project misses out on vital feedback; too many and the project is overwhelmed or loses the signal to the noise. Recently, a peculiar discussion in the Fedora community highlighted the difficulty of wrangling bug reports for distributions, which are often the final "downstream" project in the free software deployment hierarchy.

Jóhann Guðmundsson proposed that Fedora move away from the Red Hat Bugzilla (on which Fedora has relied since the project's inception). But he did not propose setting up a standalone Fedora bug tracker; rather, he suggested adopting "pure upstream reporting," in which the Fedora community would file bugs directly with the upstream projects themselves.

He cited three reasons. First, many package maintainers feel that their bug reports go ignored by the upstream project if they are reported downstream. Second, the Fedora package maintainer frequently is not a developer, so he or she cannot fix the bug, which must eventually find its way upstream in order to get the proper attention (at the cost of increased latency). Third, neither of the aforementioned problems would solved by Fedora running its own issue tracker.

Guðmundsson's proposal was met with near-universal disagreement; Fedora contributors do not need to worry that their bug tracker will be going away any time soon. But the debate in the list thread is an interesting read for what it reveals about the hassle of cultivating and managing bug reports.

The primary objections to Guðmundsson's proposal were of a distribution-development nature, pointing out (for example) that there are always a significant number of bugs in Fedora which do not belong upstream. User nonamedotc pointed out that there are always bugs caused by Fedora-specific combinations of libraries or the particulars of the distribution's builds. Jan Wildeboer said that Fedora itself develops its own components (starting with the installer) which have no upstream project, and that the distribution needs a place to track blockers and critical bugs that are part of preparing each Fedora release. Tracking release status mandates having one centralized place to view all of the outstanding bugs in the distribution.


On those technical grounds alone, there would seem to be no chance that Fedora would do away with its bug tracker. But Guðmundsson replied that Fedora already has an incomplete picture of all of the outstanding bugs because there are always bugs that are only reported upstream. Indeed, one of the points of agreement was that having separate bug trackers for the distribution and the upstream project is an ongoing challenge that needs addressing. Fedora tries to maintain good relationships with its upstream projects; staying in sync as much as possible while not inundating them with Fedora-specific bugs. Wildeboer argued that sending Fedora bug reporters directly to upstream projects would have a damaging effect on Fedora's relationship with those other projects:

Sending users to upstream doesn't work. It will upset upstream and force users to jump through even more hoops. Maintaining a working relationship with upstream is a very responsible task that should not be decentralized.

Along the same lines, Michael Schwendt said that there are upstream developers who make use of the reports in Fedora's Bugzilla, both for the big picture" and to examine specific details like stack traces. If, as Guðmundsson had suggested, there is a breakdown in communication with the upstream project, then that is a problem that needs fixing by package maintainers, triagers, and testers.

Dan Horák suggested that Fedora needs a tool that can clone a bug report from Fedora's Bugzilla to an upstream repository. Right now, developers and package maintainers can copy a bug manually over to the upstream project's issue tracker, but it is a time-consuming ordeal. That suggestion met with some approval, although it was quickly pointed out that auto-populating an upstream project's bug tracker would certainly not be a good idea. The goal should be to reduce the amount of work for the packager, not to automate the cloning. As Jakub Jelinek pointed out, an automated clone of a bug can propagate excess information—in particular, he criticized the bug reports pushed upstream by Debian, which frequently include mail headers and list discussions from that distribution's email-based issue tracking system.

The other difficulty with establishing links from Fedora bugs to upstream bug trackers is, as Bill Nottingham observed, that not every upstream project has a bug tracker—or, at least, a bug tracker that is accessible to those outside the project. Furthermore, if the bug tracker exists, it may not be easy to find. Mateusz Marzantowicz asked Guðmundsson how a bug reporter would even know the canonical location of a bug tracker for any particular application; without a centralized list or database, the location of the bug tracker would need to be stored somewhere—perhaps inside of the RPM package itself. But this is a problem that faces package maintainers already, even without Guðmundsson's upstream-first idea. Some projects have no bug tracker, new package maintainers may have to familiarize themselves with a new system for each package, and so on.

Similarly, Felix Miata noted that, in the absence of a Fedora-specific bug tracker, users would often not know where to report a bug that could have multiple causes:

What to do when someone discovers what is clearly a problem but neither he nor anyone reading his report here or on devel list can tell whether the bug is in kernel, driver, xorg, gnome/kde/xfce/etc. or something else?

Yet this, too, is already a problem that confronts bug reporters, as Adam Williamson of Fedora's QA team said. It may be unclear where the source of a particular bug lies (if it is possible to determine the source from one user's system), but bugs in Fedora itself will always be one of those possibilities. Ultimately, Nottingham concluded that the root problem is one of personnel:

Honestly, I think a good dedicated triage team that works to verify and move upstream as appropriate works better. But, you know, requires getting and keeping such a team.

The personnel issue is also at the heart of another major criticism of Guðmundsson's proposal: if bugs must be reported upstream, then bug reporters must seek out the upstream project's bug tracker and create an account for every package that they report issues against. That could easily be dozens or hundreds of accounts for active participants, and the more active a bug reporter is, the worse the problem would be. Several list participants raised this objection, with Michal Jaegermann going so far as to say that he would stop reporting bugs altogether under such circumstances.


No doubt Fedora (like any free software project) would love to have excess people-power to throw at the challenges of bug reporting and triage. But the shortage that Fedora currently feels is not likely to resolve itself, so the QA and testing community will probably continue to find a significant amount of its time occupied by managing bug reports. The last factor in the equation, perhaps, is determining which people ought to be responsible for wrangling bug reports: in particular, deciding when pushing them upstream is the correct solution.

Guðmundsson argued in his proposal that Fedora package maintainers are usually not developers and are often not even familiar with the codebase of the package itself—that was one reason he advocated reporting bugs directly to the upstream project. But several on the list disagreed with that argument. Schwendt pointed to the guidelines for package maintainers laid out in the Fedora wiki, which state (but do not mandate) that maintainers should create an account on upstream bug trackers and forward bugs upstream.

Ralf Corsepius said that pushing bug reports upstream was a maintainer's duty, as was implementing patches in some situations. Of course, the trouble is that package maintainers come and go; when important packages are orphaned, it is not uncommon for a new volunteer to take up the mantle of maintainership even if he or she is less-than-fully familiar with the package in question. No doubt it is the new maintainer's responsibility to get up to speed, but at any given time there are liable to be numerous packages whose maintainers are not prepared to send in a patch. Schwendt, however, argued that not every package maintainer needs to be a developer, too:

It is a perfectly fine scenario for somebody, let's say "a power-user", with interest in a Fedora package to join as a co-maintainer and focus on problem reports and testing of the "product" we release, even if there may be no interest in 'commit' access to the Fedora package.

The difficulty, of course, is that package maintainership sits in the gray zone that lies somewhere between "user" and "developer." The free software community often talks about itself as if these two camps are the only categories, which is an oversimplification. Guðmundsson's frustration with how bug reports are currently handled in Fedora illustrates how that oversimplification plays out: there are package maintainers who cannot process and route bug reports quickly enough to keep the bug queue clear and the end users happy.

But Guðmundsson's upstream-only proposal would essentially have addressed that problem by relieving package maintainers of the responsibility for routing bug reports altogether. That might have come as a relief to some overworked packagers, but the work would not simply disappear: it would end up being pushed to some other segment of the community, either to the Fedora QA team, to the upstream project, or all the way to the end user.

For that and the other issues pointed out, Fedora is clearly not going to discontinue the use of its bug tracker. That means, however, that managing bug reports will remain a tricky process for some time to come. Perhaps the changes to Bugzilla proposed by Horák will ameliorate the headaches to some degree, but in the long run it is likely to remain (as so many software-development hurdles are) a human problem.

Comments (3 posted)

Brief items

Distribution quotes of the week

After all, bugs often arise when you're not looking for them, as opposed to when you are, and some bugs, when looked for, vanish
-- Kent Fredric

Smart Scopes also seem to have become a little smarter. Search results were a bit more relevant than when I have tested this feature in the past, though there is still plenty of junk. Why Britney Spears albums come up on a search for "Thailand" is something only Canonical knows. Maybe.
-- The Register reviews Ubuntu 13.10 beta

Comments (none posted)

Debian Edu / Skolelinux Wheezy

Debian Edu aka Skolelinux has released Debian Edu 7.1 "Wheezy". "Debian Edu is a complete operating system for schools. Through its various installation profiles you can install servers, workstations and laptops which will work together on the school network. With Debian Edu, the teachers themselves or their technical support can roll out a complete multi-user multi-machine study environment within hours or a few days. Debian Edu comes with hundreds of applications pre-installed, but you can always add more packages from Debian."

Full Story (comments: none)

New GNU Hurd, Mach, and MIG releases

The GNU project is celebrating its 30th anniversary with the releases of GNU Mach 1.4 ("This new release bundles bug fixes and enhancements done since the release of version 1.3, eleven years ago; really too many (both years and improvements) to list them individually"), GNU MIG 1.4 (MIG being the Mach interface generator), and version 0.5 of the GNU Hurd kernel ("This new release bundles bug fixes and enhancements done since the release of version 0.2, 16 years ago"). The Hurd is still 32-bit on x86 only, but a 64-bit port is said to be in the works.

Comments (22 posted)

FreeBSD 9.2 released

The FreeBSD Release Engineering Team has announced the availability of FreeBSD 9.2. This release features some ZFS filesystem enhancements along with various updated packages. The release notes contain the details.

Comments (1 posted)

NetBSD 6.1.2 and NetBSD 6.0.3 released

The NetBSD Project has announced NetBSD 6.1.2 and NetBSD 6.0.3. Both releases contain fixes deemed important for security or stability reasons. More information can be found in the release notes.

Comments (1 posted)

Red Hat Enterprise Linux 5.10 released

Red Hat has announced the release of Red Hat Enterprise Linux 5.10. See the release notes for more information.

Comments (none posted)

Ubuntu 13.10 (Saucy Salamander) Final Beta released

A final beta is available for Ubuntu 13.10 Desktop, Server, Cloud, and Core products, along with Edubuntu, Kubuntu, Lubuntu, Ubuntu GNOME, UbuntuKylin, Ubuntu Studio and Xubuntu flavors. This is also the first beta release of Ubuntu for Phones. "The beta images are known to be reasonably free of showstopper CD build or installer bugs, while representing a very recent snapshot of 13.10 that should be representative of the features intended to ship with the final release expected on October 17th, 2013."

Full Story (comments: none)

Distribution News

Red Hat Enterprise Linux

Red Hat Enterprise Linux 5.3 Advanced Mission Critical 6-month Notice

Red Hat Enterprise Linux 5.3 Advanced Mission Critical (AMC) will be retired in six months. "In accordance with the Red Hat Enterprise Linux Errata Support Policy, Advanced Mission Critical for Red Hat Enterprise Linux 5.3 will be retired as of March 31, 2014, and support will no longer be provided. Accordingly, Red Hat will no longer provide updated packages, including critical impact security patches or urgent priority bug fixes, for Red Hat Enterprise Linux 5.3 AMC after that date. In addition, technical support through Red Hat's Global Support Services will no longer be provided after March 31, 2014."

Full Story (comments: none)

Red Hat Enterprise MRG for Red Hat Enterprise Linux 5 6-month Notice

Red Hat Enterprise MRG Version 1 and Version 2 for Red Hat Enterprise Linux 5 will be retired in six months. "In accordance with the Red Hat Enterprise MRG Life Cycle policy, the Red Hat Enterprise MRG products, which include the MRG-Messaging, MRG-Realtime, and MRG-Grid, Version 1 and Version 2 offerings for Red Hat Enterprise Linux 5 will be retired as of March 31, 2014, and support will no longer be provided."

Full Story (comments: none)

Ubuntu family

No Mir by default in Ubuntu 13.10

Developers at Canonical have concluded that the Mir desktop server (or, more specifically, the XMir layer) will not be ready in time to be shipped as the default configuration in the 13.10 release — though they do still plan to go with Mir for Ubuntu Touch. "More specifically, the multi-monitor support in XMir is working, but not to the extent we'd like to see it for all of our users. The core of Mir is working reliable, but with XMir being a key component for our 13.10 goals, we didn't want to compromise overall Ubuntu quality by shipping it."

Full Story (comments: 53)

Newsletters and articles of interest

Distribution newsletters

Comments (none posted)

Patrick d'Emmabuntüs: Emmabuntüs is more than a Linux distribution (DarkDuck)

DarkDuck talks with Patrick d'Emmabuntüs, creator of the Emmabuntüs distribution. "This distribution was designed to facilitate the refurbishing of computers given to humanitarian organizations, especially Emmaüs communities, where the name comes from, and to promote the discovery of Linux by beginners, but also to extend the life of the equipment and to reduce waste caused by over-consumption of raw materials, as I have mentioned earlier."

Comments (none posted)

Page editor: Rebecca Sobol


Application metadata with AppData

By Nathan Willis
October 2, 2013

GNOME is deploying GNOME Software, an "app store"–style software manager, with the GNOME 3.10 release. The goal is to facilitate finding an application to install by scouring a catalog of human-readable descriptions and illustrative screenshots—in contrast to the raw details exposed by a traditional Linux package manager, which can get in the user's way. One challenge that might not be immediately obvious is collecting and collating the descriptions for all of the applications in a standardized fashion. To that end, there is the newly-drafted AppData specification.

The AppData specification is the brainchild of GNOME Software developer Richard Hughes. GNOME Software is meant to complement a regular package manager, not to replace it. The idea is that GNOME Software will help users find an application when they know what they want to do, but they do not know what applications are available. So they may browse by category, look at popular applications, or search, but in all cases they rely on a well-written description of the application to guide them. For example, users might know they want a tool for scanning photographs, but might not know—or care—that such a tool involves SANE. These well-written descriptions are not easy to come by, however, nor are they readily available to import in bulk into GNOME Software.

In August of 2013, Hughes discussed several potential sources for the human-readable application descriptions, such as the .desktop files or Description of a Project (DOAP) files released by the application project, descriptions found in distribution packages, and "About" text from within the application itself. Each had drawbacks.

For instance, DOAP files are by definition one-per-project; thus LibreOffice has a single DOAP file, but GNOME Software would want separate descriptions for the office suite's Calc and Impress components—either one of which might be the application that the user is searching for. Package file descriptions and "About" text are inconsistent and frequently too terse to be helpful to an unfamiliar user in search of an application to install. Furthermore, most of the existing sources of human-readable descriptions fail to make room for multiple translations of the file to exist simultaneously, so it was almost a given that some form of new file would need to be created.

Hughes eventually drafted the AppData specification, with the intention that it would be usable for projects in addition to GNOME Software itself. Although the specification is hosted on the site, it is not currently a formal specification (to the extent that is ever "formal"). The file format is XML, and upstream projects can provide their own AppData files to be installed in /usr/share/appdata/.

The format includes an <id> element that should specify the name used in the application's .desktop file, a <licence> element that should specify the license on the AppData text itself (not the license of the application, which is found in the software package), a <description> element for the main application description text, plus <url> for the project's homepage, <updatecontact> for a contact email address, and <project_group> to associate an application with an umbrella project. The file can also link to image URLs with one or more <screenshot> elements. Subsequently, Hughes was asked by Debian's legal team to add a field for a copyright statement, which is necessary to comply with some software licenses.

The hard part: people

The format is simple enough, but the mere existence of a simple file format does not write project descriptions. For that, Hughes turned to the public to solicit contributions. The specification page includes a list of suggestions for writing a helpful description, such as ensuring that the description says what the application actually does, not what it is planned to do. Authors are encouraged to avoid lists where possible, avoid references to trademarked competitors by name, and told: "don't mention what language an application is written in, users don't care."

On August 29, Hughes started a campaign on his blog to get application developers, package maintainers, or knowledgeable volunteers to write descriptions and create AppData files. Over the course of the next month, that campaign evolved into a formal goal that all GNOME applications would provide an AppData file by the release of GNOME 3.14. An even bigger target soon followed: that all major end-user applications shipped in Fedora 20 would include AppData.

Gauging the progress of the campaign can be a little tricky, as the GNOME and Fedora goals overlap, but are (of course) tracked separately. In a series of posts on his blog, Hughes highlighted high-profile applications that were missing AppData or had incomplete AppData files (where "complete" means having both the human-readable description and at least one screenshot). As of September 27, there were still three dozen or so important applications with incomplete entries. There is a Google spreadsheet that tracks the current Fedora application list (which as of press time includes 17 entries, many of which were not listed in the blog post).

The list of incomplete or missing AppData files does appear to be shrinking, however. But the files do need some further care even when they are complete. There is a validation tool to check the syntax of the files and report formatting problems. GNOME also maintains a list a list of the 3.10 applications that have AppData files but have not passed validation.

Use your words

Aside from validation, one of the biggest challenges will be getting the application descriptions—and in many cases, the screenshots—translated into as many languages as possible. One of the benefits of the AppData specification is that (unlike DOAP), it allows multiple translations of the metadata file. Both ITS Tool and intltool are supported paths to translation; projects can ship their AppData as an file for use with intltool or use the official appdata.its template for ITS Tool. GNOME maintains a to-do list on the AppData goal page of the GNOME wiki, which indicates not only which applications still lack AppData files, but which are still missing translations.

It may seem like a lot of work has gone into AppData for what is essentially metadata that a lot of experienced users already know. But that is part of the problem: someone who has been using desktop Linux for years likely knows what the evolution and aisleriot packages install. Someone who has just discovered them through Yum, Synaptic, or another package manager would have a hard time guessing.

GNOME and Fedora seem to be on track for adopting AppData as the preferred mechanism for delivering this sort of metadata, but Hughes is interested in making it a standard that upstream projects use, rather than an add-on supplied by packagers. It will be interesting to see how that process goes. There is certainly a need; the same problem is faced by other popular desktop distributions, but none of them have attempted to formalize the process. Ubuntu, for example, uses the Debian package description format implemented by the Debian Description Translation Project. These descriptions do not come with a formal file format (they exist primarily in the distributions' databases), although they are quite similar to what is asked for by the GNOME AppData effort.

Neither Debian nor Ubuntu pushes upstream projects to supply descriptions themselves (although the comments on Hughes's blog suggest that a number of AppData entries made use of the existing Debian/Ubuntu package descriptions). But both do offer web-based tools to facilitate making translations. Perhaps some cross-project cooperation is plausible; users would no doubt be happy as a result, with the prospect of a major reduction in the amount of head-scratching required to decipher the peculiar packages names that are so popular in free software.

Comments (1 posted)

Brief items

Quotes of the week

Lisp is like a honeypot for smart people.
— Ward Cunningham, as quoted by Jonan Scheffler

Now does seem to be about the right time to scream "you will take the middle-click paste from my cold dead fingers".

Later, after the initial beta release, would be the time to say "oh, that horrid thing you've done that I never use and keep accidentally getting when I try to paste, and makes me want to throw my mouse out the window? Let's make it green instead of blue".

David Woodhouse

Comments (3 posted)

texinfo-5.2 available

Version 5.2 of texinfo, the GNU documentation format, is available. Highlights include "more inline-style conditionals; makeinfo now warns about node/menu/cross-reference names problematic in Info, and info has a --all option somewhat analogous to man."

Full Story (comments: none)

WebKitGTK+ 2.2.0 released

Version 2.2.0 of WebKitGTK+ has been released. This release includes initial Wayland support, video accelerated compositing, custom JavaScript injection, and a new Web Inspector, in addition to several updated APIs.

Full Story (comments: none)

VLC media player 2.1.0 released

Version 2.1.0 ("Rincewind") of the VLC media player is out. "With a new audio core, hardware decoding and encoding, port to mobile platforms, preparation for Ultra-HD video and a special care to support more formats, it is a major upgrade for VLC. Rincewind fixes around a thousand bugs, in more than 7000 commits from 140 volunteers."

Full Story (comments: 9)

Rust 0.8 released

Version 0.8 of the Rust language has been announced. "This was another very active release cycle that continued the trend toward refining the standard library while making minor adjustments to the language. In this release the `for` keyword has been changed to work with `Iterator` types, the runtime and task scheduler was rewritten, a new experimental I/O subsystem was added, and we added a new family of string formatting macros, `format!`, that will eventually replace `fmt!`."

Comments (12 posted)

GNU APL 1.0 available

Version 1.0 of the GNU APL interpreter has been released, after several years of development. GNU APL is designed to be a complete implementation of APL2 as defined by ISO standard 13751.

Full Story (comments: 8)

GNU Guix 0.4 released

Release 0.4 of the GNU Guix package manager is available. According to the release announcement, "this release comes with a QEMU virtual machine image that demonstrates preliminary work toward building a stand-alone GNU system with Guix. The image uses the GNU Linux-Libre kernel and the GNU dmd init system. It is console-only, and may be used primarily to try out Guix."

Full Story (comments: 2)

Apache CloudStack 4.2 Released

Version 4.2 of Apache CloudStack has been released. This is a feature release from the 4.x series first released in November 2012. New features include "integrated support of the Cisco UCS compute chassis, SolidFire storage arrays, and the S3 storage protocol."

Full Story (comments: none)

systemd 208 released

Version 208 of the systemd init-replacement has been released. The major addition in this update is "support for facilitating privileged input and drm device access for unprivileged clients" in logind, which will allow Wayland display servers to run under the user's UID, thus permitting "secure session switching without allowing background sessions to eavesdrop on input and display data."

Full Story (comments: 2)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Pasting images with automatic attribution

Peter Liljenberg has developed an add-on for Firefox that copies linked metadata to the clipboard in addition to the "copied" object itself. The initial demonstration of this technique required a specially-crafted page with RDFs metadata linked in, and thus may not seem immediately useful. However, Liljenberg has now implemented a more straightforward use case: copying and pasting an image with attribution data automatically preserved. "The friction for using a shared image is reduced when you don’t have to remember to also write an attribution. The attribution can embed the metadata, so that if someone copies the image from your page, they can also get an attribution created automatically when they paste it into their page."

Comments (5 posted)

Rempt: Ten years of working on Krita

On his blog, Boudewijn Rempt has an interesting walk down memory lane about the history of the Krita digital painting program. It started its life in 1998 as a Qt wrapper around GIMP, called "kimp", though the first real Krita code came from a KOffice application called KImage, which changed to KImageShop, Krayon, and, finally, in 2002, Krita (Swedish for crayon). His account has controversies, flame wars, development setbacks, and more, resulting in the high-quality application that we have today. "I didn't know C++ back then, but neither was I a novice programmer. I'd been earning the daily bread for me and my family for about ten years, first as an Oracle PL/SQL developer, then Visual Basic, then Java. I had written and gotten published a book on Python and Qt, so I knew Qt as well. I had no experience with graphics, though... In October 2003 it was not possible to paint with Krita: all tools except for the layer move tool had been disabled. The paint tool was the first thing I worked on, and I was very proud when I had a tool that could place squares on the canvas -- and the size of the squares was sensitive to the tablet pressure!"

Comments (8 posted)

Page editor: Nathan Willis


Brief items

30 years of GNU

Richard Stallman launched the GNU project on September 27, 1983 — thirty years ago. "GNU will be able to run Unix programs, but will not be identical to Unix. We will make all improvements that are convenient, based on our experience with other operating systems. In particular, we plan to have longer filenames, file version numbers, a crashproof file system, filename completion perhaps, terminal-independent display support, and eventually a Lisp-based window system through which several Lisp programs and ordinary Unix programs can share a screen. Both C and Lisp will be available as system programming languages. We will have network software based on MIT's chaosnet protocol, far superior to UUCP. We may also have something compatible with UUCP." Some of the details may not have come out as envisioned, but the big idea has held up well.

Comments (126 posted)

Articles of interest

Free Software Supporter - Issue 66

The September edition of the Free Software Foundation's newsletter covers a job opening, a new DRM FAQ, GNU's 30th anniversary, and several other topics.

Full Story (comments: none)

New Books

Practical Programming, 2nd Edition--New from Pragmatic Bookshelf

Pragmatic Bookshelf has released "Practical Programming, 2nd Edition" by Paul Gries, Jennifer Campbell, and Jason Montojo.

Full Story (comments: none)

Calls for Presentations

CFP Deadlines: October 3, 2013 to December 2, 2013

The following listing of CFP deadlines is taken from the CFP Calendar.

DeadlineEvent Dates EventLocation
October 4 November 15
November 17
openSUSE Summit 2013 Lake Buena Vista, FL, USA
November 1 January 6 Sysadmin Miniconf at 2014 Perth, Australia
November 4 December 10
December 11
2013 Workshop on Spacecraft Flight Software Pasadena, USA
November 15 March 18
March 20
FLOSS UK 'DEVOPS' Brighton, England, UK
November 22 March 22
March 23
LibrePlanet 2014 Cambridge, MA, USA
November 24 December 13
December 15
SciPy India 2013 Bombay, India
December 1 February 7
February 9 Brno, Czech Republic
December 1 March 6
March 7
Erlang SF Factory Bay Area 2014 San Francisco, CA, USA

If the CFP deadline for your event does not appear here, please tell us about it.

Upcoming Events

Events: October 3, 2013 to December 2, 2013

The following event listing is taken from the Calendar.

October 3
October 4
PyConZA 2013 Cape Town, South Africa
October 3
October 5
Open World Forum 2013 Paris, France
October 4
October 5
Open Source Developers Conference France Paris, France
October 7
October 9
Qt Developer Days Berlin, Germany
October 12
October 13
PyCon Ireland Dublin, Ireland
October 12
October 14
GNOME Montreal Summit Montreal, Canada
October 14
October 19
PyCon.DE 2013 Cologne, Germany
October 17
October 20
PyCon PL Szczyrk, Poland
October 19 Hong Kong Open Source Conference 2013 Hong Kong, China
October 19 Central PA Open Source Conference Lancaster, PA, USA
October 20 Enlightenment Developer Day 2013 Edinburgh, Scotland, UK
October 21
October 23
KVM Forum Edinburgh, UK
October 21
October 23
LinuxCon Europe 2013 Edinburgh, UK
October 21
October 23
Open Source Developers Conference Auckland, New Zealand
October 22
October 24 2013 Luxembourg, Luxembourg
October 22
October 23
GStreamer Conference Edinburgh, UK
October 23 TracingSummit2013 Edinburgh, UK
October 23
October 24
Open Source Monitoring Conference Nuremberg, Germany
October 23
October 25
Linux Kernel Summit 2013 Edinburgh, UK
October 24
October 25
Xen Project Developer Summit Edinburgh, UK
October 24
October 25
Automotive Linux Summit Fall 2013 Edinburgh, UK
October 24
October 25
Embedded LInux Conference Europe Edinburgh, UK
October 25
October 27
vBSDcon 2013 Herndon, Virginia, USA
October 25
October 27
Blender Conference 2013 Amsterdam, Netherlands
October 26
October 27
PostgreSQL Conference China 2013 Hangzhou, China
October 26
October 27
T-DOSE Conference 2013 Eindhoven, Netherlands
October 28
November 1
Linaro Connect USA 2013 Santa Clara, CA, USA
October 28
October 31
15th Real Time Linux Workshop Lugano, Switzerland
October 29
November 1
PostgreSQL Conference Europe 2013 Dublin, Ireland
November 3
November 8
27th Large Installation System Administration Conference Washington DC, USA
November 5
November 8
OpenStack Summit Hong Kong, Hong Kong
November 6
November 7
2013 LLVM Developers' Meeting San Francisco, CA, USA
November 8 PGConf.DE 2013 Oberhausen, Germany
November 8 CentOS Dojo and Community Day Madrid, Spain
November 8
November 10
FSCONS 2013 Göteborg, Sweden
November 9
November 11
Mini DebConf Taiwan 2013 Taipei, Taiwan
November 9
November 10
OpenRheinRuhr Oberhausen, Germany
November 13
November 14
Korea Linux Forum Seoul, South Korea
November 14
November 17
Mini-DebConf UK Cambridge, UK
November 15
November 16
Linux Informationstage Oldenburg Oldenburg, Germany
November 15
November 17
openSUSE Summit 2013 Lake Buena Vista, FL, USA
November 17
November 21
Supercomputing Denver, CO, USA
November 18
November 21
2013 Linux Symposium Ottawa, Canada
November 22
November 24
Python Conference Spain 2013 Madrid, Spain
November 25 Firebird Tour: Prague Prague, Czech Republic
November 28 Puppet Camp Munich, Germany
November 30
December 1
OpenPhoenux Hardware and Software Workshop Munich, Germany

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds