The history of Mesa

The Mesa project (aka "Mesa 3D") is "almost old enough to drink" (21 in the US), said Ian Romanick when introducing Brian Paul's X.Org Developers Conference (XDC) talk. Paul is "the father of Mesa", Romanick said; he was presenting a look back at the 20 (or so) years of development on the open source OpenGL graphics language implementation.

Paul noted that there is some question about when development on Mesa actually began. He has been doing some archeology to try to figure it out, but he doesn't think the August 1993 date noted on the Mesa web site is correct. He thinks he may have gotten started in 1992 and hopes to eventually find the Amiga floppies (with timestamps) to determine that.

He was following the comp.graphics newsgroup in 1992 when Silicon Graphics (SGI) posted an announcement of the OpenGL language. At the time, he was working at the University of Wisconsin on a visualization package using SGI's IRIS GL, which was "terrible". There were other contenders for a replacement (e.g. PHIGS), but SGI was pretty dominant in the graphics world. After the announcement, he read all the documentation "in a couple of days" and "jumped pretty quickly" into OpenGL. It was "beautiful and slick", he said, which is an interesting comparison to OpenGL today, which has 1000 functions and "tons of ways to do the same thing".

Prehistory

Before all of that, though, he got started with computers (and graphics) as a freshman in high school on the TRS-80 Model III. It had 128x48 monochrome graphics (with non-square pixels) and he wrote a paint program for it in BASIC. That's what got him "hooked on graphics", he said.

His first computer was an Atari 800XL, which was a "quantum leap" as it was the first color graphics he had access to (160x192 with four colors, or 320x192 monochrome). Using that, he wrote a 3D modeling program, a 3D renderer using Painter's algorithm, and a ray tracing program. The ultimate output from the ray tracer was a ten-frame animation that took two-and-a-half days to render.

By the time he got to college, he had an Amiga 500, which had 640x400 resolution with 4K colors. Even though it had lots more RAM (1M vs. 64K on the Atari), it still lacked the memory for a real Z-buffer. That didn't prevent him from writing a scan-line rasterizer with lighting and dithering as well as another ray tracer. That was all written in Modula-2 because that compiler was cheaper than the Amiga C compiler, he said with a laugh. He did eventually switch to Matt Dillon's C compiler when that became available.

In the early 90s, he was working on visualization software on "really expensive" workstations (like $180K). The price of those workstations made them prohibitive for most, but there were other barriers too. Each vendor had its own API for graphics, so graphics programs had to have multiple backends to support each. But SGI and IRIS GL were pretty dominant, and that was most of what he was doing then.

There were lots of 2D workstations around at the time, it was 3D hardware that was lacking. The Vis5D 3D visualization package for atmospheric science that he developed needed to be able to run on 2D workstations. He found the VOGL library that was a small subset of IRIS GL. It didn't have triangle rendering or hidden surface removal, both of which he hacked in. "It worked but it was ugly", he said.

When OpenGL was announced, he immediately knew it was the direction he wanted to go. SGI targeted it to replace all of the other alternatives, and some industry players (DEC and IBM) were supportive of that, but others (Sun and HP) were not. But it took a surprisingly long time before vendors had OpenGL implementations. It was 1994 or 1995 before SGI shipped anything, he said, and it wasn't until 1996 that it was available across all SGI models.

Starting from scratch

He began work on Mesa (though it didn't yet have that name) in either 1992 or 1993. He was working on the Amiga at home and on Unix systems at work, using floppy disks to move the code back and forth. He developed the device driver interface to accommodate the two different drawing interfaces (Amiga and Xlib).

He had a "lot more spare time then", he said, so he had a mostly complete OpenGL implementation in November 1994. It was still missing some features, but he was using it at work and thought it might be useful to others. So he started talking with SGI about open-sourcing the code.

SGI was "pretty reasonable", he said, contrasting that reaction to what he thinks Microsoft would have said to a similar request. He and SGI went back and forth a bit, but eventually the company said "go for it". SGI said he shouldn't use the letters "GL" in the name and that it needed a disclaimer saying it was not a "real OpenGL". The irony is that today Mesa is a real OpenGL "and we have the certification to prove it", as an audience member noted. The name, Mesa, "just popped into my head one day", Paul said.

He thinks that SGI was receptive to the idea of releasing Mesa because it wanted to get the interface out in front of people as much as possible. Open source would "push that goal forward". Folks that didn't have OpenGL for their workstation could get Mesa instead, so it filled a hole in the ecosystem.

He got permission to release the code in early 1995 and announced Mesa 1.0 beta on comp.graphics on February 3, 1995. When he did that, he had no feel for how much interest there would be, but within days it became clear that there was lots. He started receiving dozens of emails about the project, ranging from "wow" or "thanks" to patches.

To support the project, he wrote his first page for the web, which was just getting traction at that point. He also set up a mailing list. He only had email access at work, so he had to try to deal with the mailing list before work or on weekends. As the project quickly ramped up, his boss, Bill Hibbard, was quite supportive and allowed him to use part of his workday for Mesa. The project owes a lot to Hibbard, he said.

Early days

In the early days, the project faced a number of issues. Supporting multiple different systems (IRIX, SunOS, HPUX, ...) was sometimes difficult due to compiler incompatibilities. Some of those kinds of problems are still seen today. The displays could be problematic as well. 24-bit displays were expensive, so most displays were 8 or 16-bit. That meant dealing with color maps and dithering. He remembers spending "days and weeks" on dithering patterns, which just "seems crazy" today.

Handling lots of patches and rolling out new releases without using revision control was another problem area. He's not sure why he didn't use SCCS or RCS, but he didn't have any revision control until Mesa moved to a SourceForge predecessor in the late 1990s, he said. Performance was another issue. PCs had 16MHz 386 processors. Workstations were faster, but "still not super fast". Calling Mesa "interactive 3D graphics" was sometimes an overstatement, he said.

But OpenGL was "pretty small back then", so he could still keep all of the Mesa code in his head in those days. That "was a lot of fun", but with a million lines of code today, Mesa is "pretty overwhelming".

Late 90s

In the late 1990s, 3D was really starting to take off. There was consumer 3D hardware available from companies like NVIDIA, ATI, Matrox, 3dfx, and others, with price points that were fairly reasonable. Those cards used Direct3D, Glide (for 3D effects), or some one-off vendor API, and there was little OpenGL support.

SGI did open up its GLX client/sever code so that programs could use OpenGL in X windows. The Utah GLX project, which was an XFree86 server-side driver that provided indirect rendering, also got started. Paul said he was not directly involved in Utah, but it attracted some talented developers to Mesa. That included Keith Whitwell who had done a lot of work to increase the performance of Mesa. Around that time, John Carmack of id Software donated $10,000 to Mesa, which funded more of Whitwell's work.

In addition, Precision Insight was formed as a company to work on X and other related technology in 1998. Paul met the founders of the company at the SIGGRAPH 99 conference and joined Precision Insight in September 1999. That meant that he could finally work on Mesa, Direct Rendering Infrastructure (DRI), and GLX full time. Prior to that, most of his work on Mesa had been done in his spare time.

In the 2000s, DRI became well established. Precision Insight was acquired by VA Linux, but then the dot-com bubble popped "and we all got laid off", so he helped form Tungsten Graphics with other former Precision Insight employees in 2001. There were multiple releases of Mesa (4.x–7.x) over that decade with "lots of hardware drivers". The Gallium3D project was started by Whitwell in 2008 to simplify creating new 3D drivers. The existing device driver interface was not a good match to what the graphics hardware was doing, so Gallium3D provided a different interface that would help reduce the code duplication in Mesa drivers, he said.

Today

Paul said that he never expected Mesa to be so successful, nor to last as long as it has. He would have guessed that vendors would put out their own OpenGL libraries so that there would be no need for Mesa. But people still need software rendering, using llvmpipe and other drivers. It's also important to have an open source implementation available that people can adapt and use for new projects, he said.

There have been as many as 1000 contributors to Mesa over the years, which is far beyond what he expected. The project has grown in size and complexity beyond his expectations as well. Mesa is in all of the Linux distributions and used by thousands (or millions?) of users every day. By his measure, it is a "very successful project".

Paul outlined what he saw as the keys to that success. Mesa was "the right project at the right time". He happened to have a job that caused him to follow comp.graphics, which led him to start the project. If he hadn't, though, someone else would have around the same time, he said. He also has always liked the perspective that Fred Brooks has espoused about computer scientists as "tool makers". He took that idea to heart and made a tool that others use to do their job. In the early days, that job was visualization, today it is entertainment, which is "not as noble" as it once was, "but still pretty cool".

As project lead, he let people "smarter than me" take control of areas they were interested in. Beyond that, the Mesa development community has always been "pretty civilized". There are no insults or screaming in the mailing list. He has seen other open source leaders who do that and he doesn't like it at all, he said. But perhaps the most significant reason for Mesa's success is that it was following a specification. "We are the carpenters" who are building a "house that someone else designed".

He closed by thanking the developers who contributed "amazing work" to the project over the years. Mesa is now his calling card and the project has greatly exceeded his expectations. In addition, he met some of his closest friends through the project.

At the start of the talk, Paul said it has been around five years since he last attended an XDC. He is now with VMware (which bought Tungsten Graphics along the way). He said that he was quite impressed to see all of the new faces of those involved with X today. Keith Packard likely summed up the feelings of many in the room (and beyond) when he thanked Paul for his longtime work on an important project that is available as open source so that it can be used by all.

[I would like to thank the X.Org Foundation for travel assistance to Portland for XDC.]

Comments (8 posted)

TT-RSS makes a comeback

Veterans of The Great Google Reader Shutdown scattered in a number of different directions once the search giant's RSS- and Atom-feed reading service closed its doors in July 2013. Quite a few proprietary web services stepped up in an attempt to attract former Google Reader users, but free software projects gained prominence as well. The ownCloud project added a feed reading application, for example, but Tiny Tiny RSS (TT-RSS) had a considerable head start. The project recently released its latest update, the latest from a post–Google Reader surge in development.

TT-RSS has been in development for more than eight years, but prior to the March announcement of the Google Reader shutdown, the pace of development had slowed. The project still made regular updates every few months, but there was not a steady influx of new developers and the feature set grew in small increments with each new release. The uptick in interest after the Reader announcement was dramatic; one need only look at the Ohloh graphs for contributor count and commits to see the difference. From a low of 54 commits by two committers in January, activity shot up to 564 commits by 31 committers in March. Most of those commits came from lead developer Andrew Dolgov, but it would be fair to say that the Reader deactivation reinvigorated interest in the project.

The first new major release following this flurry of activity was version 1.8 in June. That update brought updates to feed parsing (including dropping the Simplepie parsing library in favor of a native TT-RSS parser), moved several optional plugins out of the core codebase into a separate contrib repository, and reworked the user interface. The moved plugins constitute a significant number of lines of code, including several authentication modules (such as IMAP, LDAP, and RADIUS authentication) and connections to a variety of third-party web services. Those web-service plugins include the ability to share items from TT-RSS via Twitter, Identi.ca, Google+, Pinterest, ownCloud, and several other well-known sites.

Version 1.9 followed in late July. Several parser fixes were included, as were translation updates and a refresh of the icon set. Arguably the most visible changes were the addition of a plugin that serves a feed of the user's shared articles and the migration of TT-RSS from GPLv2 to GPLv3. Version 1.10 arrived on September 21, containing only minor changes and bug fixes.

There are a glut of feed-reading options available these days, and many look the same on the surface. The TT-RSS interface will feel familiar to anyone who has used Google Reader, Feedly, Digg Reader, NewsBlur, or the like. A scrollable list on the left-hand side shows all of the subscribed feeds by category (plus special categories for saved and starred items) and how many unread entries are available, while the remainder of the screen is occupied by split-pane news reading interface: one can view feeds by headline or by full text, mark or unmark them as read, and so on.

But TT-RSS still offers a handful of features that the majority of the competition does not—even though those features may not jump out of the interface. For one thing, it is one of the few feed readers that supports subscribing to username/password-protected feeds. There are not a lot of feeds out there that require authentication, but that is something of a chicken-and-egg problem: lack of reader support decreases the demand for authenticated feeds, and vice versa.

Another example of TT-RSS's feature set is its support for automatically detecting duplicate articles and removing the duplicates from the unread item interface. This, too, is not a common problem at the global scale, perhaps, but it is one that many free software fans are likely to run into. Planet software aggregates multiple blogs, but in doing so a single person's blog may get carried by multiple planets if that person is involved in multiple projects. Having reader software like TT-RSS filter out the duplicates that come from extra-active bloggers is surely a better solution than trying to persuade those bloggers to participate in fewer projects.

As for the changes in the latest releases, TT-RSS does feel snappier than it used to. That may be attributable to the new, lightweight feed parsing code, to the reduction in the number of plugins installed, or even to the accumulated improvements of all of those new contributors and Dolgov's surge of development in March. Whatever the cause, I last used TT-RSS more than two years ago and frequently found it pausing momentarily when switching to a new folder or when loading large posts; I have noticed no such hangs with version 1.10.

A resurgence in interest is welcome news for any free software project. The lone dark cloud in the recent uptick in TT-RSS's popularity has been the project's hostility toward forum users who bring suggestions or questions that the team does not like. The FAQ directs users with Google Reader–related requests to an antagonistic forum thread where one such user is ridiculed, and the same FAQ page defends Dolgov's right to "either mock or not deal with people" who ask "stupid questions again and again because new people gracing tt-rss community with their presence largely can't be bothered to read anything". The justification given for this approach is that community management is too time-consuming.

No doubt community management is a task that requires a significant time investment, but it is hardly an unsolved problem—and one would think the ten-fold increase in volunteers seen last March could have provided some person-power that could be directed toward community building. Prickly project leads are not a new problem, but TT-RSS is quality code and it was in essence handed a ready-made stable of millions of potential users when Google Reader closed; it would be a shame to drive them away.

Comments (7 posted)

Some tidbits from XDC

The X.Org Developers Conference (XDC) is a three-day, one-track event with presentations covering many different parts of the graphics stack. This year it was held September 23–25 at Portland State University in Portland, Oregon—home of Voodoo Doughnuts, which were provided daily. XDC is an intense experience and this year's edition will lead to a few more articles in the coming weeks. There were also a few shorter sessions with some news and plans that seem worth reporting on here.

Non-profit status

X.Org Foundation board member Peter Hutterer reported on the state of the foundation. The most recent news about the foundation was that it had lost its 501(c)(3) (US) non-profit status in August. Hutterer was happy to report that had all been reversed. With help from the Software Freedom Law Center (SFLC), which helped the foundation become a non-profit in 2012, the foundation was able to regain its status. The paperwork was "still in transit", he said, but the non-profit status was restored.

Part of the problem that led to the revocation of the non-profit status was that it is a lot of work to maintain a 501(c)(3) organization. The members of the foundation board are not lawyers or accountants, he said, and some of them are not even living in the US, which makes it that much more difficult. So the board decided to look into organizations that manage free software projects, foundations, and the like. Umbrella organizations like the Software Freedom Conservancy (SFC), Apache Software Foundation (ASF), and Software in the Public Interest (SPI) are set up to handle much of the paperwork for various types of projects.

The board has voted that SPI is the right umbrella organization for the X.Org Foundation. That switch is not finalized as it may require a vote of the X.Org membership, Hutterer said. The board will be consulting with the SFLC and looking at the by-laws to determine that. Assuming the change is made, SPI would take 5% of any donations made to the foundation for the work that it does, which "seems fair", he said.

The foundation has "a slab of money" that remains from a number of years ago, when it was getting donations of $100,000 or so per year. It uses that money to put on XDC and to sponsor the travel of several participants (four this year, including a Google Summer of Code student and an LWN editor). It also funds GSoC students and participants in the X.Org Endless Vacation of Code program. The pile of money is enough to last for another four or five years, Hutterer said, before the foundation needs to consider doing some fundraising—something that's never been done since he became involved.

The foundation is also moving banks after HSBC closed its account for unclear reasons. "Banks are fun", Hutterer said with a laugh. The current plan is to move to Bank of America, he said.

The Board of Directors consists of eight people and four of those seats turn over every year. There are 78 members of the foundation, which is "lots better than it was a couple of years ago". Hutterer strongly encouraged those present to think about joining, which allows voting in elections and has a few other benefits.

X server 1.15 planning

Keith Packard took the floor on the first day to discuss plans for the X server 1.15 release. It was supposed to have been released the week of XDC, but in August he and others realized that the release itself was "really boring". He asked the assembled developers if there were any features due in 1.15 that they were "desperate to have". Hutterer mentioned that having touch input working would be nice, but Packard noted that those changes had been backported to 1.14.

In fact, as far as he knows, all of the security, stability, and usability bug fixes have been added to 1.14. The 1.14 release manager has been making minor releases with those changes, which are, of course, ABI-compatible with 1.14—unlike 1.15. At this point, Packard said, 1.15 looks like "1.14 plus an ABI change".

There are, however, many features that are awaiting the release of 1.15 before they get merged. So, an idea that had been batted around on IRC was to delay the release of 1.15 until it had some features of note. Those features might include a rewrite of the Xephyr nested X server (that deleted "many thousands of lines of code, which is what we do best at Xorg", Packard said, pronouncing the last word as "zorg"), Packard's own DRI3 and Present extensions which are getting close to being ready to merge, some XWayland changes, Adam Jackson's GLX rewrite (which removes around 40,000 lines of code), and possibly others.

Packard would talk about DRI3 and Present later in the conference, as would Jackson about the GLX rewrite, so the final decision would be made after those discussions. All of the proposed features seemed like they would plausibly be ready in time for a code freeze at the end of October. The normal pattern would be for a two-month stabilization period after that, putting the release of 1.15 at the end of the year. "A Christmas present", Jackson suggested.

An informal straw poll of those in the room found all in favor of the proposed change, but there wasn't any real emotion one way or the other. "Consensus by apathy", one developer suggested, which is a "hallmark of X.Org" added another—to chuckles around the room. Packard encouraged anyone with additional features they would like to see in 1.15 to "let us know". In the end, the X.Org calendar shows a final 1.15 release scheduled for December 25.

Links to the slides and videos for most of the sessions can be found from the XDC schedule page.

[I would like to thank the X.Org Foundation for travel assistance to Portland for XDC.]

Comments (4 posted)

Integrity and embedded devices

David Safford's talk for the 2013 Linux Security Summit was in two parts—with two separate sets of slides. That's because the US Department of Homeland Security (DHS), which sponsored IBM's work on hardware roots of trust for embedded devices—part one of the talk—was quite clear that it didn't want to be associated with any kind of device cracking. So part two, which concerned circumventing "verified boot" on a Samsung ARM Chromebook, had to be a completely separate talk. The DHS's misgivings notwithstanding, the two topics are clearly related; understanding both leads to a clearer picture of the security of our devices.

The DHS is interested in what can be done to verify the integrity of the code that is running on all types of systems. For servers, desktops, and mobile devices, there are a variety of existing solutions: crypto hardware, secure boot, Trusted Platform Module (TPM) hardware, and so on. But for low-cost embedded devices like home routers, there is no support for integrity checking. The DHS is also interested in integrity checking for even lower cost sensors, Safford said, but those don't run Linux so they weren't part of his investigations.

Because routers and the like are such low-cost and low-margin systems, the researchers focused on what could be done for "zero cost" to the manufacturer. The team looked at four devices: the venerable Linksys WRT54G router (which spawned projects like OpenWrt and DD-WRT), the Pogoplug local network attached storage (NAS) cache, and two other router devices, the TP-Link MR3020 and D-Link DIR-505. All of those typically retail for around $50.

The MR3020, which was the focus for much of the talk, retails for $30. It has very limited space in its 4M flash chip, so the challenge to adding integrity features is "not so much technical" as it is in "squeezing things in" to the flash. For example, the MR3020 only has 64K of space for U-Boot, 1M for the kernel, and 2.8M for the root filesystem.

Safford gave a handful of examples of router vulnerabilities over the last few years. Beyond just theoretical examples, he noted that 4.5 million home routers were actually compromised in Brazil in 2011. While the vulnerabilities he listed were at the higher levels (typically the web interface), they do show that these embedded devices are most certainly targets.

So, without increasing the cost of these devices, what can be done to ensure that the firmware is what it is expected to be? In the supply chain, the wrong firmware could be added when the system is built or changed somewhere along the way. Safford said that IBM had some (unnamed) customers who had run into just this kind of problem.

So there needs to be a way to "measure" the firmware's integrity in hardware and then to lock the firmware down so that rootkits or other malware cannot modify it. In addition, these devices typically do not have support for signed updates, so malicious update files can be distributed and installed. There is also no ability for the system to refuse to boot if the code has been changed (i.e. secure or trusted boot).

Providing those capabilities was the goal for the project, he said. He showed a table (also present in his slides [PDF] and associated paper [PDF]) outlining the abilities of each of the devices in four separate integrity categories: "Measure BIOS?", "Lock BIOS?", "Secure local updates?", and "Secure Boot?". All of the boxes were "No", except that both the Pogoplug and WRT54G had a way to measure—verify—the firmware (by reading it using SATA via an immutable boot ROM and JTAG, respectively). By the end of the talk, those boxes had all been changed to "Yes" by the changes Safford and his team had made.

The traditional approaches for integrity revolve around either attestation (e.g. trusted boot) or a trusted chain of signed code as in UEFI secure boot. Attestation means that a system uses a TPM to measure everything read and executed, then sends that information to a trusted system for verification before being allowed to continue. There are several National Institute of Standards and Technology (NIST) standards that govern parts of the integrity puzzle, including trusted boot, but there are none, at least yet, that govern secure boot. Safford is working with NIST to get that process started.

Since a TPM chip is "expensive" ($0.75), it violates the zero-cost constraint. But in order to verify the firmware, it must be read somehow. The firmware itself cannot be involved in that step as it may lie about its contents to avoid malware detection. The Serial Peripheral Interface (SPI) bus provides a mechanism to read the contents of the flash for devices lacking other means (e.g. JTAG). That bus can be shared if it has proper buffering, but both the MR3020 and DIR-505 lack the resistors needed.

Enter the Bus Pirate—a device that can be used to read the SPI bus. Using it requires adding three buffering resistors to the Atheros System-on-Chip (SoC) used by the devices, but that adds less than $0.01 to the cost of the device, which is close enough to zero cost that device makers can probably be convinced. That means that users (or device makers) can verify the contents of the flash fairly inexpensively (a Bus Pirate costs around $30).

Once the contents of the flash are verified, there needs to be a way to lock it down so that it can only be modified by those verified to be physically present. The SPI flash chips used by all of the devices have a status register that governs which addresses in the flash can be written, along with an overall write-disable bit. That register can be locked from any updates by holding the chip's write-protect (!WP) pin low. Physical presence can be proved by holding down a button at boot to drive !WP high.

Safford showed the modifications made to the MR3020 and DIR-505 to support the physical presence test. The WPS (Wireless Protected Setup) button was repurposed on the MR3020, while an unused sliding switch position was used on the DIR-505. The paper indicates that similar changes were made on the other two devices. Both the slides and paper have pictures of the modifications made to the devices. In addition, U-Boot was modified so that it locks the entire flash on each boot, but if !WP is held high when power is applied, U-Boot will unlock the flash.

Adding secure boot support to U-Boot was the next step. Once the root of trust is extended into the kernel, the kernel's integrity subsystem can take over to handle integrity verification from there. So it is a matter of verifying the kernel itself. The modified U-Boot will use a public key that is stored at the end of its partition to verify the signature of the kernel. That signature is stored at the end of the kernel partition.

As mentioned earlier, the trick is in getting that code (and key) to fit into the 64K U-Boot partition. Using code derived from PolarSSL, with everything unneeded removed, the modified U-Boot weighed in at 62K. Though Safford was never very specific, the U-Boot modifications must also provide RSA signature checking for updates to the firmware. Support for signed updates is one of the integrity requirements that were successfully tackled by the project.

Through some effectively zero-cost modifications, and some changes to the firmware, the team was able to achieve its integrity goals. All of the devices now support all four of the integrity requirements they set out to fulfill.

Breaking verified boot

Moving on to the DHS-unapproved portion of the talk, Safford showed how one can take control of a Samsung ARM Chromebook. The work on that was done in his spare time, he said, but many of the tools used for adding secure boot for embedded devices are the same as those for removing and altering a system with secure boot. The Chromebook is a "very secure" system, but the verified boot (VB) mechanism does not allow users to take control of the boot process.

However, a fairly simple hardware modification (removing a washer to change the !WP signal) will allow the owner to take control of the device, as Safford found. Beyond the hardware change, it also requires some scripts and instructions [tar.gz] that Safford wrote. Unlike the embedded devices described above, there is a full 4M flash just for U-Boot on the Chromebook, so there is "an embarrassment of riches" for adding code on those systems. VB has been added to the U-Boot upstream code, incidentally, but it is way too large (700K) for use in routers, he said.

In the normal VB operation, there is no way to write to the upper half of the SPI flash, which contains a copy of U-Boot and Google's root key. That key is used to verify two keys (firmware and kernel) stored in the read-write half of the SPI flash. The firmware key is used to verify another copy of U-Boot that lives in the modifiable portion of the flash. That U-Boot is responsible for verifying the kernel (which actually lives in a separate MMC flash) before booting it.

Holding down ESC and "refresh" while powering on the system will boot whatever kernel is installed, without checking the signatures. That is the "developer mode" for the system, but it circumvents secure boot, which is not what Safford set out to do. He wants to use secure boot but control the keys himself. In addition, developer mode must be enabled each time the system boots and you get a "scary screen" that says "OS verification is OFF".

A less scary approach is to use a non-verifying U-Boot that gets installed in place of the kernel. That U-Boot is signed, but does no verification of the kernel (installed in a different part of the MMC flash) before booting it. That way you don't have to invoke developer mode, nor do you get the scary screen, but you still don't get secure boot.

Removing the washer is the way forward as it allows read-write access to the entire SPI flash. Once that is done, Safford has a set of scripts that can be run from a developer-mode kernel to create new key pairs, sign the read-write U-Boot, sign the kernel, and verify all of the signatures. If any of that goes wrong, one may end up at the "Chrome OS is missing or damaged" screen, which actually means the device hasn't been "bricked" and can be restored from a USB device. Even in the event of bricking, one can recover the device using Bus Pirate as the SPI flash is properly buffered, he said (seemingly from a fair amount of experience).

As part of his demo, he wanted an easy way to show that he had gained control of the low-level boot code in the SPI flash. He decided to change the "chrome" text in the upper left of the "scary screen" to "DaveOS", which actually turned out to be one of the harder ways to demonstrate it. Because of the format of the logo and where it was stored in the flash, it turned out to be rather painful to change, he said with a chuckle.

As Kees Cook pointed out, the washer removal trick was a deliberate choice in the design of the system. Google and Samsung wanted people to be able to take control of the keys for the device, but didn't want an attacker to be able to do so quickly while the user's attention was momentarily distracted. Safford agreed that it was a reasonable compromise, but that it is important for users to be able to set their own keys.

The slides [PDF] for the second half of the talk are instructive as well, with a number of pictures of the infamous washer, scary and DaveOS screens, the Bus Pirate in action, and so on. Seeing the problem from both angles, adding and subtracting secure boot functionality, was useful to help better understand integrity verification. Techniques like secure boot certainly can be used in user-unfriendly ways to lock down devices, but it can also provide some amount of peace of mind. As long as users can provide their own keys—or disable the feature entirely—secure boot is likely be a boon for many.

[I would like to thank LWN subscribers for travel assistance to New Orleans for LSS.]

Comments (8 posted)

Security quotes of the week

Comments (10 posted)

chicken: code execution

Comments (none posted)

davfs2: privilege escalation

Comments (none posted)

firefox: denial of service

Comments (none posted)

glibc: multiple vulnerabilities

Comments (1 posted)

gpg2: information disclosure

Comments (none posted)

kernel: denial of service

Comments (none posted)

kernel: information leak

Comments (none posted)

kernel: off by one error

Comments (none posted)

monkeyd: multiple vulnerabilities

Comments (none posted)

mozilla: privilege escalation

Comments (none posted)

MRG Grid: denial of service

Comments (none posted)

nas: multiple vulnerabilities

Comments (none posted)

openstack-keystone: incorrect token revocation

Comments (none posted)

squid: information disclosure

Comments (none posted)

sudo: privilege escalation

Comments (none posted)

tpp: code execution

Comments (none posted)

txt2man: file overwrite

Comments (none posted)

vino: denial of service

Comments (none posted)

zabbix: man-in-the-middle attacks

Comments (none posted)

Kernel release status

Stable updates: 3.11.2, 3.10.13, 3.4.63, and 3.0.97 were released on September 26; they were followed by 3.11.3, 3.10.14, 3.4.64, and 3.0.98 on October 1.

Comments (none posted)

Quotes of the week

— Ingo Molnar

    /*
     * Sent from my mobile phone.  Please pardon brevity and lack of formatting.
     */

Comments (11 posted)

Exchanging two files

The problem Miklos is trying to solve is the problem of exchanging two files — both files continue to exist, but their names have been swapped. To achieve this, he has posted a patch set adding a new renameat2() system call:

    int renameat2(int olddir, const char *oldname, 
		  int newdir, const char *newname, unsigned int flags);

This system call differs from renameat() in that it has the new flags argument; if flags is zero, renameat2() behaves exactly as renameat(). If, instead, flags contains RENAME_EXCHANGE, an existing file at newname will not be deleted; instead, it will be renamed to oldname. Thus, with this flag, renameat2() can be used to atomically exchange two files. The main use case for renameat2() is to support union filesystems, where it is often desirable to atomically replace files or directories with "whiteouts" indicating that they have been deleted. One could imagine other possibilities as well; Miklos suggests atomically replacing a directory with a symbolic link as one of them.

No review comments have been posted as of this writing.

Comments (16 posted)

Transactional memory in the dentry cache

One might think that it would be hard to improve on the performance of a lockref in this area, but there is one technology that might just help in this regard: transactional memory. For some years now, transactional memory has been one of those hardware features that was destined to solve a number of our scalability problems. As is often the case, that future has been slow to arrive in the real world. But some manufacturers are now shipping CPUs with some transactional memory support built into them, and software is starting to take advantage of this feature. See, for example, this article describing the work done to add transactional memory support to the GNU C library.

Linus recently got an itch to upgrade to a newer, faster desktop computer; when he chose his new processor, he made a point of getting one that provided transactional memory support. So, he decided, the time had come to try to use that support to further speed dentry cache locking. The result was a short, proof-of-concept patch that shows how things could work. The core part of the patch is worth showing directly:

    asm goto("xbegin %l[repeat]": : :"memory","ax":repeat);
    if (unlikely(d_unhashed(dentry)))
    	goto xabort;
    if (unlikely(!is_simple_dput(dentry)))
    	goto xabort;
    if (unlikely(!arch_spin_value_unlocked(dentry->d_lock.rlock.raw_lock)))
    	goto xabort;
    dentry->d_lockref.count--;
    asm volatile("xend");
    return;

xabort:
    asm volatile("xabort $0");

repeat:
    /* Fallback to lockref code */

The first line adds the xbegin instruction that starts the transaction. This instruction behaves a little strangely, in that its influence extends over the code that follows. Execution will continue with the code after the xbegin, but, should the transaction abort for any reason, all changes made during the transaction will be rolled back and control will jump to the address provided to xbegin (the label repeat in this case).

What follows is a set of tests to determine whether it is truly safe to decrement the reference count without holding the dentry spinlock. In particular, the dentry must be hashed (have a name associated with it, essentially), be on the LRU list, and not be locked. Reading these fields of the dentry structure adds them to the transaction; should some other code make a change to one of them, the transaction will be aborted by the processor. If any of the tests show that the dentry is not in a suitable state for a quick reference count decrement, the code uses the xabort instruction to make the transaction fail. Otherwise, the reference count will be decremented and the xend instruction will bring the transaction to a successful conclusion. The reference count, too, will become part of the transaction; should any other processor also try to change it, the transaction will abort.

The code as written is clearly not intended for merging; one does not put late-model x86 assembly code directly into generic filesystem code, after all. But it is enough to get a sense for how well transactional memory works in this situation. According to Linus, "profiles with this look beautiful", with the locking cost associated with the dput() operation reduced essentially to zero. So there may well be a place for transactional memory within the dentry cache.

Whether this technique will see much wider use in the kernel remains to be seen, though. Andi Kleen posted a patch using transactional memory for most kernel locks back in March, but that work did not go very far, mostly because he could not provide any benchmark numbers showing what kind of performance improvements could be expected. In his patch, Linus made it clear that he suspects those improvements will be small, saying "I doubt the intel TSX patches are going to be useful (if they were, Intel would be crowing about performance numbers now that the CPU's are out, and they aren't)." Instead, he has suggested that this feature will only be useful in a few places:

The end result is that Andi's patches may be a good starting point for transactional memory support in a more architecture-independent way, but we may not see transactional memory used as a general locking mechanism (or lock-elision mechanism) in the kernel.

Along the lines of architecture-independence, it is worth noting that Intel was not the first to ship a processor with transactional memory support; the PowerPC architecture has had similar functionality for a couple of years. Naturally, the feature works a little differently in that architecture, but the basic functionality is the same. So it should be possible to create some sort of wrappers around transactional memory that can be used in common code. That is, if kernel-space transactional memory can be made to work at all in an efficient manner on PowerPC; there are some challenges in the way of getting that functionality working reliably.

What one might conclude is that, while transactional memory is a useful technology for speeding up specific bits of high-profile kernel code, it does not look like a general solution to the locking problem at this point. That could change in the future, of course, as hardware transactional memory gets faster and more capable. But, for now, transactional memory looks mostly like a useful trick to squeeze a little more performance out of a few highly optimized code paths.

Comments (9 posted)

How much memory power management is useful?

Srivatsa S. Bhat's memory power management patch set was last covered here in April 2013. Many of the core principles remain the same, but some of the details have changed significantly.

The patch set is still based on the notion of dividing main memory into "regions" that match the power-management boundaries of the underlying hardware. At the hardware level, a region is a range of memory that can be powered up or down as a unit. The fundamental goal of the memory power management patches is to keep as many of these regions free as possible, enabling them to be kept in a low-power state.

Regions have some similarity to the "zones" maintained by the memory management subsystem. But zone boundaries are defined by the kernel, and have little to do with the hardware architecture (though zones will not span multiple NUMA nodes). Regions are, instead, a hardware concept. Since there is little commonality between regions and zones, the patch set ends up defining regions as an independent structure, almost entirely distinct from zones.

The first step is then simple: when allocating pages, do so from the lowest-numbered region with available pages. That will tend to concentrate allocations at one end of the memory range, keeping the higher-numbered regions free. To facilitate allocation in this mode, the page allocator's free list is modified to keep free pages sorted by region; that allows pages from the appropriate regions to be found quickly, with no need to scan through the list. The allocation logic is also changed a bit. Currently, the kernel will try to avoid splitting up large contiguous areas to satisfy an allocation request if a suitable (and smaller) chunk of memory is available. The memory power management patches sacrifice that heuristic to the extent that large chunks in low-numbered zones will be split rather than allocating from a smaller chunk in a higher-numbered zone.

The next step is somewhat more invasive: a "region allocator" is added to the memory management subsystem; it manages memory in large blocks corresponding to hardware regions. In current kernels, the page allocator works directly with the system's available memory; in the new scheme, instead, the page allocator allocates memory from regions of memory obtained from the region allocator. In other words, the region allocator has been placed between the page allocator and physical memory. If the page allocator needs more memory, it will request a new region from the region allocator. If, instead, the page allocator realizes that it has a full region on the free list, that region can be returned to the region allocator.

A determined effort is made to ensure that all allocations from any given region have the same "migration type." A page's migration type describes how easily the contents of that page could be moved elsewhere. Anonymous pages for user-space use are relatively easy to move; all that is required is to copy the data and change any page table entries that point to the page. Pages used in the kernel's slab allocators, instead, are firmly nailed into place; moving such a page would require finding and changing all of the kernel-space pointers referring to objects allocated from that page — not an easily solved problem. The purpose here is straightforward enough: it only takes one non-movable page to keep an entire region powered up. If all of those unmovable pages can be kept separate from pages that are more easily relocated, it will be easier to free regions of memory.

The final part of the patch set takes advantage of movable pages to actively free zones of memory when the conditions are right. For example, if a region containing movable pages is mostly empty and free pages are available elsewhere, the kernel will attempt to relocate data out of that region and, once the region is empty, return it to the region allocator. In other cases (clean page-cache pages, for example), the pages can simply be reclaimed. In this way, it is hoped, the inevitable fragmentation that occurs over time can be cleaned up, keeping the maximum number of regions free.

Most of this 40-part patch set is relatively uncontroversial, but there are some worries about the active defragmentation measures. If they are not applied carefully, they could end up increasing power consumption rather than decreasing it: moving pages around takes power, as does replacing data in the page cache that was reclaimed prematurely. So there needs to be some clear evidence that the active measures help to reduce power consumption; thus far, that evidence does not exist, since no power savings benchmark results have been posted for some time.

More to the point, there are concerns that the active measures may reflect a bit of a mismatch between the design of the patch set and how memory power management actually happens on current hardware. The core idea is to keep as many memory regions entirely free of data as possible; that will allow those regions to be powered down — losing all data stored therein — without causing problems. In this model, a single active page can keep a region from being powered down, so regions must be completely evacuated for the power savings to be realized.

But, as Arjan van de Ven explained, memory power management, on current Intel hardware at least, works a bit differently. It has a number of reduced-power modes, most of which are entirely transparent to the operating system and preserve the data stored in memory. Transitions to and from the reduced-power states are quite fast, to the point that latency problems are not a real concern. For this type of power management to kick in, all that is required is that the relevant region of memory be idle most of the time. As Arjan put it:

In other words, if a significant number of regions can be kept mostly empty, it doesn't matter that much if the last few pages are still in occasional use. That suggests that the parts of Srivatsa's patch set that control the allocation patterns are all that's needed to reap most of the power-savings rewards — on Intel hardware at least. The active clearing of regions may turn out to be a futile effort to gain the last few percent of the available savings. Of course, not all hardware is made by Intel; results for other processors and architectures may vary. Even on Intel systems, there may well be situations where it makes sense to support full, content-destroying memory power-down. But the fact that most of the benefits can apparently be obtained without actively moving pages around will argue for, at least, that feature being turned off by default.

Note the use of the word "apparently" above, though. What is lacking from the discussion at this point is any sort of comprehensive measurements of actual power use with and without these patches. Real measurements, which generally trump more theoretical discussions, should be the deciding factor when the memory management maintainers decide how much of this work to merge and whether the active defragmentation functionality should be enabled by default or not. Either way, the bulk of the patch set seems to be relatively uncontroversial, so we should see some sort of movement on memory power management, finally, in the near future.

Comments (5 posted)

NUMA scheduling progress

NUMA balancing was a topic of fierce debate through much of 2012; that discussion culminated with the merging of Mel Gorman's NUMA balancing infrastructure patch set into the 3.8 kernel. Those patches provided the basic structure upon which a NUMA balancing solution could be built, but did not attempt to solve the problem in a comprehensive way. Since then, one might be forgiven for thinking that the developers involved have lost interest; not much NUMA-related code has found its way into the mainline. But, as can be seen in Mel's basic scheduler support for NUMA balancing patch set, which weighs in at 63 individual changesets, quite a bit of work has been happening in this area.

The NUMA balancing problem is one of overcoming a fundamental characteristic of NUMA hardware: accesses to memory local to any given NUMA node will be much faster than accesses to remote memory. If processes and their memory live on the same NUMA node, things will run quickly; otherwise performance will be significantly worse. NUMA performance on Linux has long been deemed to be worse than it should be, so there is clearly some room for improvement in this area.

The kernel has, for years, attempted to allocate memory on the same node that the allocating process is running on, but that turns out to be inadequate. Over time, it seems to be inevitable that processes and memory will move around the system, leading to poor performance. Recent efforts to improve the situation have thus been focused on cleaning things up as the system runs, moving memory to the nodes that are actually using it and moving processes to be closer to their memory. This particular patch set, as suggested by the "scheduler support" name, is mainly concerned with the latter task — ensuring that the scheduler runs processes on the nodes that host the bulk of their memory.

Locating a process's memory

If the scheduler is to keep processes near their memory, it must, clearly, have a sense for where that memory is placed. This information is not as straightforward to come by as one might expect, especially if one is concerned mostly with the memory that is actually being used, as opposed to memory which has been allocated but is not in active use. The kernel needs to continually observe a process's memory access patterns to be able to optimize its placement in the system.

One of the core changes in the current NUMA patch set is to enable this observation, using the virtual memory mechanism. The scheduler will periodically scan through each process's address space, revoking all access permissions to the pages that are currently resident in RAM. The next time the affected process tries to access that memory, a page fault will result. The scheduler will trap that fault and restore access to the page in question; it will also increment an access counter in a per-process array indexed by the NUMA node number. Over time, these counts (which are maintained as a sort of running average) will provide a picture of where the process's memory accesses are going. At that point, it is a simple matter of looping through the array to find the node with the most accesses; that becomes the "preferred node" for the process.

If the process is running on a node other than the preferred one, the scheduler will attempt to migrate it to the node where its memory lives. That should result in more node-local memory accesses and, thus, better performance. Of course, the picture is a bit more complicated than this for a number of reasons, starting with the fact that the migration of the process could fail for a number of reasons. So one of the patches in the series will periodically retry the attempt to migrate a process to its preferred node if things didn't work out the first time. Even if the initial migration attempt fails, the process should eventually end up on the preferred node.

This process-follow-memory migration can also run counter to another one of the scheduler's core goals: distributing the load across the system to make the best use of the available CPU time. If a process that has been migrated closer to its memory cannot actually run because the destination NUMA node is overloaded, the goal of improved performance is unlikely to be achieved. So it may be necessary to migrate other processes off the overloaded node. That complicates the picture somewhat. The patch set does not try address the full problem, but it does take on one specific subproblem: cases where swapping two processes between two CPUs will make both processes run more efficiently. Even that task is fraught with hazards: moving two processes at once is more complicated than migrating a single process. To make it work with a minimum of fuss, the patch set adds a variant of stop_machine() that effectively halts work on the two processors involved while the exchange is made.

It also became necessary to avoid tracking NUMA access information for shared executable pages. Including those pages will tend to pull processes toward the nodes where pages holding shared library code can be found. But there is relatively little value in migrating processes near their executable pages, as it turns out, because much of the needed data is in the CPU caches much of the time anyway. So patch 35 in the series avoids tracking faults from those pages.

NUMA groups

As described thus far, the patch set adds a basic mechanism for putting a process near its (exclusively owned) pages. But the concept of "a process's memory" is not necessarily simple either; processes often share pages with each other. That is especially true of threaded applications where all of the threads run in the same address space and have access to the same memory. Threaded applications written with NUMA performance in mind will partition their memory accesses, but they can still end up sharing pages, especially if a feature like transparent huge pages is in use. In other cases, entirely separate processes may still share a large memory area; neither "owns" it, but both can benefit by being placed close to it. Either way, there can be performance benefits from placing such processes on the same node whenever possible.

To improve performance in the shared memory case, the patch set adds a mechanism that tries to detect when processes are sharing memory with each other — and, importantly, when they are accessing the same pages. Doing so requires tracking which processes are accessing each page. Andrea Arcangeli's AutoNUMA patch set, one of the contenders in the 2012 discussions, added an extensive tracking infrastructure for this purpose, but the resulting memory overhead was deemed to be too high. So, as is so often the case, this patch set tries to track this data by shoehorning the data into struct page instead.

In particular, the data is put into the flags field, which is already rather more complex than a simple set of flags. The ID of the NUMA node containing each page is stored there in current kernels; this patch set changes that to the CPU ID, and adds the process ID of the last process to access the page as well. Needless to say, the full process ID does not fit into a subfield of flags, so some compromises need to be made. In this case, only the bottom eight bits of the process ID are used, with the understanding that some collisions will be unavoidable. Whenever the system handles a page fault, this "cpupid" number is stored in the relevant page structure.

With that infrastructure in place, the scheduler can respond to a NUMA scanning fault by checking whether the process currently accessing the page is the same as the last process to do so. If not, the scheduler will consider putting the two processes into a "NUMA group." But first, it must find the other process that accessed the page — not a trivial matter, given that the full process ID for that process is not available. This problem is handled by looking at whichever process is currently running on the CPU that last accessed the page; if the relevant part of the process ID matches, the scheduler assumes (after applying a few sanity tests) that it is the same process as the one that accessed the page. In this case, the processes are placed into a NUMA group to indicate that they are sharing pages; if both processes are already in groups, those two groups will be coalesced into one larger group.

Once again, shared library pages threw a wrench into the works. Since every process shares access to, for example, the C library, all processes in the system tended to get pulled together into a single NUMA group. Deeming that to be less than fully useful, Peter Zijlstra tossed in a patch avoiding group tracking for read-only pages.

Once a process is part of a NUMA group, page faults for shared pages will be tracked at the group level. That gives the scheduler two somewhat independent sets of statistics from which to make decisions: accesses to private pages, and accesses to shared pages. If both sets of numbers agree on the preferred node, the decision is easy. If, instead, there is a disagreement, the preferred node will be chosen by iterating through the arrays and picking the node where the sum of private and shared faults is the highest. That sum is distorted, though, by weighting shared faults a bit more heavily in an attempt to push the system toward decisions that put processes with shared pages on the same node.

There is a lot more than this to the patch set, including a number of algorithmic tweaks and fixes, but the above text describes the most significant themes. Mel has included quite a bit of benchmark data with the patch set; there is a lot of information there, but the bottom line, as he described it, was: "some reports indicate that the performance is getting close to manual bindings for some workloads but your mileage will vary." A number of details are yet to be addressed; in particular, support for CPU hotplugging is not yet there. Even so, Mel thinks that the code is getting ready for movement into the "tip" tree, and, thus, linux-next. That suggests it could be ready for merging in the 3.13 or (perhaps more likely) 3.14 development cycle.

Comments (13 posted)

Fedora and bug tracking

Bug reports are a blessing and a curse: not enough and the software project misses out on vital feedback; too many and the project is overwhelmed or loses the signal to the noise. Recently, a peculiar discussion in the Fedora community highlighted the difficulty of wrangling bug reports for distributions, which are often the final "downstream" project in the free software deployment hierarchy.

Jóhann Guðmundsson proposed that Fedora move away from the Red Hat Bugzilla (on which Fedora has relied since the project's inception). But he did not propose setting up a standalone Fedora bug tracker; rather, he suggested adopting "pure upstream reporting", in which the Fedora community would file bugs directly with the upstream projects themselves.

He cited three reasons. First, many package maintainers feel that their bug reports go ignored by the upstream project if they are reported downstream. Second, the Fedora package maintainer frequently is not a developer, so he or she cannot fix the bug, which must eventually find its way upstream in order to get the proper attention (at the cost of increased latency). Third, neither of the aforementioned problems would solved by Fedora running its own issue tracker.

Guðmundsson's proposal was met with near-universal disagreement; Fedora contributors do not need to worry that their bug tracker will be going away any time soon. But the debate in the list thread is an interesting read for what it reveals about the hassle of cultivating and managing bug reports.

The primary objections to Guðmundsson's proposal were of a distribution-development nature, pointing out (for example) that there are always a significant number of bugs in Fedora which do not belong upstream. User nonamedotc pointed out that there are always bugs caused by Fedora-specific combinations of libraries or the particulars of the distribution's builds. Jan Wildeboer said that Fedora itself develops its own components (starting with the installer) which have no upstream project, and that the distribution needs a place to track blockers and critical bugs that are part of preparing each Fedora release. Tracking release status mandates having one centralized place to view all of the outstanding bugs in the distribution.

Upstream/downstream

On those technical grounds alone, there would seem to be no chance that Fedora would do away with its bug tracker. But Guðmundsson replied that Fedora already has an incomplete picture of all of the outstanding bugs because there are always bugs that are only reported upstream. Indeed, one of the points of agreement was that having separate bug trackers for the distribution and the upstream project is an ongoing challenge that needs addressing. Fedora tries to maintain good relationships with its upstream projects; staying in sync as much as possible while not inundating them with Fedora-specific bugs. Wildeboer argued that sending Fedora bug reporters directly to upstream projects would have a damaging effect on Fedora's relationship with those other projects:

Along the same lines, Michael Schwendt said that there are upstream developers who make use of the reports in Fedora's Bugzilla, both for the big picture" and to examine specific details like stack traces. If, as Guðmundsson had suggested, there is a breakdown in communication with the upstream project, then that is a problem that needs fixing by package maintainers, triagers, and testers.

Dan Horák suggested that Fedora needs a tool that can clone a bug report from Fedora's Bugzilla to an upstream repository. Right now, developers and package maintainers can copy a bug manually over to the upstream project's issue tracker, but it is a time-consuming ordeal. That suggestion met with some approval, although it was quickly pointed out that auto-populating an upstream project's bug tracker would certainly not be a good idea. The goal should be to reduce the amount of work for the packager, not to automate the cloning. As Jakub Jelinek pointed out, an automated clone of a bug can propagate excess information—in particular, he criticized the bug reports pushed upstream by Debian, which frequently include mail headers and list discussions from that distribution's email-based issue tracking system.

The other difficulty with establishing links from Fedora bugs to upstream bug trackers is, as Bill Nottingham observed, that not every upstream project has a bug tracker—or, at least, a bug tracker that is accessible to those outside the project. Furthermore, if the bug tracker exists, it may not be easy to find. Mateusz Marzantowicz asked Guðmundsson how a bug reporter would even know the canonical location of a bug tracker for any particular application; without a centralized list or database, the location of the bug tracker would need to be stored somewhere—perhaps inside of the RPM package itself. But this is a problem that faces package maintainers already, even without Guðmundsson's upstream-first idea. Some projects have no bug tracker, new package maintainers may have to familiarize themselves with a new system for each package, and so on.

Similarly, Felix Miata noted that, in the absence of a Fedora-specific bug tracker, users would often not know where to report a bug that could have multiple causes:

Yet this, too, is already a problem that confronts bug reporters, as Adam Williamson of Fedora's QA team said. It may be unclear where the source of a particular bug lies (if it is possible to determine the source from one user's system), but bugs in Fedora itself will always be one of those possibilities. Ultimately, Nottingham concluded that the root problem is one of personnel:

The personnel issue is also at the heart of another major criticism of Guðmundsson's proposal: if bugs must be reported upstream, then bug reporters must seek out the upstream project's bug tracker and create an account for every package that they report issues against. That could easily be dozens or hundreds of accounts for active participants, and the more active a bug reporter is, the worse the problem would be. Several list participants raised this objection, with Michal Jaegermann going so far as to say that he would stop reporting bugs altogether under such circumstances.

Role-playing

No doubt Fedora (like any free software project) would love to have excess people-power to throw at the challenges of bug reporting and triage. But the shortage that Fedora currently feels is not likely to resolve itself, so the QA and testing community will probably continue to find a significant amount of its time occupied by managing bug reports. The last factor in the equation, perhaps, is determining which people ought to be responsible for wrangling bug reports: in particular, deciding when pushing them upstream is the correct solution.

Guðmundsson argued in his proposal that Fedora package maintainers are usually not developers and are often not even familiar with the codebase of the package itself—that was one reason he advocated reporting bugs directly to the upstream project. But several on the list disagreed with that argument. Schwendt pointed to the guidelines for package maintainers laid out in the Fedora wiki, which state (but do not mandate) that maintainers should create an account on upstream bug trackers and forward bugs upstream.

Ralf Corsepius said that pushing bug reports upstream was a maintainer's duty, as was implementing patches in some situations. Of course, the trouble is that package maintainers come and go; when important packages are orphaned, it is not uncommon for a new volunteer to take up the mantle of maintainership even if he or she is less-than-fully familiar with the package in question. No doubt it is the new maintainer's responsibility to get up to speed, but at any given time there are liable to be numerous packages whose maintainers are not prepared to send in a patch. Schwendt, however, argued that not every package maintainer needs to be a developer, too:

The difficulty, of course, is that package maintainership sits in the gray zone that lies somewhere between "user" and "developer." The free software community often talks about itself as if these two camps are the only categories, which is an oversimplification. Guðmundsson's frustration with how bug reports are currently handled in Fedora illustrates how that oversimplification plays out: there are package maintainers who cannot process and route bug reports quickly enough to keep the bug queue clear and the end users happy.

But Guðmundsson's upstream-only proposal would essentially have addressed that problem by relieving package maintainers of the responsibility for routing bug reports altogether. That might have come as a relief to some overworked packagers, but the work would not simply disappear: it would end up being pushed to some other segment of the community, either to the Fedora QA team, to the upstream project, or all the way to the end user.

For that and the other issues pointed out, Fedora is clearly not going to discontinue the use of its bug tracker. That means, however, that managing bug reports will remain a tricky process for some time to come. Perhaps the changes to Bugzilla proposed by Horák will ameliorate the headaches to some degree, but in the long run it is likely to remain (as so many software-development hurdles are) a human problem.

Comments (3 posted)

Distribution quotes of the week

Comments (none posted)

Debian Edu / Skolelinux Wheezy

Full Story (comments: none)

New GNU Hurd, Mach, and MIG releases

Comments (22 posted)

FreeBSD 9.2 released

Comments (1 posted)

NetBSD 6.1.2 and NetBSD 6.0.3 released

Comments (1 posted)

Red Hat Enterprise Linux 5.10 released

Comments (none posted)

Ubuntu 13.10 (Saucy Salamander) Final Beta released

Full Story (comments: none)

Red Hat Enterprise Linux 5.3 Advanced Mission Critical 6-month Notice

Full Story (comments: none)

Red Hat Enterprise MRG for Red Hat Enterprise Linux 5 6-month Notice

Full Story (comments: none)

No Mir by default in Ubuntu 13.10

Full Story (comments: 53)

Distribution newsletters

• This Week in CyanogenMod (September 28)
• Debian Project News (September 30)
• DistroWatch Weekly, Issue 527 (September 30)
• Ubuntu Weekly Newsletter, Issue 336 (September 29)

Comments (none posted)

Patrick d'Emmabuntüs: Emmabuntüs is more than a Linux distribution (DarkDuck)

Comments (none posted)

Application metadata with AppData

GNOME is deploying GNOME Software, an "app store"–style software manager, with the GNOME 3.10 release. The goal is to facilitate finding an application to install by scouring a catalog of human-readable descriptions and illustrative screenshots—in contrast to the raw details exposed by a traditional Linux package manager, which can get in the user's way. One challenge that might not be immediately obvious is collecting and collating the descriptions for all of the applications in a standardized fashion. To that end, there is the newly-drafted AppData specification.

The AppData specification is the brainchild of GNOME Software developer Richard Hughes. GNOME Software is meant to complement a regular package manager, not to replace it. The idea is that GNOME Software will help users find an application when they know what they want to do, but they do not know what applications are available. So they may browse by category, look at popular applications, or search, but in all cases they rely on a well-written description of the application to guide them. For example, users might know they want a tool for scanning photographs, but might not know—or care—that such a tool involves SANE. These well-written descriptions are not easy to come by, however, nor are they readily available to import in bulk into GNOME Software.

In August of 2013, Hughes discussed several potential sources for the human-readable application descriptions, such as the .desktop files or Description of a Project (DOAP) files released by the application project, descriptions found in distribution packages, and "About" text from within the application itself. Each had drawbacks.

For instance, DOAP files are by definition one-per-project; thus LibreOffice has a single DOAP file, but GNOME Software would want separate descriptions for the office suite's Calc and Impress components—either one of which might be the application that the user is searching for. Package file descriptions and "About" text are inconsistent and frequently too terse to be helpful to an unfamiliar user in search of an application to install. Furthermore, most of the existing sources of human-readable descriptions fail to make room for multiple translations of the file to exist simultaneously, so it was almost a given that some form of new file would need to be created.

Hughes eventually drafted the AppData specification, with the intention that it would be usable for projects in addition to GNOME Software itself. Although the specification is hosted on the Freedesktop.org site, it is not currently a formal specification (to the extent that Freedesktop.org is ever "formal"). The file format is XML, and upstream projects can provide their own AppData files to be installed in /usr/share/appdata/.

The format includes an <id> element that should specify the name used in the application's .desktop file, a <licence> element that should specify the license on the AppData text itself (not the license of the application, which is found in the software package), a <description> element for the main application description text, plus <url> for the project's homepage, <updatecontact> for a contact email address, and <project_group> to associate an application with an umbrella project. The file can also link to image URLs with one or more <screenshot> elements. Subsequently, Hughes was asked by Debian's legal team to add a field for a copyright statement, which is necessary to comply with some software licenses.

The hard part: people

The format is simple enough, but the mere existence of a simple file format does not write project descriptions. For that, Hughes turned to the public to solicit contributions. The specification page includes a list of suggestions for writing a helpful description, such as ensuring that the description says what the application actually does, not what it is planned to do. Authors are encouraged to avoid lists where possible, avoid references to trademarked competitors by name, and told: "don't mention what language an application is written in, users don't care".

On August 29, Hughes started a campaign on his blog to get application developers, package maintainers, or knowledgeable volunteers to write descriptions and create AppData files. Over the course of the next month, that campaign evolved into a formal goal that all GNOME applications would provide an AppData file by the release of GNOME 3.14. An even bigger target soon followed: that all major end-user applications shipped in Fedora 20 would include AppData.

Gauging the progress of the campaign can be a little tricky, as the GNOME and Fedora goals overlap, but are (of course) tracked separately. In a series of posts on his blog, Hughes highlighted high-profile applications that were missing AppData or had incomplete AppData files (where "complete" means having both the human-readable description and at least one screenshot). As of September 27, there were still three dozen or so important applications with incomplete entries. There is a Google spreadsheet that tracks the current Fedora application list (which as of press time includes 17 entries, many of which were not listed in the blog post).

The list of incomplete or missing AppData files does appear to be shrinking, however. But the files do need some further care even when they are complete. There is a validation tool to check the syntax of the files and report formatting problems. GNOME also maintains a list a list of the 3.10 applications that have AppData files but have not passed validation.

Use your words

Aside from validation, one of the biggest challenges will be getting the application descriptions—and in many cases, the screenshots—translated into as many languages as possible. One of the benefits of the AppData specification is that (unlike DOAP), it allows multiple translations of the metadata file. Both ITS Tool and intltool are supported paths to translation; projects can ship their AppData as an .xml.in file for use with intltool or use the official appdata.its template for ITS Tool. GNOME maintains a to-do list on the AppData goal page of the GNOME wiki, which indicates not only which applications still lack AppData files, but which are still missing translations.

It may seem like a lot of work has gone into AppData for what is essentially metadata that a lot of experienced users already know. But that is part of the problem: someone who has been using desktop Linux for years likely knows what the evolution and aisleriot packages install. Someone who has just discovered them through Yum, Synaptic, or another package manager would have a hard time guessing.

GNOME and Fedora seem to be on track for adopting AppData as the preferred mechanism for delivering this sort of metadata, but Hughes is interested in making it a standard that upstream projects use, rather than an add-on supplied by packagers. It will be interesting to see how that process goes. There is certainly a need; the same problem is faced by other popular desktop distributions, but none of them have attempted to formalize the process. Ubuntu, for example, uses the Debian package description format implemented by the Debian Description Translation Project. These descriptions do not come with a formal file format (they exist primarily in the distributions' databases), although they are quite similar to what is asked for by the GNOME AppData effort.

Neither Debian nor Ubuntu pushes upstream projects to supply descriptions themselves (although the comments on Hughes's blog suggest that a number of AppData entries made use of the existing Debian/Ubuntu package descriptions). But both do offer web-based tools to facilitate making translations. Perhaps some cross-project cooperation is plausible; users would no doubt be happy as a result, with the prospect of a major reduction in the amount of head-scratching required to decipher the peculiar packages names that are so popular in free software.

Comments (1 posted)

Quotes of the week

Comments (3 posted)

texinfo-5.2 available

Version 5.2 of texinfo, the GNU documentation format, is available. Highlights include "more inline-style conditionals; makeinfo now warns about node/menu/cross-reference names problematic in Info, and info has a --all option somewhat analogous to man."

Full Story (comments: none)

WebKitGTK+ 2.2.0 released

Version 2.2.0 of WebKitGTK+ has been released. This release includes initial Wayland support, video accelerated compositing, custom JavaScript injection, and a new Web Inspector, in addition to several updated APIs.

Full Story (comments: none)

VLC media player 2.1.0 released

Full Story (comments: 9)

Rust 0.8 released

Comments (12 posted)

GNU APL 1.0 available

Version 1.0 of the GNU APL interpreter has been released, after several years of development. GNU APL is designed to be a complete implementation of APL2 as defined by ISO standard 13751.

Full Story (comments: 8)

GNU Guix 0.4 released

Release 0.4 of the GNU Guix package manager is available. According to the release announcement, "this release comes with a QEMU virtual machine image that demonstrates preliminary work toward building a stand-alone GNU system with Guix. The image uses the GNU Linux-Libre kernel and the GNU dmd init system. It is console-only, and may be used primarily to try out Guix."

Full Story (comments: 2)

Apache CloudStack 4.2 Released

Version 4.2 of Apache CloudStack has been released. This is a feature release from the 4.x series first released in November 2012. New features include "integrated support of the Cisco UCS compute chassis, SolidFire storage arrays, and the S3 storage protocol".

Full Story (comments: none)

systemd 208 released

Version 208 of the systemd init-replacement has been released. The major addition in this update is "support for facilitating privileged input and drm device access for unprivileged clients" in logind, which will allow Wayland display servers to run under the user's UID, thus permitting "secure session switching without allowing background sessions to eavesdrop on input and display data".

Full Story (comments: 2)

Development newsletters from the past week

• Caml Weekly News (October 1)
• What's cooking in git.git (September 25)
• Haskell Weekly News (September 25)
• OpenStack Community Weekly Newsletter (September 27)
• Perl Weekly (September 29)
• PostgreSQL Weekly News (September 30)
• Ruby Weekly (September 26)
• Tor Weekly News (October 2)

Comments (none posted)

Pasting images with automatic attribution

Peter Liljenberg has developed an add-on for Firefox that copies linked metadata to the clipboard in addition to the "copied" object itself. The initial demonstration of this technique required a specially-crafted page with RDFs metadata linked in, and thus may not seem immediately useful. However, Liljenberg has now implemented a more straightforward use case: copying and pasting an image with attribution data automatically preserved. "The friction for using a shared image is reduced when you don’t have to remember to also write an attribution. The attribution can embed the metadata, so that if someone copies the image from your page, they can also get an attribution created automatically when they paste it into their page."

Comments (5 posted)

Rempt: Ten years of working on Krita

Comments (8 posted)

30 years of GNU

Comments (126 posted)

Free Software Supporter - Issue 66

Full Story (comments: none)

Practical Programming, 2nd Edition--New from Pragmatic Bookshelf

Full Story (comments: none)

CFP Deadlines: October 3, 2013 to December 2, 2013

If the CFP deadline for your event does not appear here, please tell us about it.

Events: October 3, 2013 to December 2, 2013

If your event does not appear here, please tell us about it.