LWN.net Weekly Edition for April 16, 2009

PostgreSQL 8.4 Beta: "We've got momentum"

The PostgreSQL project has released the beta of its 8.4 version, with the final release expected in late June or July. Like previous PostgreSQL releases, 8.4 features dozens of enhancements throughout the code, but Bruce Momjian, a member of PostgreSQL's core team since the project was founded in 1996, described it as a "more surgical release" with improvements tending to cluster in select areas. In particular, that area seems to be administrative features. However, almost as interesting as the features in the beta are the ones that failed to meet the cut this time, and the project's struggle to control the release process as it undergoes major growth.

Momjian explained the release's more targeted approach as the result of the project's advanced state. "You see a real consolidation in this release," he said, "and I think that's because you're seeing a much more complete feature set. You're seeing a real maturity of the code base, which is kind of surprising, because, if you looked a few years ago, you'd see changes all over the map."

At the same time, Momjian characterized the PostgreSQL code as being constantly revised, so that the project is unlikely to need any time soon the kind of major revision that KDE underwent last year.

Nor is the project hesitant about altering behavior or deprecating legacy code in the name of what Momjian called "high standards and the promise of reliability."

New features

PostgreSQL releases tend to be one to two years apart, with far too many features to mention in any detail. Many of these features are highlighted in a PDF presentation by Momjian entitled, "Upcoming PostgreSQL Performance Features [PDF]", including Column Level Permissions and Per-Database Locales. However, if you ask active contributors what the major enhancements are in the 8.4 release, their answers emphasize automated administrative features.

Selena Deckelmann, user groups leader in the project, emphasized changes to Free Space Map, which maps unused space in a database. In previous releases, PostgreSQL could only detect newly freed space by an administrator manually running the Vacuum utility. By contrast, in the 8.4 release, freed space is re-mapped automatically, saving both time and effort.

Deckelmann also called attention to a new feature called Visibility Map. While in previous releases, Vacuum had to re-map all rows in a table, regardless of whether they had changed, Visibility Map improves performance by allowing the utility to skip rows that have not changed.

Yet another automated feature mentioned by Deckelmann is Auto Explain, which captures the explain plan for a query — that is, how the query finds results — and saves the results, information that can be used to improve system load and the efficiency of queries, and, ultimately, to reduce the costs of a database operation. "It's something that a lot of admins end up doing anyway," Deckelman observed, "They write a script that looks for long-running queries, and then they go in and manually figure with each one what's going on. It's kind of a neat feature that came from the Japanese PostgreSQL team."

Josh Berkus, another core team member, noted that Oracle's Statspack, a set of tools to generate statistics useful to database administrators, inspired pgstat. As Berkus explained the situation, users migrating from Oracle expect to see comparable tools in PostgreSQL, and Statspack:

Berkus noted, though, that pgstat will not be stable enough to be installed by default in 8.4. Instead, it will be shipped in postgresql-contrib, the repository for tools that, for one reason or the other, are not part of the regular installation. Some tools in contrib may be too specialized for most installations, or illegal to ship under American restrictions on the exporting of cryptographic tools, while others, like pgstat, are still in development.

According to Deckelmann, statistics in PostgreSQL receive another boost in the 8.4 release with increased control over them. To reduce system overhead, statistics are no longer automatically collected and written to a file, operations that can have significant system overhead if done regularly. Instead, statistics can now be configured to run from a RAM disk to improve performance.

However, if the latest release has a single outstanding feature, it is parallel restore. As the name suggests, parallel restore allows admins to restore a database with multiple processes, rather than a single thread. Berkus, who runs his own PostgreSQL consulting business, said, "[Parallel restore] is the feature i've been making the most use of. I've been using it in beta already — pretty heavily. Because if you have 300 gigabyte databases, upgrading them single-threaded is lethally slow. It's a real issue."

He went on to describe parallel restore as a particularly difficult feature to implement, all the more so because it had to accommodate past changes in the PostgreSQL file format. While employed as a supervisor at Sun Microsystems, Berkus said, he had two employees working on a similar feature for a year and a half "without coming up with more than a rough prototype." By contrast, PostgreSQL developed its version in three weeks, followed by three months of debugging.

Features for the next release

As mentioned earlier on LWN, PostgreSQL's core team had hopes of other major features being in the 8.4 release, particularly Hot Standby and SE-PostgreSQL.

Hot Standby is a feature that is a major step in improving PostgreSQL's replication. Replication is an area in which PostgreSQL lags behind MySQL; it has been identified by the core team as one of the major priorities for the project, according to Berkus. PostgreSQL does have some replication, Berkus said, but it is needlessly complicated and "not for someone who can't make a large time investment in finding out how it works. And that's a real problem for the sort of average-case web developer that has two servers and just wants to make sure that PostgreSQL is mirrored."

Hot Standby is an important improvement in replication that allows administrators to run queries on a database that is being recovered from an archive. The module allows replication of the database logs in order to create read-only duplicates of the database. Unfortunately, funding for development of the feature only came through in August 2008, leaving only a few months before the November code freeze. Nor could Hot Standby be finalized in the extended testing period that followed.

SE-PostgreSQL is an even greater innovation, which will add the SE Linux security model, making PostgreSQL the first database to use the same security model as many distributions, such as Fedora. The problem is, Berkus explained

Having failed to be ready for the 8.4 release, both Hot Standby and SE-PostgreSQL are high on the priority list for the next release. In fact, according to Momjian, 8.4 is partly designed to ease their later implementation. Momjian suggested that, given the maturity of PostgreSQL's code, these might be simply the first of many new features that are too large to be implemented in a single release. Meanwhile, he planned on promoting SE-PostgreSQL in the release notes in the hopes of encouraging interest in it and perhaps attracting a few developers with SE Linux experience.

"To continue to grow and continue to have the sort of reputation we have, you have to make some hard decisions," Momjian said. Like Berkus and Deckelmann, Momjian clearly regretted the necessity for omitting these features. However, having faced long delays because of the refusal to jettison unfinished features before — notably in the implementation of the Windows port in version 7.4 and more generally in the sheer number of features in version 8.3 — the PostgreSQL core team has learned the hard way not to insist that a release include all the features they hoped it would have.

Lessons from the release

Besides the features, the 8.4 release has been important to the PostgreSQL team in its effort to regain control of its development processes. The project has always had an extensive review process, in which veteran developers were counted on for the final review of contributions. However, ever since the 8.3 release, Momjian explained:

To make matters worse, the review process can be daunting, especially to new contributors. Consequently, many were working in private and submitting their code only at the last moment before a freeze. To reduce these problems, PostgreSQL implemented a series of what it calls CommitFests — alternate months in which no contributions are allowed, and project members concentrate on reviewing existing contributions. This solution helped reduce the problems, but did not eliminate them.

In addition, Berkus said that CommitFests did not address complaints he had heard about people who would review a patch, only to find that a veteran had reviewed and submitted the patch before they had finished. Another challenge was how to enlist the aid of those who said that they were willing to help review out of public-spiritedness, but did not want the trouble of picking out patches to review by themselves. Solving such issues seemed central to increasing the efficiency of reviewing and in keeping CommitFests from extending beyond their originally allotted time.

Realizing that greater efforts were needed, the core team is now using a wiki to coordinate each CommitFest. In addition, over the last half year or so, Berkus created a team of what he calls "Round-Robin Reviewers" — those who wanted their reviews assigned. He also implemented reviews for routine matters such as patch structure by less experienced contributors. The core team and veterans still have to do the majority of the work, but the process has reduced the work load, and added 25 reviewers to the previous pool of 15.

The changes can still make for a slow process, but Berkus apparently judges them a qualified success, noting that the 8.4 release will likely take several months less than the 8.3 release.

Conclusion

Overall, Momjian seems satisfied with the general direction of PostgreSQL. He even suggested that the project is starting to gain increased recognition.

Momjian speculated that the recession may be driving this increased respect.

Berkus pointed, too, to the increased sponsorship of the program, which now means that more developers are being paid to work on the project full-time, and to the increased number of PostgreSQL conferences and user groups world wide. "Compared to other open source databases, we are still developing very quickly," Berkus said. "We still have developer momentum." As you look over the feature list for 8.4 as well as the features that were dropped, it is hard to disagree. While stopping short of being revolutionary, the new release suggests sustained, steady progress in development.

Comments (9 posted)

ELC/LFCS2009: A tale of two panels

Two kernel panel sessions were held last week in San Francisco, one for each of two conferences sharing facilities—and participants. In both cases, the kernel developers making up the panel were asked about various kernel features and developments, both from a historical and future perspective, but each had its own focus as well. The Embedded Linux Conference (ELC) panel was, unsurprisingly, focused on topics of most interest to the embedded community, while the Linux Foundation Collaboration Summit (LFCS) panel looked at more mainstream kernel concerns.

ELC: Embedded Linux Kernel Features and Developments

Besides the venue, the panel sessions also had another thing in common: LWN Executive Editor Jonathan Corbet, who moderated the LFCS panel and was a member of the ELC version. The ELC panel was moderated by CE Linux Forum (CELF) architecture group chair Tim Bird, while embedded maintainer David Woodhouse and Matt Mackall, developer of the SLOB memory allocator (along with various other kernel tasks), rounded out the panel. Bird asked most of the questions, but the audience also got the opportunity to ask a few too.

One of the themes of the discussion—as well as Woodhouse's earlier keynote—was the convergence of features between so-called "big iron" (servers and mainframes) and embedded devices. Corbet was amused to see "highmem" support recently added for ARM processors, noting that it was a controversial feature at the time it was added for servers; supporting a full 1GB of RAM on a 32-bit processor was once a "big iron" problem. Mackall also pointed to SMP and NUMA support moving into the embedded architectures. But things are not only moving in that direction, Mackall said, there is recognition from the big iron developers that there is value for their systems in some of the embedded features too.

Bird asked the panel about the proliferation of embedded distributions and whether that was a good or bad thing. Woodhouse said "fragmentation doesn't have to be bad"; it's only bad when a distribution doesn't work well with the various upstreams and goes off and does its own "weird things". Multiple distributions are one of the "great strengths of Linux", Corbet said, as it provides playgrounds where folks can experiment with different approaches.

Mackall pointed to a lack of community involvement in the various embedded Linux distributions, noting that the most successful desktop distributions were those with a strong community. In the mobile space, the distributions are "coming from the top down", he said, for any of them to be successful, they need to get community feedback.

The impact and usefulness of new "social networking" sites for Linux developers—like MontaVista's meld and the LF's relaunch of the Linux.com community—was another question Bird put to the panel. Woodhouse didn't really see the need, but "communication is always good". Mackall was concerned that these other services not become a "substitute for talking to the Linux kernel community through its normal channels". Corbet noted that there is value in "small town environments", but there is a risk that they can become inbred. "It rarely leads to good things" when a small community gets headed off in their own direction, he said.

One of the more interesting exchanges centered around the question of what a developer who just has a small amount of time can do to assist the larger community. The discussion spread out from there, though. Woodhouse stated that every developer needed to make sure that what they are working on can go upstream even if their managers "need to be whipped to allow you to do that". But Mackall wanted to "back up a step" and ensure that developers are running Linux on their desktop.

Mackall said that developers should be running Linux at home as well; if they are going to work with Linux, they should "live it". Making it work on a laptop is a good exercise; if it doesn't work, figure out why and fix it. He has seen too many embedded Linux developers with Windows desktops who don't understand Linux well enough to properly develop on it. "They don't have the Linux mindset", he said.

Those thoughts were echoed by Woodhouse as he related an anecdote about some embedded developers who would FTP a file to the Windows box, edit it using Notepad, then FTP it back to the Linux machine. It is not efficient to do things that way, he said. Doing the development on Linux will lead to a better result, Mackall said. Doing everything on a Linux desktop will help that, Mackall pointed out, "you should read your mail on it too".

Towards the end of the hour-long session, Bird asked "have we won?", is embedded Linux unstoppable or "is it possible to lose?". Mackall and Corbet had similar thoughts, worrying about the proliferation of devices running Linux that could not be modified by their users. "We haven't won until I can put my code on my phone", Mackall said. Corbet echoed that: "If we end up populating the world with locked-down Linux systems, then we've lost".

In closing, Bird noted that embedded Linux has made an "awful lot of progress". This is the fifth year for ELC and he has been working on embedded Linux for 17 years, over that time, "things have gotten way better", he said.

LFCS: The Linux Kernel: What's Next

Corbet opened the panel by having the participants introduce themselves to an audience of around 400 people. The panel consisted of X.org project lead Keith Packard of Intel, Andrew Morton the kernel "odd job man" from Google, USB maintainer Greg Kroah-Hartman of Novell, and Ted Ts'o of IBM who is currently on loan to the LF as its CTO. After that, Corbet got started by asking Kroah-Hartman about the -staging tree.

Approximately one-third of the code that was merged as part of the 2.6.30 merge window came in via the -staging tree, which Kroah-Hartman maintains. Corbet said there was a lot of confusion about the tree and asked for an explanation. Basically, it is a collection of drivers that used to live outside of the tree, Kroah-Hartman said, consisting of bad code with bad API usage and other major problems barring their acceptance into the mainline, in other words, "crap". But there is a lot of hardware in use that requires those drivers and the code was not getting improved out of the tree, so moving them gives a centralized location where people can get them and hopefully improve them.

Kroah-Hartman said that there were several drivers that had graduated from -staging and into the mainline, so the process seemed to be working. "If you want to get involved in the kernel, that's a good place to start", he said. He noted that there are TODO files in each driver's directory listing the kinds of changes needed before the driver will be accepted into the mainline.

Corbet mentioned that he had been going to conferences for years hearing about all the great things that were going to be done in the Linux graphics area, but that we had now reached a point where much of it had actually been done. He asked Packard to fill the audience in on what had been done and "why it's cool". Packard described how X.org had "turned the graphics stack upside down" by moving the device configuration out of user space and into the kernel.

By doing that, X becomes just an API for existing applications, and other APIs such as OpenGL or Wayland can be considered, he said. Support for Intel graphics is good, and there is lots of work going on for Radeon (ATI) chipsets, but NVIDIA is "not helping at all". He pointed out that Fedora 11 will be shipping with the Nouveau driver for NVIDIA hardware because it has surpassed the free 'nv' driver in capabilities. He also noted that moving the configuration and initialization into the kernel allows people to experiment with graphics acceleration without spending an inordinate amount of time figuring out how to initialize the hardware.

Next, Corbet asked Ts'o about the status of the ext4 filesystem. Ts'o reported that Fedora and Ubuntu would be shipping it in their next releases that are coming within a few months. He said that the user community was growing and "to be brutally honest, that will sometimes find bugs". He said one goal is to get it into the next round of enterprise distributions. He also noted that ext4 is a temporary solution, based on BSD FFS, which is technology from the 70s. Btrfs, nilfs, and others were where the interesting filesystem development is happening. All of those make it an "exciting time" to be a filesystem developer.

Morton responded to a query about the linux-next tree by saying that it is working out well, overall, as a place for integration and testing. But, he said that he was "a bit disappointed with the uptake it has", especially from a testing perspective. Fewer maintainers are taking advantage of the opportunity to integrate and test using linux-next than he would like to see. It is often the case that when there is a problem that shows up in Linus Torvalds's tree, it is because the code never made into linux-next.

From the audience, ftrace developer Steven Rostedt asked about the pressure to merge new code upstream into the mainline, but that there is major resistance to certain things—he mentioned SystemTap and utrace—being merged. He wanted to know what can be done to resolve that. Morton responded that for device drivers or supporting new architectures, the path is easier, but that the two examples Rostedt gave touch core kernel code. Morton likened the utrace battle to an "incestuous family struggle", but noted that the code needed improvement before it could go in.

One of the reasons that utrace didn't make it into the kernel was a lack of an in-kernel user of the code, Rostedt noted. Morton responded that having an in-kernel user for a feature is a "nice checkbox", because it gives the kernel community a means to test the code. But, Kroah-Hartman pointed out that "changing core kernel code is hard, and it should be". Ts'o also pointed out that several core kernel developers are helping out with utrace, which should significantly smooth its path into the mainline.

That discussion led Corbet to ask about tracing, noting that there were several tracing solutions that were still out of the tree, but that ftrace got new tracers added for each kernel release. Morton would like to "see evidence that people are using them and getting good results". Both he and Ts'o pointed at the lack of documentation for various tracers, saying that adding that and making the tracing more usable would help get more of that code into the mainline kernel.

The recently proposed nftables packet filtering subsystem was raised by Corbet as an example of a place where a user-space interface—the existing iptables—might be supplanted. He asked how that transition could be accomplished. Morton called it a "pretty traumatic transition" that would require a compatible set of tools, with several years of warning along with buy-in from the distributions. That takes three to four years according to Kroah-Hartman. Ts'o called the packet filtering interface more of an administrative interface that didn't have to be kept as stable as others, but that the iptables command does need to be stable.

All of that led Packard to complain about the difficulties of keeping the current user-space interface for X servers while moving modesetting into the kernel. According to Packard, there are exactly two users of the interface, both of which are under his control, so why does he need to provide backward compatibility? Ts'o said that the problem would be for distribution users who wanted to upgrade their kernel. Because the distribution might use an old X server, that interface—which Packard described as "open /dev/mem"—needs to be maintained. Kernel hackers want as much testing of new kernels as they can get, so any barrier to that testing is problematic.

At the end of the session, LF Executive Director Jim Zemlin announced the first ever LF "Unsung Hero" award, which he then presented to Morton. He explained that Morton is an avid car racer, so the LF arranged for him to have a day at the track as a reward. It was no surprise that there was much applause for Morton—one of the few people actually able to follow the linux-kernel mailing list. He also reviews an incredible amount of the code that ends up in the kernel.

These sessions provide an interesting view into the thinking of the members of the panel—one not easily derived from just keeping up with the technical side of Linux development via the LWN Kernel page or even by sifting through linux-kernel. They also give attendees a look at what's coming in the future that can be hard to discern, though Corbet's Linux Weather Forecast is helpful there. In the end analysis, though, the biggest benefit may just be putting kernel developers and users together in a fairly informal setting so that both sides get a better feel for the other. Faces and personalities don't necessarily jump out of the normal communication channels, so panel sessions like those that went on in San Francisco are useful well beyond their technical content.

Comments (11 posted)

Book Review: Pragmatic Version Control Using Git

Given the ubiquity of Git as a version control system throughout the free software community, one would expect there to be more books about it. So far, that is not the case—though there are indications that is changing—so Travis Swicegood's Pragmatic Version Control Using Git is welcome for those trying to come up to speed on Git. Overall, the book provides a nice starting point, though there are some rough spots.

Like any book covering a free software package, this one begins with some important basics: where to get and how to install the tool. For Linux users, this guide is probably unnecessary as Git is packaged for most distributions these days—Mac OS X or Windows users may find more of interest. The discussion of Git configuration, along with the reminder to set the user.name and user.email parameters before doing any commits, something that I regularly forget when setting up a new machine, is quite useful for all.

Unlike some other authors, though, Swicegood takes the time to give a bit of the flavor of Git through a discussion of its concepts—along with some indication of why one might want to use it—before descending into the much more boring installation guide. He takes a "30,000 foot" view of the tool and, with no command syntax or specific usage details, spells out what Git can do.

One of the primary problems that any text on a version control system (VCS) must overcome is the need to give "real-world" examples while still keeping the book to a reasonable size. Swicegood does a good job here, by following one example repository throughout the text. One could quibble with the scope of some of his examples, but, by and large, they give a good idea of how things work. In some ways, the simplicity of those examples appears to encourage curious readers to do some experimentation. That is, after all, a pretty good way to learn how to use a tool.

The book is broken up into three main sections (plus an Appendix with a reference and some pointers to more information), but the meat of the text is in Section II, "Everyday Git". For whatever reason, the last chapter of the first section covers setting up local repositories as well as cloning remote repositories. That might make sense, but it is rather puzzling that it starts talking about things like git rebase, branches, and doing releases here. Much of that is covered in further detail later and it doesn't seem to belong.

In Section II, the book does an excellent job of covering how to use Git on a day-to-day basis. I have found myself referring to it several times since reading it to remind myself of the syntax of a command—or the name of a command itself. The sequence is logical, starting with adding and committing files, moving through branch creation and management as well as examining and working with history in Git, and completing the core with a look at remote repositories. Two additional chapters covered somewhat more advanced—or just less often used—features such as organizing the repository and working with multiple remote projects as well as things like compacting a repository and working with the reflog.

Swicegood uses the term "staging" for what is commonly referred, at least in other Git documentation, as the "index". Some readers, especially if they are already well-versed in Git, may find this a bit confusing, but I found that it made sense and, in some ways, simplified the concept. In any case, it seems clear that is how Swicegood envisions the Git index, so passing it along to his readers is a nice touch.

There is no specific mention of the Git version covered by the book—though some early examples mention 1.6.0.2—which is a rather large oversight. Git development moves rapidly, so some of what Swicegood talks about could well be out-of-date. New Git features, such as the unmentioned git stash, were left out, but it isn't clear whether that was done on purpose or because they were added after the book was completed. Most of what is covered should be unaffected, though, as the basic operation of the tool is fairly stable.

The third, thinnest and weakest section is "Administration", which covers migrating to Git and running Git servers. Both chapters seem to suffer from a lack of breadth. In the migration chapter, nothing but CVS or Subversion are considered, and tools like tailor are not even mentioned.

Two things about Swicegood's choices of Git features stood out in a negative way. He seems overly enamored of git rebase, which certainly has its place, but it has some drawbacks that he doesn't fully caution against. His solution for how to create a repository for others to use was somewhat unsatisfying; Git itself can be configured to support such things. Instead, Swicegood reaches for Gitosis, a Python tool for managing remote git repositories. The project seems to have no web page (other than a gitweb page) and one must install it by cloning its repository. Given that there is no mention of how to "manually" set up a Git server, it all seems a bit strange.

There are a handful of less-substantive complaints I could make as well: a throwaway George Santayana quote on the history chapter was a bit annoying, an embarrassing "EMCAScript" typo in one of the examples stood out, as did a few other minor flaws. Swicegood complains frequently about having to truncate or otherwise modify the output of commands to fit on the page, which seems a bit silly. Either fix the problem somehow in the production process or ignore it to the extent possible; involving the reader in the pain of the typographic process seems unnecessary. But these are nits.

While I had some complaints—it is a rare book indeed where I don't—Pragmatic Version Control Using Git has certainly found a spot for itself on my shelf. It especially shines as a quick reference to commands needed daily or nearly so. It will also provide a good starting point for those who wish to learn Git from scratch. Once other Git books come out, it will be interesting to see which end up on my shelf and which are shuffled off to long-term storage. In the end, that is the best test for a good book.

Comments (13 posted)