PostgreSQL 8.4 Beta: "We've got momentum"

The PostgreSQL project has released the beta of its 8.4 version, with the final release expected in late June or July. Like previous PostgreSQL releases, 8.4 features dozens of enhancements throughout the code, but Bruce Momjian, a member of PostgreSQL's core team since the project was founded in 1996, described it as a "more surgical release" with improvements tending to cluster in select areas. In particular, that area seems to be administrative features. However, almost as interesting as the features in the beta are the ones that failed to meet the cut this time, and the project's struggle to control the release process as it undergoes major growth.

Momjian explained the release's more targeted approach as the result of the project's advanced state. "You see a real consolidation in this release," he said, "and I think that's because you're seeing a much more complete feature set. You're seeing a real maturity of the code base, which is kind of surprising, because, if you looked a few years ago, you'd see changes all over the map."

At the same time, Momjian characterized the PostgreSQL code as being constantly revised, so that the project is unlikely to need any time soon the kind of major revision that KDE underwent last year.

Nor is the project hesitant about altering behavior or deprecating legacy code in the name of what Momjian called "high standards and the promise of reliability."

New features

PostgreSQL releases tend to be one to two years apart, with far too many features to mention in any detail. Many of these features are highlighted in a PDF presentation by Momjian entitled, "Upcoming PostgreSQL Performance Features [PDF]", including Column Level Permissions and Per-Database Locales. However, if you ask active contributors what the major enhancements are in the 8.4 release, their answers emphasize automated administrative features.

Selena Deckelmann, user groups leader in the project, emphasized changes to Free Space Map, which maps unused space in a database. In previous releases, PostgreSQL could only detect newly freed space by an administrator manually running the Vacuum utility. By contrast, in the 8.4 release, freed space is re-mapped automatically, saving both time and effort.

Deckelmann also called attention to a new feature called Visibility Map. While in previous releases, Vacuum had to re-map all rows in a table, regardless of whether they had changed, Visibility Map improves performance by allowing the utility to skip rows that have not changed.

Yet another automated feature mentioned by Deckelmann is Auto Explain, which captures the explain plan for a query — that is, how the query finds results — and saves the results, information that can be used to improve system load and the efficiency of queries, and, ultimately, to reduce the costs of a database operation. "It's something that a lot of admins end up doing anyway," Deckelman observed, "They write a script that looks for long-running queries, and then they go in and manually figure with each one what's going on. It's kind of a neat feature that came from the Japanese PostgreSQL team."

Josh Berkus, another core team member, noted that Oracle's Statspack, a set of tools to generate statistics useful to database administrators, inspired pgstat. As Berkus explained the situation, users migrating from Oracle expect to see comparable tools in PostgreSQL, and Statspack:

Berkus noted, though, that pgstat will not be stable enough to be installed by default in 8.4. Instead, it will be shipped in postgresql-contrib, the repository for tools that, for one reason or the other, are not part of the regular installation. Some tools in contrib may be too specialized for most installations, or illegal to ship under American restrictions on the exporting of cryptographic tools, while others, like pgstat, are still in development.

According to Deckelmann, statistics in PostgreSQL receive another boost in the 8.4 release with increased control over them. To reduce system overhead, statistics are no longer automatically collected and written to a file, operations that can have significant system overhead if done regularly. Instead, statistics can now be configured to run from a RAM disk to improve performance.

However, if the latest release has a single outstanding feature, it is parallel restore. As the name suggests, parallel restore allows admins to restore a database with multiple processes, rather than a single thread. Berkus, who runs his own PostgreSQL consulting business, said, "[Parallel restore] is the feature i've been making the most use of. I've been using it in beta already — pretty heavily. Because if you have 300 gigabyte databases, upgrading them single-threaded is lethally slow. It's a real issue."

He went on to describe parallel restore as a particularly difficult feature to implement, all the more so because it had to accommodate past changes in the PostgreSQL file format. While employed as a supervisor at Sun Microsystems, Berkus said, he had two employees working on a similar feature for a year and a half "without coming up with more than a rough prototype." By contrast, PostgreSQL developed its version in three weeks, followed by three months of debugging.

Features for the next release

As mentioned earlier on LWN, PostgreSQL's core team had hopes of other major features being in the 8.4 release, particularly Hot Standby and SE-PostgreSQL.

Hot Standby is a feature that is a major step in improving PostgreSQL's replication. Replication is an area in which PostgreSQL lags behind MySQL; it has been identified by the core team as one of the major priorities for the project, according to Berkus. PostgreSQL does have some replication, Berkus said, but it is needlessly complicated and "not for someone who can't make a large time investment in finding out how it works. And that's a real problem for the sort of average-case web developer that has two servers and just wants to make sure that PostgreSQL is mirrored."

Hot Standby is an important improvement in replication that allows administrators to run queries on a database that is being recovered from an archive. The module allows replication of the database logs in order to create read-only duplicates of the database. Unfortunately, funding for development of the feature only came through in August 2008, leaving only a few months before the November code freeze. Nor could Hot Standby be finalized in the extended testing period that followed.

SE-PostgreSQL is an even greater innovation, which will add the SE Linux security model, making PostgreSQL the first database to use the same security model as many distributions, such as Fedora. The problem is, Berkus explained

Having failed to be ready for the 8.4 release, both Hot Standby and SE-PostgreSQL are high on the priority list for the next release. In fact, according to Momjian, 8.4 is partly designed to ease their later implementation. Momjian suggested that, given the maturity of PostgreSQL's code, these might be simply the first of many new features that are too large to be implemented in a single release. Meanwhile, he planned on promoting SE-PostgreSQL in the release notes in the hopes of encouraging interest in it and perhaps attracting a few developers with SE Linux experience.

"To continue to grow and continue to have the sort of reputation we have, you have to make some hard decisions," Momjian said. Like Berkus and Deckelmann, Momjian clearly regretted the necessity for omitting these features. However, having faced long delays because of the refusal to jettison unfinished features before — notably in the implementation of the Windows port in version 7.4 and more generally in the sheer number of features in version 8.3 — the PostgreSQL core team has learned the hard way not to insist that a release include all the features they hoped it would have.

Lessons from the release

Besides the features, the 8.4 release has been important to the PostgreSQL team in its effort to regain control of its development processes. The project has always had an extensive review process, in which veteran developers were counted on for the final review of contributions. However, ever since the 8.3 release, Momjian explained:

To make matters worse, the review process can be daunting, especially to new contributors. Consequently, many were working in private and submitting their code only at the last moment before a freeze. To reduce these problems, PostgreSQL implemented a series of what it calls CommitFests — alternate months in which no contributions are allowed, and project members concentrate on reviewing existing contributions. This solution helped reduce the problems, but did not eliminate them.

In addition, Berkus said that CommitFests did not address complaints he had heard about people who would review a patch, only to find that a veteran had reviewed and submitted the patch before they had finished. Another challenge was how to enlist the aid of those who said that they were willing to help review out of public-spiritedness, but did not want the trouble of picking out patches to review by themselves. Solving such issues seemed central to increasing the efficiency of reviewing and in keeping CommitFests from extending beyond their originally allotted time.

Realizing that greater efforts were needed, the core team is now using a wiki to coordinate each CommitFest. In addition, over the last half year or so, Berkus created a team of what he calls "Round-Robin Reviewers" — those who wanted their reviews assigned. He also implemented reviews for routine matters such as patch structure by less experienced contributors. The core team and veterans still have to do the majority of the work, but the process has reduced the work load, and added 25 reviewers to the previous pool of 15.

The changes can still make for a slow process, but Berkus apparently judges them a qualified success, noting that the 8.4 release will likely take several months less than the 8.3 release.

Conclusion

Overall, Momjian seems satisfied with the general direction of PostgreSQL. He even suggested that the project is starting to gain increased recognition.

Momjian speculated that the recession may be driving this increased respect.

Berkus pointed, too, to the increased sponsorship of the program, which now means that more developers are being paid to work on the project full-time, and to the increased number of PostgreSQL conferences and user groups world wide. "Compared to other open source databases, we are still developing very quickly," Berkus said. "We still have developer momentum." As you look over the feature list for 8.4 as well as the features that were dropped, it is hard to disagree. While stopping short of being revolutionary, the new release suggests sustained, steady progress in development.