Vulnerability handling in the PostgreSQL project

On April 4th, 2013, the PostgreSQL project announced a security vulnerability (CVE-2013-1899) and resulting patch for one of the worst security holes in project history. According to the project web page, "this is the first security issue of this magnitude since 2006." Given the broad usage of PostgreSQL, the update caused quite a stir. Even more controversy was created by Heroku's early deployment of the fix. The PostgreSQL project's handling of this security release raises some important questions about how open source projects should handle this kind of release.

Discovery

On March 12th, Mitsumasa Kondo and Kyotaro Horiguchi of NTT Open Source Software Center (of Japan) reported an issue to the PostgreSQL security mailing list. They had uncovered the issue while doing security testing of PostgreSQL, which is one of the ways that NTT has assisted the PostgreSQL project over the last several years. They immediately noted the potential of this vulnerability to destroy data using nothing more than a psql client connection.

In general, this is typical of how security vulnerabilities in PostgreSQL are discovered. Most are found by existing code contributors. Over the last few years, an increasing number of security vulnerabilities have been discovered by professional security researchers from organizations such as NTT, Secunia, the Japanese government, and the Red Hat security team. This has allowed the project to find, and patch, vulnerabilities which have gone unnoticed and unremarked for years.

PostgreSQL has yet to have a major security exploit or worm known to be "in the wild" prior to a fix being available for a vulnerability. With the increasing popularity of the database system, though, it's only a matter of time before the "black hats" outrace the contributors.

Vulnerability and exploits

The core of the vulnerability is a command-line switch designed for internal use of the backend server that was not supposed to be exposed to users through the API. This switch is accessed by passing a "-r" before the database name in an initial, unauthenticated, connection to the database server, which causes PostgreSQL to instead pipe the session output (generally an error message) directly to a file with the same name as the supplied database. No database of that name needs to actually exist, so the error output can be piped to arbitrary database server files.

The vulnerability was inadvertently added by PostgreSQL lead hacker Tom Lane in the course of a general refactoring of client connection code that generally made it faster and more maintainable. Once the oversight was pointed out, members of the security team were able to quickly and easily patch the server backend code to have it ignore the unsafe switch.

The vulnerability affects PostgreSQL versions 9.0, 9.1, and 9.2. Users on version 8.4 are safe from this particular vulnerability. Users of version 8.3 and earlier are also safe, but those versions are past end-of-life (EOL), so they are missing other security fixes since they are no longer being updated by the PostgreSQL community — though distributors and other third parties may still be backporting security fixes.

First, any attacker who can reach an open PostgreSQL port (usually port 5432), can, without a valid login, cause the PostgreSQL server to shut down and refuse to restart by using the "-r" switch to append error messages and other garbage to vital database server files. This is what the project calls a "persistent denial of service" because the administrator will have to either hand-clean the files, or restore from backup, to restart the server. This exploit has a certain "time bomb" nature to it as well, since generally the effects of adding garbage to files won't be noticed until later, such as when the server is restarted next.

Additional exploits become possible if the attacker has some type of valid login. For example, if a user has a legitimate login which matches a database name (for example, user "django-prod", database "django-prod"), then they can use the "-c" switch to change a single server configuration variable with the privileges of the database superuser for the duration of their database session. This privilege escalation becomes useful to an attacker mainly if they also have other types of access to the server. Particularly, if the attacker also has filesystem access, even to a "tmp" directory, they could use the superuser configuration exploit to dynamically load a C library, thus permitting the execution of arbitrary C code.

Users whose databases are protected from untrusted networks are generally safe from these exploits. So in addition to patching PostgreSQL, effective use of network security and firewalls greatly reduces the impact of the vulnerability. Additionally, tight control of login credentials and use of advanced security options like SELinux limit the amount of damage an attacker can do. It is important to note that settings in pg_hba.conf, the Postgres access control file, are not effective protection against the denial-of-service vulnerability, although modifying postgresql.conf to not listen on external IP addresses is.

Impact

Several years ago, PostgreSQL was primarily used on back-office servers that were not connected directly to the Internet, so a vulnerability like this one would not have been such a major issue. Today, however, PostgreSQL is run on thousands of "cloud" servers with public IP addresses, most notably ones run by application hosting companies such as Heroku, EngineYard, and VMware Cloud Foundry. Heroku is the largest of these "Database As A Service" (DBAAS) companies; it does not disclose user numbers, but is assumed to have hundreds of thousands of users based on the level of adoption shown when it was acquired by Salesforce.com in 2010. Individual web hosts are running with PostgreSQL exposed as well. This scan from Shodan shows over 300,000 PostgreSQL servers exposed to the Internet, 40% of them at Polish megahost Home.PL.

As a result, this security vulnerability risks large-scale denial-of-service to thousands of web applications, and worse problems if combined with a second security weakness elsewhere in users' applications. That made it imperative to get users to update their PostgreSQL installations as soon as the vulnerability was disclosed.

Announcement process

The last time the PostgreSQL project had a remotely exploitable vulnerability of this magnitude was the "backslash escape" issue in 2006 (CVE-2006-2314). As a result, the vast majority of current PostgreSQL users have no experience with major security holes in the DBMS. The PostgreSQL team was a bit "rusty" when it comes to this kind of security issue as well. Simply put, the last time the project had a issue this serious, there were a lot fewer PostgreSQL users.

PostgreSQL releases its own binary packages for several platforms, in particular Red Hat variants, Debian, Ubuntu, and Windows. This is done by a large volunteer team of packagers who generally get access to the new code a few days before the release date in order to create binaries. Recently, the project has begun adding the major DBAAS platforms to the packagers list, since those companies do their own builds and distribute to a large, public user base.

The first notice which the public received on the bug was on March 26th when Tom Lane notified the PostgreSQL developer list that an update was coming on April 4th which patches "a seriously nasty security problem in recent releases of Postgres." This was followed by a notice to the general public on the 29th, and closing the main PostgreSQL git server to the public on the 31st, which was also the day the packagers started working with the new code. The final release came out April 4th at 1500 UTC.

In general, this process went as planned. "This security update has been coordinated and handled in an exemplary manner, everyone got told and prepared well in advance," said Martin Pitt of Canonical.

As I will document below, there are arguments as to whether the procedure should be revised, but messages and updates went out as intended — except for one thing.

The Heroku issue

The last time the PostgreSQL team executed this high-risk security release procedure, DBAAS was barely a gleam in a technology forecaster's eye. Thus the release procedure didn't take into account how building for DBAAS is different from rolling RPMs.

On April 1st, Heroku posted a status update telling users that they would have a unscheduled sixty-second outage to update customer databases between April 1st and 3rd. Users and reporters easily put this together with the PostgreSQL project announcement to figure out that Heroku was deploying the security update early, ahead of its official availability. This caused some significant unrest among users who felt that their security was being ignored in favor of large vendors.

Heroku had been permitted to update early, with full knowledge of the PostgreSQL security team, because of their large size and high level of vulnerability. Developers also felt that Heroku would be able to supply feedback about the security update, and report on any incompatibilities or regressions introduced by the code patches. However, they had not anticipated that the nature of DBAAS would make this early deployment effectively public.

Other DBAAS vendors, such as EngineYard, EnterpriseDB, Gandi.net, and VMware, were also given information and code in order to have builds ready early, but did not deploy until April 4th.

Security releases, disclosure, and open source

This entire process and some of the complaints about it have raised a number of questions in the PostgreSQL community and other communities on how to handle high-exposure security vulnerabilities. The practices, conventions, and values of proprietary software don't necessarily work for open source projects, so there are a number of questions our communities need to resolve. Since the answers to many of these questions are far from clear-cut, we took them to a group of open source project leaders to discuss policy and procedures for security releases.

Jacob Kaplan-Moss of the Django Project felt that the public warning about an imminent security release was "a mistake" which "fails in three ways." He explained, "First, it causes some undue angst. We all know intellectually that all software has bugs, including security issues, but that's different from knowing that there's something concretely wrong with the software you're using right now ... Second, it calls attention to the fact that some people get special treatment (Heroku in your case). Finally, I don't actually think the extra pre-notice helps. Knowing that this was coming on Thursday didn't really change things; we would have dropped everything anyway."

Ben Laurie of the Apache Foundation and the OpenSSL project pointed out the danger, "It's amazing what people work out given minimal clues."

Yet in this particular case, the forewarning of the upcoming security release generated extra press coverage, which arguably caused many more users to be aware that there was an update than otherwise would have been.

Another debatable point is differentiating major and minor security releases. The PostgreSQL project has been in the habit of doing this, as has the Mozilla project. But Karl Fogel, author of Producing Open Source Software, took exception to it, saying, "I think it's best to handle all security releases using basically the same procedure." Kaplan-Moss agreed with the viewpoint, pointing out, "Risk is the product of two factors: ease of exploit, and impact of exploit. As maintainers, we can do a good job estimating the first factor, but the second is going to differ based on use case."

Dave Page of the PostgreSQL Core Team, however, argued practicality: "We can't treat every security issue the same way. Consider the effort and number of people who had to drop everything this time ... We just can't expect that level of support for every minor issue that crops up, yet we need to expect it for a category A like this one."

The biggest area where open source projects differ in handling their security update notifications has to do with the question of advance notice to large, important, or especially vulnerable users. The PostgreSQL Project's approach of informing only packagers of public builds is quite common, and the same one (sometimes) used by Kernel.org. Other projects do it differently, such as the Django Project, which has a private list and formal policy for early notification users.

"I think large services whose operators you trust can get advance notice. But you control that list, and you never make it public -- otherwise people start asking 'Why am I not on the list when so-and-so is?'," Fogel suggested. "There may be large sites where you can prevent a lot of damage if you trust the admins to handle the news discreetly and you handle it discreetly yourselves.

On the other hand, Laurie didn't find this to be a good idea at all. He said, "Everyone thinks they're special. They're all wrong. The public have as much, or more, right to know than special interests."

Within the PostgreSQL community, the issue of advance notice for critical users is likely to be debated for some time. PostgreSQL is currently in the process of revising its security release policy based on lessons learned from this release.

Release

Regardless of people's perspectives on the PostgreSQL security release process, the updates were released on April 4th. This included versions 9.2.4, 9.1.9, 9.0.13 and 8.4.17, updating all maintained branches. Because of the earlier controversy and attention, this update release received considerably more press coverage than a standard PostgreSQL update.

Of course, in addition to the security issue discussed above, the update also fixes a number of other minor PostgreSQL issues, including four other security issues. CVE-2013-1900 is an issue weakening key security in the optional pgcrypto extension. CVE-2013-1901 is a permissions bug which allows a regular user to interfere with concurrently executing database backups. The release also fixed two security issues with the graphical installers for Linux and Mac OS X, CVE-2013-1903 and CVE-2013-1902. The update also includes a number of non-security bug fixes, including two issues which can cause replication failover to not work properly.

Within two hours of the release announcement, Metasploit had posted a test for differentiating patched from unpatched servers. Other scanner projects have followed suit, so it should be possible for administrators to be sure they've patched their systems.

Schemaverse compromised

Within a day of the release announcement, we had our first actual compromise of a public system, courtesy of Schemaverse, which is a space-combat database-security training game invented by Josh McDougall for DefCon. Since successfully attacking the database is an acceptable way to win the game in Schemaverse, McDougall decided to hold back updating:

    -rw-------. 1 postgres postgres   535 Apr  5 05:33 SECURITY_RISK_PLEASE_UPGRADE_TO_9.2.4_NOW

So, a white hat was behind it, but it was a break-in nonetheless. And it shows the danger of the vulnerability.

Conclusion

As of April 8th, no PostgreSQL users have reported being compromised as a result of the vulnerability, and the PostgreSQL developers hope that they made enough noise to get the vast majority of users to update before the exploits become common scripts. They also hope to not need to practice their security release procedure again for some time, but are revising their security policy and preparing for the worst.

Certainly, we can expect, in any given week, a major security vulnerability from some key open source project. Our industry and communities still need to improve and refine our ways of handling security vulnerabilities, patch releases, and notifications. Perhaps we can do better than the proprietary software industry with regard to sharing information, ideas, and best practices.

In the meantime, patch your PostgreSQL servers!

[ Josh Berkus is a member of the PostgreSQL Core Team. ]