Responsible disclosure in open source: The crypt() vulnerability
At the end of May, five separate open source projects released patches to close the same security hole in their software. This coordinated release and vulnerability handling is a demonstration that "responsible disclosure" can work, especially in open source.
Responsible disclosure is the practice of security researchers discovering a vulnerability and contacting the software vendor to give them a reasonable time to fix it before the vulnerability is published. It contrasts with the policy of "full disclosure" in which security people publish the full details of any vulnerability immediately, in order to get information to the public as quickly as possible. Mostly, these two terms have shown up in the media as part of controversies, or even legal battles, which pit security researchers against software companies and each other.
While the inflammatory confrontations gain most of the news headlines, it doesn't have to be that way. In fact, among open source projects, it isn't that way most of the time. The recent multi-product Crypt-DES vulnerability patch shows that responsible disclosure can and does work well in the open source world.
The Crypt-DES vulnerability
Robin Xu and Joseph Bonneau at Cambridge University had been investigating how non-ASCII passwords were handled by various systems for more than a year. Bonneau started on this research because of the massive Gawker security breach in 2010. In the course of investigating that, his team uncovered several issues with non-ASCII passwords in commonly used software. While the one at Gawker was quickly addressed — to some degree — he and Xu began a research project on the insecurities introduced by applying algorithms designed for ASCII to Unicode text.
The version of crypt() using the DES algorithm (hereafter crypt-DES) is a simple irreversible hash designed to prevent storing passwords in plain text. Introduced in old Unix days, it had the advantages of easy implementation, portability between systems and programming languages, computational speed, and is hard enough to crack that dictionary attacks and social engineering were generally easier ways to grab passwords. Given the age and limited computational "strength" of crypt-DES, however, this is no longer true; brute-force computation of crypt() passwords is easily done. Programmers are encouraged to use more modern hashing and encryption algorithms, such as SHA1 and Blowfish. The "extended" DES version was introduced in BSDi in the early 1990's, improving the algorithm to have a larger "salt", more rounds of encryption, and also to support passwords longer than eight characters by "folding" them down to eight 7-bit characters using a first round of DES hashing.
The last improvement is the problem which causes the crypt() vulnerability. Crypt-DES was designed for ASCII characters, and programmers who upgraded systems to support Unicode didn't really check to see how crypt-DES would work with Unicode passwords, since by that point crypt-DES was no longer mainstream. As it turns out, the folding is broken; the algorithm regards characters containing the byte 0x80 as a "stop" character and disregards any parts of the password after that byte. In many Unicode encodings, characters — such as the common character À — can contain a 0x80 byte, causing all characters after that one to be disregarded. This means if your password was Àlbanez60, then crypt-DES would match it with any password beginning with À.
This is also a good illustration of how security is a process and not an end result. Crypt-DES was an adequately secure password hashing approach well into the mid 1990's, which is why people stopped testing it. It was the introduction of popular Unicode-compliant versions of programming languages and databases which has made it less secure than anyone realized.
Contacting projects
Having found this issue, Xu, Bonneau, and other Cambridge graduate students spent several weeks examining some common software and found that the defective version of Crypt-DES was still shipping with several open source software packages, among them PostgreSQL and FreeBSD. Having found the vulnerability, they emailed the private security mailing lists for the affected projects.
The PostgreSQL security team received this email on April 24th:
The Cambridge team had previously contacted a few other projects, including FreeBSD. The FreeBSD and PostgreSQL projects had to decide what to do about patching the vulnerability. For anyone affected by it, an updated version of crypt-DES would require that all affected passwords (ones containing the 0x80 byte) be regenerated. While neither PostgreSQL nor FreeBSD used crypt-DES for system authentication, both supply functions which are used to hash application passwords. Because of this disruption to some users' applications, it couldn't be done casually.
The FreeBSD security team contacted the OpenBSD, NetBSD and DragonflyBSD projects. Rubin Xu's research indicated that PHP's crypt() also had the faulty algorithm, and had attempted to contact the PHP security team without success. Members of FreeBSD contacted them and brought them into the discussion. NetBSD turned out not to be vulnerable.
Coordinating a release
Among the affected projects, this vulnerability was considered moderate in severity, since it only affected a minority of users of each project. Not only did users need to build applications using crypt() with DES, despite other, more modern hashing options being available, but the password vulnerability only affects passwords with Unicode characters including the 0x80 byte. Specifically, the vulnerability was limited to:
- BSD users who used the included crypt() library with DES (the default) to support their applications.
- PostgreSQL users who installed the optional pgCrypto extension and used its crypt() function with the default DES encryption.
- PHP users who used the PHP crypt() function with DES on platforms without native encryption support, most notably Windows.
On the other hand, the vulnerability affects passwords, which means it's specifically a hole in code people have written to secure their systems. That raised this vulnerability from obscure to moderately serious. So FreeBSD filed for a Common Vulnerability and Exposure number (CVE), and the projects began trying to coordinate a release.
From the perspective of the projects, once one project announced a release and CVE-2012-2143 became public, it wouldn't take much cleverness for a even a newbie black hat to figure out the vulnerability in other products. That meant coordinating a release date among five different projects. In surprisingly short order, they reached a compromise date of May 30th, which was the earliest reasonable release date. On that date:
- FreeBSD released a security update.
- OpenBSD updated its hotfixes repository.
- DragonflyBSD released a security update.
- PostgreSQL released a source code patch for pgCrypto, with a full update release to follow on June 4.
- PHP released a new Release Candidate with the fix, although it did not publicize the issue.
The entire timeline from the discovery of the vulnerability to deploying fixes for multiple projects took about three months. The majority of this time (about seven weeks) was taken up by the researchers finding and contacting affected projects. If there's room for improvement in the process of responsible discovery, it's that finding affected projects or products and contacting their security teams is slow and time-consuming. The remaining five weeks is only two weeks longer than the minimum time for most packaged projects to do a release at all, due to packaging, scheduling, testing, and coordination requirements. One could easily argue that immediate disclosure would have gotten the news about the vulnerability out much sooner, but it's not clear how that would have benefited affected users until fixes for their software were available.
In relatively short order, five major open source projects were updated to close it. Nobody was threatened, no single project's users or developers were singled out, the security researchers were thanked for their work, and nobody needed to spend more than a few hours of their time getting the fixes made and released. At least from the perspective of software maintainers and regular users, this episode looks like a success.
Why it worked this time
This whole episode had two important factors to make it a relative success: the security researchers were university staff unmotivated by fame or profit, and the open source projects are community non-profits lacking incentives to defer or deny patching security holes. This meant that everyone involved was motivated to fix the vulnerability in the fastest, most responsible way possible.
This is by no means exceptional in the open source world. On the PostgreSQL project today, as with many other open source projects, companies and academic researchers regularly practice responsible disclosure, letting the developers know about a security issue in a reasonable time to fix it. If anything, this is the rule in the non-profit open source world. So why does disclosure cause friction, user exposure, blog wars, and legal threats in the for-profit world?
Well, when you look at failures of security disclosure, the overwhelming trend is bad faith. Software companies don't want to do expensive releases and get bad press for security issues, so they put off security researchers forever, or even threaten them. Security people or their employers want fame and attention so they publicize security holes as widely as possible without verification, or giving the vendor a chance to patch issues. Or worse, researchers, companies and agencies participate in a marketplace of secret security exploits.
So, while responsible disclosure can and does work in the non-profit open source world, it's not clear how to transfer these practices to the for-profit world, or even if it's possible to do so. Maybe the answer is simply to use more open source software.
[ Note that MITRE has not updated their CVE database. As such, the CVE link for the exploit will still show as "pending". ]
| Index entries for this article | |
|---|---|
| Security | Bug reporting |
| Security | Passwords |
| GuestArticles | Berkus, Josh |
Posted Jun 6, 2012 20:48 UTC (Wed)
by ballombe (subscriber, #9523)
[Link] (12 responses)
Maybe "coordinated disclosure" ?
Posted Jun 6, 2012 22:40 UTC (Wed)
by jberkus (guest, #55561)
[Link] (10 responses)
As a writer, though, I have to use the terms that people are familiar with.
Posted Jun 7, 2012 5:54 UTC (Thu)
by JoeBuck (subscriber, #2330)
[Link] (9 responses)
Posted Jun 7, 2012 9:41 UTC (Thu)
by dgm (subscriber, #49227)
[Link] (8 responses)
Responsible disclosure = Coordinated disclosure
Posted Jun 7, 2012 10:33 UTC (Thu)
by Funcan (subscriber, #44209)
[Link] (1 responses)
In the world of closed source security research, it might be better to say full disclosure == disclosure at all...
Posted Jun 7, 2012 11:51 UTC (Thu)
by dgm (subscriber, #49227)
[Link]
Posted Jun 7, 2012 14:51 UTC (Thu)
by randomguy3 (subscriber, #71063)
[Link] (5 responses)
Posted Jun 7, 2012 15:26 UTC (Thu)
by dgm (subscriber, #49227)
[Link] (3 responses)
And early would be compared to the release of the patch, of course.
Posted Jun 7, 2012 17:04 UTC (Thu)
by nybble41 (subscriber, #55106)
[Link] (2 responses)
Posted Jun 7, 2012 17:15 UTC (Thu)
by dgm (subscriber, #49227)
[Link] (1 responses)
Posted Jun 7, 2012 18:06 UTC (Thu)
by k8to (guest, #15413)
[Link]
Posted Jun 14, 2012 6:21 UTC (Thu)
by fest3er (guest, #60379)
[Link]
Language can be, and often is, problematic. The task is to find the fewest words that adequately communicate the desired idea. Perhaps three words would better describe the process: 'responsibly coordinated disclosure'.
Posted Jun 7, 2012 9:15 UTC (Thu)
by epa (subscriber, #39769)
[Link]
Posted Jun 7, 2012 17:27 UTC (Thu)
by joey (guest, #328)
[Link] (1 responses)
Posted Jun 7, 2012 21:14 UTC (Thu)
by DennisJ (subscriber, #14700)
[Link]
Posted Jun 7, 2012 20:56 UTC (Thu)
by intgr (subscriber, #39733)
[Link] (9 responses)
Seriously? This is bad advice! While crypt-DES is indeed old and should be deprecated, it is a proper salted password hashing function.
SHA-1 is NOT a replacement for crypt-DES at all -- it's a hash designed to be fast and it defines no salting mechanism. Thus, all passwords in a compromised database can be checked in parallel and cracking can be very fast using precomputed (publicly available) rainbow tables; yet it's faster than crypt-DES even in the brute force case. Worse, it's vulnerable to collision attacks, which makes it unsuitable for some applications, so it shouldn't be recommended at all for anything new.
PBKDF2 is *the* standard for password hashing. It defines a variable iteration count and salting, with some other nice properties. Or if you're feeling adventurous, scrypt is interesting.
And Blowfish... It's even further from a crypt-DES replacement. It has no known vulnerabilities, but even its designers recommend using AES instead.
Posted Jun 8, 2012 8:37 UTC (Fri)
by tialaramex (subscriber, #21167)
[Link] (2 responses)
Probably this is because of the same error that Poul-Henning Kamp has made in the last few days here: http://phk.freebsd.dk/sagas/md5crypt_eol.html, giving advice with the assumption that the people receiving it are smart, well-motivated people who know the topic.
A handful of people will muse on what PHK wrote and then write robust, heavily pessimised and salted hashing algorithms, custom for their application, properly review and test them and put them into production. But far more people will take this as an opportunity to either:
(a) continue using crude variants of MD5(password) using e.g. a small per-application salt under the belief that a little obscurity buys them more than the security of an established algorithm such as PHK-MD5 or even the DES crypt.
(b) argue that these crypto programmers don't understand real world problems such as the desire to re-use passwords across applications and thus ought to be ignored by "real" programmers who need to get that PHP web forum working to keep the client happy.
Overall I think PHK's article was mistimed, not least because the numbers he gives to "scare you straight" seem to be exaggerated or based on some additional assumptions he didn't provide.
Posted Jun 9, 2012 7:50 UTC (Sat)
by bsdphk (guest, #85042)
[Link] (1 responses)
With respect to mistimed, no, it was very carefully timed and I hit it very close to perfect.
To get a message like this out to all the people who need to hear it is very difficult.
I don't have a Vogon Tanoy where I can press a button and go "People of Earth, Your attention please..."
Piggybacking on the LinkedIn attention-wave, meant that news-providers had "password" in their short term memory and my message got very efficiently relayed.
With respect to the guidance I give: I you think that is a bad idea, I suggest you write an easy to use, liberally licensed password scrambler which actually solves the problem for a decade or two.
The present wave can be surfed for about a week more...
Thanks for using my code
Poul-Henning
Posted Jun 11, 2012 10:31 UTC (Mon)
by intgr (subscriber, #39733)
[Link]
There are tons of free implementations of PBKDF2 and a few of scrypt. Those are the only two that are worth using. Surely you were aware of both? I'm surprised you're even suggesting that we need any new libraries/standards.
Posted Jun 8, 2012 8:58 UTC (Fri)
by epa (subscriber, #39769)
[Link] (5 responses)
I'd go further and say that for web sites there is no reason to handle the authentication in the application code at all - you should use what Apache or another web server provides, with a small utility page for adding users - but most web developers seem to disagree with me on that. Perhaps because it's felt that users are not able to cope with the plain username/password dialogue box that browsers give for http authentication.
Posted Jun 8, 2012 13:55 UTC (Fri)
by dps (guest, #5725)
[Link] (4 responses)
If you use a form instead then you can implement more secure protocols which do not have those problems, for example proving you can decrypt RSA given the E_K(secret key) where K=KDF(salt,password). This is not difficult to implement and a lot better anything a browser or server is likely to support.
Sending passwords and using unsalted hashes is plain stupid and should be severely punished. Did anyone tell these people about rainbow tables and them not working if you use salt?
Posted Jun 8, 2012 15:14 UTC (Fri)
by epa (subscriber, #39769)
[Link]
As for sending passwords over the wire, I suppose that in principle you could use some awesome piece of Javascript to implement a zero-knowledge proof, but surely 99.999% of HTML password forms out there on the web just send the password back to the server anyway.
HTTP digest authentication uses salted MD5, which is not ideal but not awful.
Posted Jun 8, 2012 15:37 UTC (Fri)
by tialaramex (subscriber, #21167)
[Link] (2 responses)
Rainbow tables are a time-space tradeoff, RAM and disk space continues to be only marginally cheaper than it was a few years ago, while GPU based "brute force" attacks can be thousands of times faster than CPU based attacks from a few years ago. So rather than buy 50TB of storage for rainbow tables targeting a single algorithm, it is more tempting for attackers to buy, rent, or indeed steal (through a botnet) a dozen mid-range gaming GPUs and have the flexibility to attack dozens of different algorithms for which GPU-based brute forcers are available.
Salt does still mean the attackers have to attack every individual password, rather than gathering a list of the most common hashes and attacking those for disproportionate gain, and it also means if some day rainbow tables or a newer time-space tradeoff would otherwise have been viable that's sidestepped. But it's far less important today than it was a few years ago.
Posted Jun 8, 2012 15:57 UTC (Fri)
by intgr (subscriber, #39733)
[Link]
The time-space tradeoff goes both ways. By using GPU instead of CPU for the time-intensive parts, you can increase the storage efficiency of rainbow tables (generate longer hash chains); the speedup would still be the same proportion. However, I don't know whether there are any actual implementations of this.
Posted Jun 8, 2012 20:18 UTC (Fri)
by jackb (guest, #41909)
[Link]
Posted Jun 9, 2012 18:14 UTC (Sat)
by halfline (guest, #31920)
[Link]
For instance, the algorithm treats all passwords greater than 8 characters to be equivalent to the first 8 character prefix of those passwords. (the password "password" is equivalent to the password "password hashing is hard")
It's documented to only look at the lower 7bits of each character as well, so "ůh?" is equivalent to "Eh?", I believe, etc
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
I understand, but I kind of like the term "coordinated disclosure". It is neutral and descriptive.
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
Full disclosure = Early disclosure
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
The name "Responsible disclosure" is inflamatory
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Responsible disclosure in open source: The crypt() vulnerability
Rainbow tables
Rainbow tables
Rainbow tables
GPU-based parallel brute force attacks
You could probably blame Bitcoin in part for making these kinds of rigs popular.
Responsible disclosure in open source: The crypt() vulnerability