Ubuntu Forums account information breached

[Posted July 21, 2013 by jake]

The Ubuntu Forums site is currently displaying a "down for maintenance" message that describes a security breach of the accounts database, which revealed all usernames, passwords, and email addresses to the attacker(s). Ubuntu One, Launchpad, and other Canonical web sites are not affected by the breach. "The passwords are not stored in plain text. However, if you were using the same password as your Ubuntu Forums one on another service (such as email), you are strongly encouraged to change the password on the other service ASAP."

to post comments

Ubuntu Forums account information breached

Posted Jul 21, 2013 16:21 UTC (Sun) by imgx64 (guest, #78590) [Link] (33 responses)

Oddly, they haven't emailed the users about the breach. I only learned about it through OMG Ubuntu and (hours later) the Canonical blog.

Ubuntu Forums account information breached

Posted Jul 21, 2013 17:55 UTC (Sun) by Flukas88 (subscriber, #87138) [Link]

Same thought as me

Ubuntu Forums account information breached

Posted Jul 21, 2013 19:49 UTC (Sun) by Cato (guest, #7643) [Link] (19 responses)

Unfortunately this is very common when hashed passwords are compromised. It seems that most large sites find it hard to email their entire user base, as it requires a mail-transport service that can send to millions of users without getting marked as spam.

Ubuntu Forums account information breached

Posted Jul 21, 2013 20:15 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (18 responses)

But it's not as though such services don't exist, and regardless how you feel about their methods (hint: money talks when it comes to major ISPs and other email providers) they're very good at the type of problem which we're talking about here, delivering a form email to a vast number of people as soon as possible without it landing in everybody's unread spam pile. For a price.

I deal with one of them (actually I've dealt with several, but I'm now obliged to deal with the one that's owned by the corporation I work for) and they're useless and frustrating in many ways, but they sure can deliver an email form letter to a few million people when you need.

Are these forums a Canonical project? They seem to be, judging from the domain name owner. If so, Canonical ought to have agreed a procedure for what they do when something like this happens. Some group of people need to get woken up to make things happen. Things need to be turned off, preserved for any investigation, forensic people need calling in, and somewhere in the list a bulk email has to go out to warn the site's users. This wasn't a bank or a hospital, we're not expecting them to carry specialist insurance, but they do need a _procedure_ for such entirely foreseeable circumstances if they're going to play at being a serious business.

Ubuntu Forums account information breached

Posted Jul 21, 2013 20:52 UTC (Sun) by LightDot (guest, #73140) [Link] (3 responses)

What evidence you have that there isn't a procedure in place for such events at Canonical? You're assuming this solely because an email wasn't sent to you?

Ubuntu Forums account information breached

Posted Jul 22, 2013 15:26 UTC (Mon) by drag (guest, #31333) [Link] (2 responses)

Him and everybody else.

Did you get a email?

Ubuntu Forums account information breached

Posted Jul 22, 2013 15:47 UTC (Mon) by johill (subscriber, #25196) [Link] (1 responses)

My brother said he did get an email, I have no reason to believe otherwise.

Ubuntu Forums account information breached

Posted Jul 22, 2013 19:09 UTC (Mon) by andybruk (guest, #31794) [Link]

I received an email, but not as soon as I would have expected.

Ubuntu Forums account information breached

Posted Jul 21, 2013 23:16 UTC (Sun) by dlang (guest, #313) [Link] (2 responses)

If it's a matter of negotiating payments with thousands of ISPs to prevent a bulk mailing like this from being marked as SPAM, is it any wonder that it takes a bit of time to set something like that up?

and why should any site go through all this work ahead of time?

frankly, I think they should have sent out an e-mail to everyone, and if the ISP marks it as SPAM, shame on the ISP. It's not the senders responsibility to pay people to not mark messages as SPAM.

Ubuntu Forums account information breached

Posted Jul 22, 2013 9:10 UTC (Mon) by tialaramex (subscriber, #21167) [Link] (1 responses)

The point is that there are companies which offer this as a service. You hand them an email template, a huge list of email addresses (and optionally information like first name, account number to be mail merged), and some money. They send the emails. They deal with the technical, political and where necessary financial complications of getting those emails delivered. And of course they offer a lot more stuff that they've found helps sell this core service.

I'd be surprised if Canonical don't have a contract with such a company. But I would not be at all surprised to discover that Canonical's marketing department has the contract and nobody involved in the emergency response to this breach knew it existed or how to make use of it.

Ubuntu Forums account information breached

Posted Jul 22, 2013 9:24 UTC (Mon) by Cato (guest, #7643) [Link]

Exactly right - Postmark, SendGrid and many others spend a lot of time working with ISPs and other email inbox providers so that valid emails are not marked as spam (or getting them un-marked), and are very tough on any customer of theirs who does spam. This "high deliverability SMTP as a service" is paid-for since it takes significant time to do this, and is much easier than doing this yourself.

Companies such as Canonical may already be using such a service for activation emails, in fact, so they just need to work with the SMTP provider to ensure that the "your forum account was compromised" ahead of time to ensure that's highly deliverable.

Ubuntu Forums account information breached

Posted Jul 22, 2013 5:47 UTC (Mon) by Cato (guest, #7643) [Link] (10 responses)

I agree completely that companies could and should use these high-volume email services (Postmark and others), and test them ahead of time, but they almost never do. I think Steam sent me an email when their forums were hacked, but I can't find it.

It's also disappointing that Ubuntu apparently used MD5 without any iterations/stretching as its password hash algorithm. And yes, the forums are run by Canonical.

Some background links:

- this hack: http://arstechnica.com/security/2013/07/hack-exposes-e-ma...
- general: http://arstechnica.com/security/2012/08/passwords-under-a...

Ubuntu Forums account information breached

Posted Jul 22, 2013 9:17 UTC (Mon) by tpo (subscriber, #25713) [Link] (6 responses)

I'd also like to mention this article on ArsTechnica, which by means of looking over the shoulders of some good crackers, is very enlightening wrt which passwords are good:

http://arstechnica.com/security/2013/05/how-crackers-make...

Ubuntu Forums account information breached

Posted Jul 22, 2013 9:30 UTC (Mon) by Cato (guest, #7643) [Link] (1 responses)

That article is highly relevant, since apparently Ubuntuforums used plain MD5 to hash its passwords, and the article shows how 90% of such MD5 hashed passwords were crackable.

MD5 is a really bad choice for password hashing, as it's so fast to calculate an MD5 hash (by design). Salting helps a bit, but stretching (doing many iterations such that a normal password check takes perhaps 0.5 sec) is really required - or use a better algorithm such as bcrypt or scrypt (better against FPGAs, search for 'tarsnap scrypt').

Ubuntu Forums account information breached

Posted Jul 22, 2013 10:36 UTC (Mon) by pboddie (guest, #50784) [Link]

Thomas Waldmann gave a nice talk about password hashing at EuroPython this year: video, overview and link to slides. The talk tries to educate about the issues of hashing and the threats to stored hashes, and it gives details of libraries that help developers integrate recommended practices.

Ubuntu Forums account information breached

Posted Jul 24, 2013 12:52 UTC (Wed) by njwhite (guest, #51848) [Link] (3 responses)

Thanks for that, it was a very interesting read.

It makes me wonder, do sites often allow arbitrary bytes as passwords? That would increase the search space a little, right? From 95^7 for a 7 character password to 128^7. If so one could just create a usable password with something like: "dd if=/dev/random bs=32 count=1 | xclip".

Is there some reason this wouldn't work?

Ubuntu Forums account information breached

Posted Jul 24, 2013 13:55 UTC (Wed) by Cato (guest, #7643) [Link] (1 responses)

The only thing to be careful of here is that every webapp may potentially strip unusual characters from the input - unless you know exactly which ones are stripped, and are sure the rules for this are not changed, it's hard to be sure this will work.

Ubuntu Forums account information breached

Posted Jul 24, 2013 15:30 UTC (Wed) by mpr22 (subscriber, #60784) [Link]

Many webapps reject passwords containing characters their designers have decided aren't properly passwordy, which includes some for which isprint() returns non-zero.

Ubuntu Forums account information breached

Posted Jul 24, 2013 15:56 UTC (Wed) by jimparis (guest, #38647) [Link]

> From 95^7 for a 7 character password to 128^7. If so one could just create a usable password with something like: "dd if=/dev/random bs=32 count=1 | xclip".

I don't think any site (or browser) would support nulls and many other characters you'd get. But "tr -dc '[:alnum:]' < /dev/urandom | head -c 32 | xclip" would be well supported and 62^32 is plenty huge.

Ubuntu Forums account information breached

Posted Jul 22, 2013 9:45 UTC (Mon) by tialaramex (subscriber, #21167) [Link] (2 responses)

It's not clear to me what was used here. Journalists haven't been very good at reporting this stuff and can't be relied on to understand the difference between MD5 (a cryptographic primitive intended to be very fast, and thus unsuitable for password hashes) and PHK-MD5 (Poul-Henning Kamp built a crypt scheme that uses many iterations of MD5 with salt, it was commonly used on Unix systems around the turn of the century)

Because this was specifically forum software I'm tempted to conclude that it was some simplistic MD5(salt + password) scheme of the sort used in several popular PHP forum programs. But it's by no means impossible to arrange to use PHK-MD5, either from PHP or from other popular web forum platforms.

PHK-MD5 isn't perfect, but it would put a serious crimp on attempts to reverse non-trivial passwords out of their hash, buying people time to hear about the problem and do something about it.

If this was off-the-shelf forum software we can probably determine for ourselves. After a good start a few years back, it seems forums and particularly PHP forums became rather complacent and most haven't ever got beyond adding some salt to reduce exposure to rainbow tables. So there certainly are products out there which still do MD5(salt + password) or SHA1(salt + password) or other things that should have been laughable forty years ago.

Ubuntu Forums account information breached

Posted Jul 22, 2013 10:08 UTC (Mon) by Cato (guest, #7643) [Link] (1 responses)

Ubuntuforums.org is using vBulletin and according to this answer http://stackoverflow.com/a/2202810/992887 that means MD5 applied just twice with no extra iterations. At least they now use salt...

It's possible that vBulletin upgraded their hash algorithm, but this 2011 answer about cracking is based on the same algorithm: http://security.stackexchange.com/questions/7229/what-is-...

Ubuntu Forums account information breached

Posted Jul 22, 2013 14:52 UTC (Mon) by tialaramex (subscriber, #21167) [Link]

If it's vBulletin then you're quite right that it's just

MD5(MD5(password)+salt)

... which is what happens when you tell a bunch of people who don't get it that MD5(password) is no good. It's almost comical that a commercial product (vBulletin costs hundreds of dollars per license) for which the source code is open to every purchaser (because it's in an interpreted language) has such terrible security, but it's not as though they're alone. Even supposedly well-run open source projects to build web software have a poor record on this stuff.

PHP itself finally shipped a version with a more or less sane password hash feature out of the box only this year.

Ubuntu Forums account information breached

Posted Jul 22, 2013 13:54 UTC (Mon) by simosx (guest, #24338) [Link] (10 responses)

> Oddly, they haven't emailed the users about the breach. I only learned about it through OMG Ubuntu and (hours later) the Canonical blog.

It is not odd; they are still working to get the system up, so that users CAN ACTUALLY change their password.

http://ubuntuforums.org/ says it is still down.

Ubuntu Forums account information breached

Posted Jul 22, 2013 14:15 UTC (Mon) by farnz (subscriber, #17727) [Link] (9 responses)

Breach notification isn't just about changing my password on the breached site - it's also the point at which you tell me that if I've been foolish enough to use the same password in two places (which most people do), I need to change it in the other place, too. For example, if I used the same password for ubuntuforums.org and for my bank ("because Ubuntu is secure, so I don't need to worry"), I need to change my banking password ASAP.

Ubuntu Forums account information breached

Posted Jul 22, 2013 15:34 UTC (Mon) by drag (guest, #31333) [Link] (8 responses)

Most people, unfortunately, are under the impression that they shouldn't write passwords down.

Having to memorize a hundreds of passwords for each new website you join is a impossible task. Having been told over and over again 'not to write down passwords' from so-called 'security experts' who really are completely and utterly ignorant of security... most people just resort to getting a password they think is really secure and then using it over and over again.

Given the fact that saying "Don't write down passwords", "Use secure passwords", "Use unique passwords" is just setting users up for failure then it's natural that they fail.

It's really the fault of the 'security culture' of misinformation and bad practices that is commonplace in computer-land.

You can't blame the 'average person' for listening to people who are purported as experts, but are not, but follow memes that are so widely repeated that most people just accept them as fact.

Ubuntu Forums account information breached

Posted Jul 22, 2013 15:57 UTC (Mon) by raven667 (subscriber, #5198) [Link]

> It's really the fault of the 'security culture' of misinformation and bad practices that is commonplace in computer-land.

Unfortunately I agree, most security practices are cargo-cult repetition of best security practices which were codified 20 years ago. Ideally security practices would be based on performing a thorough risk assessment process and then that would drive priorities on taking reasonable and appropriate measures to mitigate the risk to whatever level is acceptable to your organization. Without defining what is acceptable risk, what the risks are and how your mitigation strategies affect the total risk you aren't doing real security, you are just doing cargo-cult security.

Ubuntu Forums account information breached

Posted Jul 22, 2013 16:21 UTC (Mon) by lopgok (guest, #43164) [Link] (6 responses)

So true. The era of not writing passwords down anywhere has past.

I have done some minor password cracking over the years. Now with GNU acceleration, very little is safe. I ended up changing all of my passwords to be unique and randomly generated. I use keepas and a random password generator I wrote myself. For example, my lwn password is 20 characters long, and impossible for me to remember.

It is quite interesting the restrictions different sites place on passwords. Some have length limits that they don't publicize. Some have character restrictions they don't publicize. The most common limitiation I have found is no spaces are allowed. Some make it hard to find out where to go to change the password.

The random generation defeats all dictionary lists.
Not reusing passwords limits the exposure if a site does get compromised.
My password do resemble line noise, but that is a small price to pay for security.

Ubuntu Forums account information breached

Posted Jul 22, 2013 22:37 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (4 responses)

How do you deal with needing to login from some arbitrary endpoint without your password database? Do you just not have this usecase?

phone

Posted Jul 23, 2013 9:56 UTC (Tue) by tialaramex (subscriber, #21167) [Link] (1 responses)

Fortunately these days most of us carry pocket-sized computers around which we have taken to calling "phones". So you can just have a cryptographically synchronised password store between your phone and say a desktop or laptop. And I even made a telephone call on mine the other day.

I say "just", this is clearly a ridiculously over-complicated system, but that's the price we pay for marginally better (email resets mean it can never be more than marginal) security in the face of interfaces designed for the appearance of convenience. Real authentication systems are too difficult for the zero friction experience that is desired by advertisers, and thus by the services they fund.

phone

Posted Jul 23, 2013 15:21 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> email resets mean it can never be more than marginal

Well, this is what 2-factor authentication on email is for :) . Or SSL client certs if you run your own email.

As for using your phone, yeah, I'm aware the phone can fill the gap, but that doesn't mean its the only solution. Plus, given the sync solutions available, services which encrypt.the data would be nice (personally, I'm anxious for git-annex to be supported on Android).

Ubuntu Forums account information breached

Posted Jul 23, 2013 16:34 UTC (Tue) by raven667 (subscriber, #5198) [Link]

That's actually a totally acceptable failure mode in my opinion, I prefer to not ever log in as myself using a computer that I don't control. You are exposing yourself to a higher chance of having a key logger other malware from whatever festering cesspool of a kiosk you try to use.

With the deployment of mobile computers and wireless data, needing to log in from some unknown computer is a fading use case.

Ubuntu Forums account information breached

Posted Jul 23, 2013 23:47 UTC (Tue) by lopgok (guest, #43164) [Link]

Obviously it is a difficult problem.

The best solution is a smartphone with keepass or something compatible with it. I do have a few memorized passwords.

However, I have a total of about 350 unique passwords, and there is no way to memorize all of them, with reasonable password complexity.

If I am logging in somewhere, I almost always have my notebook available, and I have keepass for it. If I don't have a notebook or smartphone, I am severely limited in what I can access. I think that is a reasonable tradeoff for my security. Beats getting my apple/ubuntu/sony account hacked.

Ubuntu Forums account information breached

Posted Jul 25, 2013 10:03 UTC (Thu) by pabs (subscriber, #43278) [Link]

If you need passwords that are memorable, then look into Diceware, which is intended for passphrases that are not to be recorded, like the ones used to protect disk encryption or SSH/OpenPGP keys.

Ubuntu Forums account information breached

Posted Jul 24, 2013 22:25 UTC (Wed) by geuder (subscriber, #62854) [Link]

> Oddly, they haven't emailed the users about the breach.

I have received an email 6 hours ago. On one side it took nearly 4 days since they detected the problem, but on the other side it's still too early. The site isn't up yet, so who will remember to change the password later. Unless they send another mail once it is up. (I don't consider myself really affected because I'm registered with a disposable email address and a medium strong password not used anywhere else.)

As far as I can see the mail was sent directly from a Canonical server, gmail did not classify it as spam (as it uses to it with the occasional genuine Paypal mails I receive)