Drupal.org resets login credentials after hack exposes password data

Passwords for almost 1 million accounts affected after malicious files are found.

Passwords for almost one million accounts on the Drupal.org website are being reset after hackers gained unauthorized access to sensitive user data.

Drupal.org is the official website for the popular open-source content management platform. The breach is the result of an attack that exploited a vulnerability in an undisclosed third-party application and not in Drupal itself, according to Holly Ross, executive director of the Drupal Association, in a blog post published Wednesday. The hack exposed usernames, e-mail addresses, country information, and cryptographically hashed passwords, although investigators may discover additional types of information were compromised.

"Malicious files were placed on association.drupal.org servers via a third-party application used by that site," Ross wrote. "Upon discovering the files during a security audit, we shut down the association.drupal.org website to mitigate any possible ongoing security issues related to the files. The Drupal Security Team then began forensic evaluations and discovered that user account information had been accessed via this vulnerability."

There's no indication credit card data was intercepted. There's also no evidence that any unauthorized changes were made to Drupal source code or projects.

Drupal.org administrators have responded by rebuilding production, staging, and development systems and enhancing most servers with grsecurity, a set of security patches for the Linux operating system. The admins have also hardened their configuration of the Apache Web server application and added antivirus scanning to their security routine. Some Dupal.org subsites, particularly those with older content, have been converted to static archives so they can't be updated in the future.

Drupal.org account holders will be required to change their password by visiting this link, entering their username or e-mail address, and following the link included in the e-mail message that follows. Ross also encouraged account holders to change login credentials on other sites that used the same or a similar password used on Drupal.org.

Most of the passwords stored by Drupal.org were both salted and, more importantly, passed through a cryptographic hash function multiple times using the open-source phpass application. Some older passwords weren't salted. If Drupal engineers followed good practices—and there's no indication they didn't—the repeated hash iterations will go a long way towards preventing anyone who obtains the data from quickly cracking the hashes and exposing the underlying plaintext that generated them. (Cryptographic salting, which appends unique characters to each password before it's hashed, is also helpful, although people frequently overstate the protection it provides. For much more on password protection see the Ars feature Anatomy of a hack: How crackers ransack passwords like “qeadzcwrsfxv1331”.)

Promoted Comments

I'm a regular Drupal community contributor, but not part of the security team and I didn't work to respond to the breach today, but I can provide some additional information here based on what I know.

Ars says correctly that 1 million passwords were *affected*, meaning they were reset. However 1 million passwords were not necessarily *stolen*. Most all subsites on the drupal.org infrastructure use a module called Bakery to manage account logins across subdomains of drupal.org. This makes it so that you can easily create an account (or have one created automatically) on any subdomain of drupal.org, including the affected association.drupal.org domain. However when these accounts are created, the password and user data stays on the central site (drupal.org), and only the critical information for the site is pulled into the subsite, which includes the username and e-mail address. Authentication is done by passing the user to the central site, setting a cookie, then redirecting back to the subdomain (so the main site is the "bakery" that makes "cookies", see?). So if the Drupal association site's data was compromised individually, only users who had signed into the association website would have their username and e-mail accessible to the hackers. Their password and profile information would remain only on drupal.org itself, which as far as we know from the information released, was not affected.

Anyway, the scope of the attack/leak isn't very clear at this point. But if the attack was limited only to the Drupal association website, most personal details (and all passwords) were not accessible. Perhaps a small consolation in the face of a large data breach, but I don't think Drupal.org has done the wrong thing by proactively requires all passwords to be reset, even in the event that none of them may have been stolen.

OTOH, we'll find out if all the drupal.org infrastructure was affected shortly, in which case maybe my "it's not as bad as it sounds" perspective may be nullified.

In response to my own post above, the official response had a lot more detail than my original hearsay. I'd take the Drupal Association at their word that profile information from the main site was leaked:

"Information exposed includes usernames, email addresses, and country information, as well as hashed passwords."

So although the use of the Bakery module and the separation of Drupal.org from its subsites is an interesting case-study in separation of user accounts and user data, it doesn't look like it's going to help us in this situation.

Possibly the best response to being hacked I've seen. They acknowledge that they've been hacked, that at least some passwords have been stolen, and that there is a real risk to the users whose passwords may have been stolen. If every website that got hacked did this I suspect we would all be moderately more secure. The fact that they are trying to alert people about the hack as fast as possible, before everything is really known about it, is really only a bonus.

Everything can be hacked these days, the fact that they noticed it, were fairly prepared (very good password hashing apparently, distributed systems apparently, ect), and had a great response should in all honestly be a reason to use them, not a reason to hate on them, or avoid their service.

Just wanted to provide a little bit of clarity we've confirmed since our initial announcement. Understandably there are questions today about whether the vulnerability we experienced is something that may affect others. We want to let you know that we have worked with the software vendor to confirm it is a known vulnerability and has been publicly disclosed. We are still investigating and appreciate your patience. We have updated the FAQ to include this information.

That is it, it is a binary choice. The correct answer is to salt. The need to salt can in fact not be overstated.

I doubt rainbow tables are much used these days. If you can go through a billion md5s/second, why bother?

I think you are correct here. The trend seems to be using GPUs to quickly calculate hashes, rather than storing a ton of them in massive databases. Salts are almost always stored right next to the hashed password in the database, so they really only protect you against rainbow tables, which isn't saying much. A lot of people advise against using a single site-wide salt that's stored in a config file, but IMO that makes more sense to me, because the entire server would need to be compromised to get the needed info. A combination of the two methods would probably be your best bet.

The problem with posts like the one Azethoth666 left is they're based on decade-old attacks that no one uses anymore. This kind of outdated thinking gives administrators a false sense of security. Much more important than salt is the use of bcrypt, scrypt or PBKDF2 to hash passwords. Don't get me wrong, salt is crucial, but its importance is secondary to use of a slow hashing function.

23 Reader Comments

Drupal.org account deleted and nasty email sent. This shit has GOT TO STOP. I can't even count the number of times my personal details have been released by irresponsible companies exposing dumbass vulnerabilities. Sick of it.

Such is the nature of highly complex software, only now more than ever are vulnerabilities in software that is usually not under attack are being exposed.

What was drupal supposed to do? There is no other alternative to "switch" to because all the alternatives are being attacked as well. Its fortunate that unlike so many other breaches this one was at least fortified their information better.

The admins have also hardened their configuration of the Apache Web server application and added antivirus scanning to their security routine.

I'm curious about that. If they are only now adding antivirus scanning, with the understanding that it will only now catch this mysterious and newish third party badness, maybe that's understandable. But if there has been anything else on the site that would put users, if not their own servers, at risk then perhaps it should have been in place already. I guess I'm not really grasping the antivirus scanning implementation in their case and why they didn't scan previously.

I'm a regular Drupal community contributor, but not part of the security team and I didn't work to respond to the breach today, but I can provide some additional information here based on what I know.

Ars says correctly that 1 million passwords were *affected*, meaning they were reset. However 1 million passwords were not necessarily *stolen*. Most all subsites on the drupal.org infrastructure use a module called Bakery to manage account logins across subdomains of drupal.org. This makes it so that you can easily create an account (or have one created automatically) on any subdomain of drupal.org, including the affected association.drupal.org domain. However when these accounts are created, the password and user data stays on the central site (drupal.org), and only the critical information for the site is pulled into the subsite, which includes the username and e-mail address. Authentication is done by passing the user to the central site, setting a cookie, then redirecting back to the subdomain (so the main site is the "bakery" that makes "cookies", see?). So if the Drupal association site's data was compromised individually, only users who had signed into the association website would have their username and e-mail accessible to the hackers. Their password and profile information would remain only on drupal.org itself, which as far as we know from the information released, was not affected.

Anyway, the scope of the attack/leak isn't very clear at this point. But if the attack was limited only to the Drupal association website, most personal details (and all passwords) were not accessible. Perhaps a small consolation in the face of a large data breach, but I don't think Drupal.org has done the wrong thing by proactively requires all passwords to be reset, even in the event that none of them may have been stolen.

OTOH, we'll find out if all the drupal.org infrastructure was affected shortly, in which case maybe my "it's not as bad as it sounds" perspective may be nullified.

Possibly the best response to being hacked I've seen. They acknowledge that they've been hacked, that at least some passwords have been stolen, and that there is a real risk to the users whose passwords may have been stolen. If every website that got hacked did this I suspect we would all be moderately more secure. The fact that they are trying to alert people about the hack as fast as possible, before everything is really known about it, is really only a bonus.

Everything can be hacked these days, the fact that they noticed it, were fairly prepared (very good password hashing apparently, distributed systems apparently, ect), and had a great response should in all honestly be a reason to use them, not a reason to hate on them, or avoid their service.

In response to my own post above, the official response had a lot more detail than my original hearsay. I'd take the Drupal Association at their word that profile information from the main site was leaked:

"Information exposed includes usernames, email addresses, and country information, as well as hashed passwords."

So although the use of the Bakery module and the separation of Drupal.org from its subsites is an interesting case-study in separation of user accounts and user data, it doesn't look like it's going to help us in this situation.

I <3 Drupal. I'm a contributor to a couple of modules, and develop Drupal sites regularly at an agency. The core community responsible for maintaining Drupal and Drupal.org are very responsible folk, and I'm very happy about the way they handled this incident. Makes me feel a bit more secure in the community I trust and hold dearly to my professional heart.

"(Cryptographic salting, which appends unique characters to each password before it's hashed, is also helpful, although people frequently overstate the protection it provides."I want to nit-pick this statement.

Salting defeats rainbow table and similar pre-computing attacks. It also defeats the basic Birthday Paradox math. Each salt-hash combo has its own keyspace and needs to be independently attacked. It is not magical protection, it is simply necessary protection.

No salt = I do not care about security.Salt = I care about security.

That is it, it is a binary choice. The correct answer is to salt. The need to salt can in fact not be overstated.

However, and perhaps this is what you are trying to get at, it is only one of many things that needs to be done right. All the salt in the sea will not help you if you use some common hash instead of scrypt, bcrypt or less ideally pbkdf2. Nor will it prevent cracking of the known passwords or short passwords or common pattern passwords that users love.

I think an article on the history of encryption schemes vs the attacks on them would be interesting.

Some simple attacks:1) Timing attack. Measure the time it takes a badly designed check to reject. Worst case, you can defeat in basically linear time. Solution: only use properly implemented and time tested schemes that among other things only return pass fail as a single atomic result with no timing difference between success / fail results.2) Power attack. Attackers that can measure the power consumed during decryption can potentially tell exactly what the decryption code is doing and thus break the scheme. Solution: decryption code designed to use the same power profile no matter what the internal failures detected during decryption.3) Forgot a necessary step in the algorithm. Oh noes, without that last xor with the key or something similar the cypher is no longer secure and one xor later the text is completely revealed.4) Using a bad implementation. Oh noes, instead of getting scrypt security out of my platform / language crypto package, instead due to 1, 2, 3 or some other flaw, I get basically the same security as a plain hash or worse. (This is not just a historical issue).

Just wanted to provide a little bit of clarity we've confirmed since our initial announcement. Understandably there are questions today about whether the vulnerability we experienced is something that may affect others. We want to let you know that we have worked with the software vendor to confirm it is a known vulnerability and has been publicly disclosed. We are still investigating and appreciate your patience. We have updated the FAQ to include this information.

Why is everyone being so positive about Drupal? I suspect it's because people generally like Drupal; if this was Microsoft or HP, there's be more (correct) cries of WTF were they doing before the attack?

It's all well and good to follow the script after you are attacked, but they should have been following best practices for a long time.

Why is everyone being so positive about Drupal? I suspect it's because people generally like Drupal; if this was Microsoft or HP, there's be more (correct) cries of WTF were they doing before the attack?

It's all well and good to follow the script after you are attacked, but they should have been following best practices for a long time.

Not to boost for the home team (I design and develop in Drupal, and am active in the community), but those responsible for security are pretty conscientious, and regularly post patches and updates in response to emerging threats, even if those aren't directly affecting Drupal (at least at that time).

Unlike some major players in the industry who shall go unnamed, good FOSS communities like Drupal are quick to get the word out and correct issues.

No system or organization is bulletproof, no matter how well organized. Vigilance and regular evaluations are required to keep things (relatively) secure...

Drupal.org account deleted and nasty email sent. This shit has GOT TO STOP. I can't even count the number of times my personal details have been released by irresponsible companies exposing dumbass vulnerabilities. Sick of it.

What is your solution?

Rolling your own OS/CMF?

Pretty much every major financial and government institution worldwide has been hit at some time, and why there are highly-skilled people dealing with intrusions on a constant basis.

You could go off the grid and hide your money under the mattress, but there are some definite downsides to that lifestyle...

Drupal.org account deleted and nasty email sent. This shit has GOT TO STOP. I can't even count the number of times my personal details have been released by irresponsible companies exposing dumbass vulnerabilities. Sick of it.

What is your solution?

Rolling your own OS/CMF?

...

While I agree that iamaelephant is overreacting a little bit, let's not pretend like Drupal is the only choice here. Lots of - if not most - software development companies use frameworks.

That is it, it is a binary choice. The correct answer is to salt. The need to salt can in fact not be overstated.

I doubt rainbow tables are much used these days. If you can go through a billion md5s/second, why bother?

I think you are correct here. The trend seems to be using GPUs to quickly calculate hashes, rather than storing a ton of them in massive databases. Salts are almost always stored right next to the hashed password in the database, so they really only protect you against rainbow tables, which isn't saying much. A lot of people advise against using a single site-wide salt that's stored in a config file, but IMO that makes more sense to me, because the entire server would need to be compromised to get the needed info. A combination of the two methods would probably be your best bet.

That is it, it is a binary choice. The correct answer is to salt. The need to salt can in fact not be overstated.

I doubt rainbow tables are much used these days. If you can go through a billion md5s/second, why bother?

I think you are correct here. The trend seems to be using GPUs to quickly calculate hashes, rather than storing a ton of them in massive databases. Salts are almost always stored right next to the hashed password in the database, so they really only protect you against rainbow tables, which isn't saying much. A lot of people advise against using a single site-wide salt that's stored in a config file, but IMO that makes more sense to me, because the entire server would need to be compromised to get the needed info. A combination of the two methods would probably be your best bet.

The problem with posts like the one Azethoth666 left is they're based on decade-old attacks that no one uses anymore. This kind of outdated thinking gives administrators a false sense of security. Much more important than salt is the use of bcrypt, scrypt or PBKDF2 to hash passwords. Don't get me wrong, salt is crucial, but its importance is secondary to use of a slow hashing function.