Posted
by
Soulskill
on Saturday January 21, 2012 @05:59PM
from the another-day-another-intrusion dept.

New submitter Ccmods writes "Below is a snippet from an email Dreamhost sent to subscribers early Saturday morning, describing an intrusion into the database storing FTP and SSH usernames and passwords: 'We are writing to let you know that there may have been illegal and unauthorized access to some of your passwords at DreamHost today. Our security systems detected the potential breach this morning and we immediately took the defensive precaution of expiring and resetting all FTP/shell access passwords for all DreamHost customers and their users. ... Only the FTP/shell access passwords appear to have been compromised by the illegal access. Web panel passwords, email passwords and billing information for DreamHost customers were not affected or accessed.'"

As a Dreamhost customer, I watched this unfold in real time. Apparently the passwords were hashed, and there's no indication that they were compromised, other than the fact that it was technically possible. So they changed the passwords because it's cheaper, PR-wise, than being wrong.

There's a big warning up on the panel, which has a password stored in a different, non-compromised DB. Between the panel and the email, I doubt anybody's confused as to what's going on.

In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.

Like many Dreamhost customers, I have used many other hosts over the years. None has even come close to Dreamhost. Many companies try to project an aura of professionalism but are really mickey mouse operations on the inside. Dreamhost is the opposite. I think they make a point to act like clowns only to scare off the clueless, high-maintenance market.

Well, several years ago, before they moved their servers, I was in the same datacenter with them. My cage was almost next to theirs. On several occasions, I talked to them. All of their folks knew their stuff, and showed me around the inner workings a good bit. I was impressed. I highly recommended them at the time. Unless someone made some horrible decisions, I strongly suspect they're still worth the praise.

I'd be hard pressed to believe you've dealt with anyone other than DreamHost. When I failed to renew my service with them it was because their hosting was glacial. It could barely keep up with a lightly used PHP image gallery. That was years ago. When I migrated clients away from DH last year it was because of chronic downtime. "Oops we fucked up" is great, and it's honest. It's also not something you want to keep seeing [dreamhoststatus.com]. "Oops we fucked up, but your worthless blogs about your kittens' trousers are s [dreamhost.com]

status.dreamhost.com = dreamhoststatus.com = down when they bork one of their core routers. After last year, I'd sure hope that they've put the status blog and corporate e-mail on a separate network. Pretty sure that they've been too busy [dreamhost.com] posting pictures of cat anuses [dreamhost.com] to have bothered changing anything tho. Phone support/is/ available, but you've got to plead for a call back via e-mail. That's all fine and dandy except for the fact that it's all on one network. A single point of failure is a single p

As a Dreamhost customer, I watched this unfold in real time. Apparently the passwords were hashed, and there's no indication that they were compromised, other than the fact that it was technically possible. So they changed the passwords because it's cheaper, PR-wise, than being wrong.

There's a big warning up on the panel, which has a password stored in a different, non-compromised DB. Between the panel and the email, I doubt anybody's confused as to what's going on.

In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.

It's good to see they took the matter seriously, even with the circumstances you describe. Bad that it happened in the first place, but it sounds like the situation was nicely handled.

The only offered method for dealing with a forgotten ftp/shell password is to force a reset through the panel, either by having one generated on request or by providing one yourself. It is displayed only once after that point, in the panel, as a confirmation.

That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.

That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.

Either scenario would allow the user to retrieve the password without the host or an attacker being able to see what the associated email or password is.

No, they would not. This is simple. They store only the email hash. Your account id is tied to that hash. Upon login, you are required to provide your email address and password; the email is used to try to decrypt the password, and is then hashed and compared to their stored hash. If the provided password matches the decrypted password, and the hashed email matches the stored hash, you get logged in.

For password recovery, all you provide is the email, which is again used to perform a hash lookup, and

That is probably true in this case, but is not necessarily true in all cases. Imagine, for example, that the password is encrypted with the email address as its key, and the email address is hashed. Upon login, a hash lookup is done for the email address, and the encrypted password is decrypted and compared to the one sent. Or alternatively, the password is stored both encrypted as mentioned, and hashed, so that logins are done by 2 hash checks.

Store the password hashed in the "production" database, and keep an encrypted copy in a separate database hosted on a service that is responsible for sending out emails relating to password reminders (or resets). All the frontend services can do are to validate that a supplied password matches (through hashing) or request a reminder or reset for a particular user; nothing else should be possible since the encrypted version is kept out of reach.

Sending the actual password to the end-user via email in clear-text is stupid. The end-user will likely go "ohh, right" and keep using it. Much better to send them a random one-time use password or a link that allows them to reset the password once.

Where? I've been a DH customer for 5 years and I've never been able to recover a password, only reset it. You can see the password when you first set it (i.e. it just confirms that you've chosen XYZ as your password), but after that you can only reset it, which would lead to the conclusion that it's hashed.

The "forgot my password" link on the webpanel login page (discovered today by virtue of needing to log in to set user passwords again).

You are right that for users within your webpanel account there is no email reset option - you log into the webpannel to set these passwords.But the webpanel account itself - passwords are emailed in plaintext.

Not a wordpress database. Upload a rootkit to DH, and you can read any file on the system as if you were root. It has nothing to do with wordpress other than that being the mechanism by which the rootkit was dropped into DH's servers. The rootkit runs with the privileges of the web server, not the user. You can read all the files in/etc

I actually think it's a big deal, but not for the reason most people are crying about.

It's a bit deal that they have been open, honest, & cautious about the intrusion. Having seen so many high profile companies take the opposite stance lately, the DH intrusion should be made a big deal of, if anything, to show other companies how you react to being hacked without losing face with customers.

For me, there is only one chance when it comes to security to get it right. If you try to hide intrusions, lie to customers, or stonewall tech sites trying to get more information, you aren't a company to be trusted with my data.

Let me second that. I got the email, checked into my dreamhost account, used the excuse to call my sister (and will have a conversation with someone else). and then I was done with the *protective* aspect. Actually, the protection happened right away because dreamhost locked the possibly-compromised accounts immediately, as I understand it. The *recovery* aspect, then, just took a few minutes, and involved an enjoyable family chat.

Had I found out months later, that hackers had compromised dreamhost, and that dreamhost had kept it quiet, I would have been an unhappy customer. As it is, I'm a happy one.

I'm happy for all the DH customers that this's turned out to be little more than "HEADS UP", and it appears DH went out of their way to handle the damage correctly. Great! Bravo!

However, as an IT guy interested in system security, I *hate* that this happened in the first place, and seems to happen far too often regularly. Why is FTP still being used, and why don't you guys know how powerful a shell account can be in the hands of a master (or a gifted amateur, for that matter)?

1.) few people upload sensitive data to a web hosting service.2.) it requires less CPU overhead.3.) FTP transfers, while better with a client like Filezilla/Cyberduck/xFTP, don't *require* a client since both Windows and OSX support it natively.

The problem with FTP isn't that it transmits data as cleartext (though it does that too). The problem is it transmits passwords as cleartext. Anyone snooping on your FTP session will know your username and password.

"Shell account" is why I'm with DH. I upload everything using rsync-over-ssh. It's not like those other options aren't there.

Yeah, but stuff that was determined more than a decade ago to be inherently insecure should not be supported and ought to be turned off. Make the users happy, sure, but try to keep them from shooting themselves too, yes?

DH customer: "Why can't I get to my ftp login?!?"DH Tech Support: "Because we can't know whether your box is infected with a keylogger trojan (or worse) or if your network connection's being sniffed. Please use the much more secure ssh & etc. You'll prefer it once you get to know it,

Here's an example of somebody getting it badly wrong but little press about it.
A few weeks ago Telstra Bigpond, one of Australia's largest ISPs, was caught out with the utterly stupid situation of having their customer list of username, plain text password, email address and mailing address out there naked on the internet. Outsourced workers in call centres needed the information but some idiot decided instead of them having to log in somewhere to get access that they should simply be able to use a URL with the customers username on the end of it. The site with the passwords was still up ten hours after it hit the mainstream news.Now that's the sort of thing I expect when I see something like the article summary above, but instead it's the opposite - full disclosure early instead of being caught out by the press and not plain text passwords.

Brian H. from Dreamhost initially posted on the Dreamhoststatus page that FTP/SSH passwords are only stored hashed. Later he deleted that statement. Why?

Web panel passwords are definitely stored in a retrievable way, because when you forget your web panel password they mail it to you. Not a nonce key that allows you to set a new password, they mail you the actual password. According to Dreamhost CEO Simon Anderson [dreamhost.com], they're now evaluating if they could

Well, it *IS* a big deal, but only for people who are using the same password on dreamhost and other services. Obviously, people shouldn't do that, for reasons that are now obvious, but people do. Whoever got this password list is likely to start looking at facebook and other sites for accounts with similar names and use any passwords they can crack from this database.

The compromise is sometimes not the obvious one... For example, I had an account on a service that was recently compromised, and that acco

Except that they found a symptom, and not the actual problem. Someone has unauthorized access to their servers. Until they figure out how they've gotten in and closed the door, it's pointless to scramble passwords. This also wasn't a "quick response" as people have been complaining about their accounts getting hacked and their WP configus and.htaccess files getting modified for months.

As a dreamhost customer (who doesn't store anything overly sensitive there), I've noticed that they also have a tendency to send me mail with my real password in the past, which indicates that it's stored somewhere in cleartext (which is BAD).

I'm not sure that's quite true across the board. According to a blog comment here [dreamhost.com] by Simon Anderson (dreamhost CEO):

Zachary:- some more detail - our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted. We've now confirmed that there are no more legacy unencrypted passwords in our systems. And we're investigating furthe

In other words, it's really not that big of a deal. The database shouldn't have been compromised, and I'll expect a full postmortem of how they screwed that up, but in terms of damage (or even inconvenience), there really isn't any to speak of.

I run dozens of web sites on DreamHost with almost as many shell accounts associated with them. Going through all those account and assigning new passwords to them, then reconfiguring my development tools with the new passwords is a major undertaking. I think that qualifies as an inconvenience (especially since the password change sync seems to be taking much longer than usual right now).

However, given what had happened, I wouldn't have it any other way. I'm glad they recognized the problem and responded

I'll see your SFTP and raise you disabling password authentication entirely, and using SSH public key authentication only.

If your SSH server is visible over the Internet, you should use public key authentication instead of passwords if at all possible. If you don't think it's important, try logging all of the malicious login attempts you get for the next week.

I'll see your SFTP and raise you disabling password authentication entirely, and using SSH public key authentication only.

I do this on my own servers but I don't use plain file transfer at all. Instead I use a distributed version control system (mercurial) and I push to the server. Mercurial lets me define a hook to update the remote copy to the repository tip when new changesets are pushed to it. Working this way I have a full version history at the local and remote end. Additionally I only have to manage the directory tree locally. The remote end is taken care of.

Another advantage is that mercurial hashes the whole repository so if anybody does fiddle with any files, I hear about it as soon as I touch the repository.

Your suggestion is actually even worse. A passphrase or password of some kind shows that there is a human being that should be allowed in instead of whoever just happens to possess something with that key.I think you've misunderstood some good advice of using BOTH a key AND a password/passphrase. The quote you've used has left out the point that typically when you generate a key it also prompts you for a password/passphrase. Passwordless keys are useful within internal networks but are a very bad idea fo

Nobody implied that you shouldn't encrypt your private key with a strong passphrase.

Apart from some guy under the handle of "sakdoctor"?

disabling password authentication entirely, and using SSH public key authentication only

It's not a "reading comprehension" thing if you state the opposite of what you now say you intended.My point stands - as written it's very bad advice to do it all with keys and no password authentication with the keys.What the fuck do you expect readers to think you mean by "disabling pa

That'd be nice, but for some reason it still seems to be the standard to support FTP for web hosts. My host, fatcow,com, does so as well. Why, I don't know; I'm guessing it's probably because a bunch of people use shitty website-authoring tools that don't support SFTP. I wouldn't be surprised if MS FrontPage is the big reason; my host supports FP and FP extensions, even though last time I checked, FP is defunct and unsupported, and has been replaced by some other MS tool with a totally different name. B

rsync only works if the hosting service supports it, IIRC. My hosting provider, fatcow.com (owned by Tucows I believe), does not seem to support it from everything I could find in their help pages, only FTP and SFTP. rsync would be much better because it's really fast and works really well, but I guess most hosting account customers are clueless about it, probably mostly being Windows users using some sort of commercial web authoring software, and rsync being mostly a Unix-only tool it seems. I ended up

I'm pretty sure it's the same password for both. Inside the control panel there's a popup to assign each user "ftp", "sftp+ftp", or "shell+sftp+ftp" access. But if you choose either of the latter two, you have "disallow ftp" checkbox. Fairly bassackwards in my opinion, but does let you block ftp into your account - one user at a time.

Content management software from commercial vendors lack it. I guess they assume you have your own ftp servers in your big corporate office. They forget not everyone has the luxury of 10 t1s for a good network connection and their own IT management and servers in the same building. Most small to medium businesses use an ISP.

Is that a serious question? If so, how about/usr/bin/ftp just for starters? And I imagine c:\windows\system32\ftp.exe as well, though I don't use toy operating systems myself. Those are by far the most common ftp clients.

This has been going on since last June [dynamoo.com]. Dreamhost were completely unresponsive to reports that their services were being abused. Hey, it only took 'em half a year to figure out there was a breach..

It got so bad at one point that I recommended that readers of my blog . [dynamoo.com]

This has been going on since last June [dynamoo.com]. Dreamhost were completely unresponsive to reports that their services were being abused. Hey, it only took 'em half a year to figure out there was a breach..

Probably because that has all the hallmarks of a software PHP vulnerability web-hack of a site, NOT an FTP compromise. I've seen plenty of those, they use some vulnerability to gain access, then upload a file (through the web software) that gives them what's basically a PHP web-based shell. There's no need for the FTP account password to be compromised (and it usually isn't).

All web hosting companies get a lot of that type of attack because their customers don't all update and/or secure their sites properly. WordPress is a particularly popular target.

The entire point behind using a hosting provider and not just running your own server is for them to deal with shit like that.

No it's not. It's like saying it's a car rental companies fault if someone steals a rented car and crashes into a school bus. All a hosting provider does is provide the hosting environment, after that it's up to the customer to deal with things unless they pay for some specific service.

If I recall correctly, the traditional way to manage WP was by the customer installing it. Then DH introduced one-click-installs to make it easier to install and update, but the customer was still responsible for updating since an automatic update might break their site. And I *believe* (but don't use) there's a way to use a single DH managed instance of WP. This is less flexible - I think you're limited on themes and plugins - but is managed by DH.

Any admin that has to manage a large number of machines with common authentication is INSANE to use/etc/passwd or/etc/shadow. It's FAR too much work, FAR too much trouble, and will cause FAR too many problems.

I'd bet the farm they're using some combination of Kerberos + LDAP. Kerberos+LDAP is the standard for holding the authentication and user information - even Microsoft uses it in their Active Directory.

PAM = Pluggable Authentication Modules. This means you just use a Kerberos+LDAP plugin for authentic

As a long time Dreamhost customer, I have never encountered an incident like this before. However, I think it was a good decision to reset the passwords as it was a painless process both for them and for their users. In any case, I'm bracing myself for any aftershock.

I disagree that the process is entirely painless. As a developer and reseller, my account has several dozen different sFTP users set up. Each of them required a password change. Easy? Absolutely. Frustrating? A little, but I'm really glad DreamHost chose to play it safe. Painless? Not really, since the process took over an hour and involved me emailing my users to let them know why they couldn't log in with their old user/pass combinations.
In all, I prefer having to spend an hour or two changing passwords

If you're reselling to several dozen users, then that would be a different story. I could be totally wrong about the nature of reselling through Dreamhost, but why did you have to individually change their passwords? I was assuming you could've just sent an email blast to your users providing some "explanation."

Why do you just assume the passwords are cleartext? Do you not understand that even hashed or crypted passwords are easy things to crack?

There are numerous programs specifically designed to crack passwords. They start with the obvious things (dictionary), then adds in misspellings, moves on to simple substitutions (1=l, 4=a, @=a), and finally to more difficult things, like multiple words+substitutions. Many crackers combine clustering with GPUs, which makes for highly effective cracking.

Dreamhost seems to have a surprisingly participatory workplace, runs on a free software stack, and has an environmentally-friendly setup. I've set up many sites with them and have tried all the other big hosts, and I now recommend only Dreamhost to friends/family/coworkers.

Their support is very responsive, and the occasional technical hiccups with hosting packages are handled quickly and professionally. Although this break-in is a bit scary, it seems like they're playing "better safe than sorry" by res

some more detail – our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted. We’ve now confirmed that there are no more legacy unencrypted passwords in our systems. And we’re investigating further measures to ensure security of passwords including when a customer requests their password by email (this was not the issue here, though). Re your shell accounts, I’d suggest that you select a new password just to be sure.

Search for "January 21st, 2012 at 11:55 am" at this link [dreamhost.com]

Also, due to the number of customers changing their passwords, the password sync time is very slow right now. More info here. [dreamhoststatus.com]