Linode administrators have discovered and blocked suspicious activity on the Linode network. This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed.

We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems. We have implemented all appropriate measures to provide the maximum amount of protection to our customers. Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset. In so doing, we have immediately expired all current passwords. You will be prompted to create a new password the next time that you log into the Linode Manager. We also recommend changing your LISH passwords and, if applicable, regenerating your API key.

The following represent best practices in creating new passwords:

Avoid using simple passwords based on dictionary words

Never use the same password on multiple sites or services

Never click on ‘reset password’ requests in unsolicited emails – instead go directly to the service

We apologize for the inconvenience. If you have any questions, please do not hesitate to contact our support team at support@linode.com.

34 Responses

> Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset.

Caution is always appreciated, but taking such a drastic measure only makes sense if Linode was compromised to a more severe extent than just one single customer getting his virtual machine hacked into.

If Linode truly values its customers confidence in its services, it should disclose to what extent the intruder managed to escalated their privileges, and to exactly which services these privileges gave him or her access to.

Firstly, I appreciate the openness of you about this, although I would be interested in the exact details in what happened, I know that you probably can’t tell us. It’s probably just a dictionary attack on the site or something. Resetting everyone’s password is probably overkill, but better safe than sorry, I guess.

Anyway, there is no reason to not trust the email, if it turned out to be a spam email, the worse that could happen is that you log into your Linode thing and it won’t prompt you for a new password. The email doesn’t contain any links to the Linode manager, so unless someone wants to send a spam email to confuse people and maybe make them change their passwords; not very profitable.

@TJ Fontain wrote, “””New distribution mechanism, html content, and a security alert — all at the same time.””” Yes, definitely, and you hit the nail on the head more than anyone else.

My own response, after initially simply believing it and thinking to report phishing, was:

“””This looks an awful lot like phishing. Can you confirm to me that the email, which suspiciously did not come from the Linode domain, represents a real request for a password reset rather than “We think your account has been cracked; for security reasons, please change your password to ‘ID10T'”?”””

(Yes, I expect that the more security-conscious will download a copy, inspect their copy, and run it only if it appears to be doing neither more nor less than what it is said to do). It takes three dictionary words, joins them by a digit and a special character, and twiddles a little with capitalization. And it gives you a choice of 10 or so, with a button to regenerate new.

The resulting password, in exchange from being about 20 characters to type, is relatively quick to memorize, and may possibly be faster than typing with the cognitive load of remembering characters in a random line-noise password.

This presents at least concrete examples of good passwords–if you’re paranoid about security and not confident you could detect malicious code in a downloaded copy, you can do just as well by pulling a paper, dead trees dictionary and opening to a random page, placing your finger on the page, and record the word. Repeat three times, join them with a digit and a punctuation mark, and maybe tweak capitalization.

It’s easy enough to say what good passwords aren’t: they aren’t too short, based on a dictionary word, lacking either an uppercase, lowercase, digit, or punctuation character, repeating the same letter more than twice, and so on ad infinitum. Some psychologists say that only telling people what they cannot do is frustratingly difficult to obey rather than saying “Do something like this” and pulling an exemplar. And I believe my password generation page is good at concretely showing what is easy to remember, hard to guess and secure, and may be faster to type than a line noise password that you don’t use all the time (typing a word you know is faster than my typing speeds for remembering my line-noise passwords at least).

You can access my code under the terms of the MIT license, if you want to check my code for malware or simply observe the apparent algorithm and use a dictionary with paper and pencil to achieve the same effects in a way that I could not be pulling something other than what my password generator appears to make.

My reactions were like this:
See “From: .*@linode.com” and “Subject: .* Password Reset .*”.
Ok, it’s a scammer, I think to myself.
Then I saw the email address I use especially for my Linode account as the recipient.
This is getting nasty, I thought, let’s check the message headers.
There, I find “Received: from mc023.e2ma.net” and think to myself “Great, they’ve been compromised…”
More references to “app.e2ma.net” AND “e2.ma” in the message body don’t help either.

I only felt a little relief when I saw there were no “Click here to reset your password” links. After that I decided to check your official channels, which confirmed the legitimacy of the message.

However, I still feel uncomfortable that you shared my email address with a generic 3rd party remailer for such a trivial reason. Setting up Mailman under a linode.com domain name and importing your account database would have taken no more than 30mins – especially for highly technically capable people like you.

I fully expect to start receiving spam at this email address in the near future.

While I agree using a third-party mailer is not fantastic, do not mind the passphrase reset and further appreciate the expiration option. It would be pretty awesome if you guys offered two-factor authentication option for your customers accounts.

While I agree with everyone above for added security with authenticators, SMS, etc. and that Linode’s guys could really have their own mailer, let’s all keep in mind that Linode is on a rampage of new features, I doubt this situation will stay like this. Moreover, their support is of very high quality, fast and personalized.

They even had the decency to do a public announcement about the whole thing. If anything, I feel even more secure than before

Really wish Linode would support 2-factor. It’s trivial to integrate with Twilio to send a 6-digit code to a mobile device. Even I’ve done it. C’mon guys, seriously. One week’s worth of your web developer’s time will pay serious dividends. I’d even be willing to pay for the feature.

James – Google Authenticator is merely an implementation of the TOTP standard. When Linode implements this, you will be able to use any TOTP client you’d like, Google Authenticator, Yubikey, etc.

I should also mention that the Google Authenticator app has **no** network communication capabilities. It runs completely standalone, and in fact, it will happily run on an iPod touch with no network connections. The algorithm only needs an accurate clock to function properly. So while your paranoia about google may be justified for other reasons, this should not be one of them.

Turns out I had just been reading the news on Krebs’ site, as well as /. and a few others, about a wave of WordPress brute force attacks from a growing botnet of WordPress servers. Third party mailer or not, obviously I’m going to check things out when that mail comes in.

Turns out the suckers have set their efforts to my server as well. Fortunately, I’m not dumb enough to leave the default password or use a common password, but all this was enough to make me toughen it up a bit.

So folks, maybe Linode could have handled this differently, but at least they handled it. And though it wasn’t related to my issue, at least I got a heads up and now I know to keep an eye on my server and watch for any developments on the WP botnet.

Most people would have expected a follow-up on the latest rumours. However “We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems.” stood out to me as unusual when I first read it a few days ago. It suggests that there may be reasons why Linode cannot make a statement at this time.

Certainly Linode’s public handling of the issue could improve, but use your paid support option if you actually have a need to be addressed. So open a support ticket, ask your questions, and if you can’t be satisfied by the support (which, reiterating, you are paying for) then make your decision to stay or move.