This is what Linode employees along with the fine men and women from the Galloway police department had to deal with this afternoon – their SWAT team storming the Linode office, forcing everyone out for about an hour while they performed a sweep of the building room to room, complete with an explosives-sniffing dog (who was very happy). They had received a false report which provoked them to respond in this manner – and it’s their job, after all, to respond to reports, even if it turns out to be a hoax. They were great, and I thank them.

Not so coincidentally, about the same time we were made aware that an old personal server had a database accessed using old forum credentials obtained from the incident last year. This server is not under the umbrella of our security team because this server plays no role in Linode infrastructure. Unfortunately, it did have a restore of the phpBB forum database on it from 2010-03-03. Forum users that existed at that time and who haven’t changed their credentials since have had them revoked and will need to reset them. We regret that this happened and apologize for the oversight. We will be discussing new security policies to address scenarios like this.

On the subject of security, last year we stopped all other developments and focused on nothing but security for over six months. We did everything we could think of, from significantly reducing our Internet-facing footprint, to defining, testing, or improving practices and policies for going forward, to third-party penetration testing. We did this until we ran out of things to fix and ran out of ideas to pursue, and our security team continues to proactively assess our infrastructure and services. This was a monumental effort and a story that deserves to be told, but these efforts and their outcomes belong in a post of their own. Stay tuned.

We know how important transparency is and how we’ve needed to do a better job with it in the past, and well … this is the story.

26 Responses

Basic rule of computer security – there is no such thing as a “low risk” machine. In corporate environments (eg banks) all desktops, all laptops, all services (even DEV machines) need to be secure. If it’s on the network then it’s a risk and needs security. If it was on the network and isn’t any more then it needs proper decomminissioning (disk wipe, etc).

Similarly backups; any backup tape needs security; any machine that has a backup restored to it needs security.

“Basic rule of computer security – there is no such thing as a “low risk” machine.”

The rallying cry of amateurs everywhere.

You have to prioritize, or you’ll get nothing done. At most, you might say that the server should have been under the remit of the security team, but the end result may have legitimately been the same: “this machine is not a major priority due to the low potential impact of an attack against it”.

Data needs to be classified. Even the lowest classification of data isn’t allowed out of the bank. Production data _never_ goes to UAT or DEV. You segment and firewall. Restore prod data to another machine and that server is now classified as PROD.

Servers with customer data on it (even if it’s just a forum) is considered holding PII data and is under higher scrutinity.

Every single machine is behind a firewall. BYOD _is_ on a segmented network.

Every single server (approx 100,000 servers) has centralised monitoring and controls. Every single desktop (approx 250,000) is locked down. You plug your own device in… disciplinary action.

Yes, we’re a bank; we have to take this shit seriously.

I’ve been doing this for 20 years; the one thing I ain’t is an “amateur”.

All that said; yes you need to prioritise; this is “high risk”, this is “medium risk”, this is “low risk”; “this is a risk we accept because the probability is low (firewalled; controlled access; etc) and the consequences are minor (developers can’t work for a day)”, “this is a risk we’ll fix in 3 months (higher chance of it happening, no customer impact)”, “this is a risk we’ll fix tomorrow (shit, panic!)”.

BUT caker wrote “We did this until we ran out of things to fix and ran out of ideas to pursue”. I gave a tonne more ideas to pursue. Security never stops. If you run out of ideas then you’re not doing your job as a security professional because there are _always_ more things to do.

>This server is not under the umbrella of our security team because this server plays no role in Linode infrastructure. Unfortunately, it did have a restore of the phpBB forum database on it from 2010-03-03.

So is it normal that Linode takes database dumps and puts them on servers outside of the Linode infrastructure?

If I worked at Amazon and took a database dump of one of their systems and loaded it up on a server outside of their control, I’m pretty sure I’d be fired.

Banks are different in that they have so much money efficiency is not an issue. It’s not a business model you could copy in any another line of business. Least of all the innovative business of cloud servers 🙂

When you do security there are no such term as “low risk”. Example – Linode.com. Somebody install phpBB and Linode just got two step attack. Get out everyone by falser report Attack old forum – get it. So, questions are: does Linode employes doesn’t

Chris,
I know you guys are relieved that the SWAT visit turned out to be a false alarm, but the alarm your users feel about the data breach is very real.

I’m concerned about the flip and casual nature of your dismissal of the database that was exposed, as well as the fact that it was exposed at all. The “oh, BTW, this happened too, but it’s not a big deal” attitude IS a big deal to us.

I’m sure you understand that we have to have confidence in you the same way that you have to have confidence in your service providers. How comfortable would you feel if one of the service providers you absolutely depend on posted about a data breach in this fashion?

If your attitude toward security mirrors the overall tone of this announcement then we all have something to worry about.

Check every keyboard and keyboard cable for keyloggers. Check the insides of every computer for anything that should not be there. Check every network port in the wall, under the floor, and in the ceiling for signs that anything has been put in there. Check every inch of cable you can physically get access to. Then reflash the BIOS on every machine they could have had physical access to. That includes printers, switches, and anything else that has an IP stack.

The NSA has shown they are willing to go to extreme lengths to steal data and Linode holds quite a lot of foreign data.

This rather obvious playing down of a data breach is very worrying. Linode’s history regarding security is not exactly good, add on to that a flippish attitude toward a security breach, and you have me seriously doubting my choice of provider.

Sorry, not to minimize your credentials, but I’m guessing that the people who know best how to secure the Linode systems are the people who work with them daily. A bank security specialist and the NSA security gurus probably have different tasks (which have almost nothing to do with securing Linode), but what kills me – is the morons who are criticizing the police officers who had nothing to do with the security breach at all.

@Samson: “heavy handed” police response? Wow – you must have known it was a false report before the police even did – so you can sit back in your chair like you know what you’re talking about when it comes to policing all, and not actually providing any solutions to your perceived concerns. Way to go champ.

@Freedom: Your comment is just way under-supported with actual fact. Love the expert critique you must have on the swat teams though. Sounds like you are in one to know these details. Actually, the photo does not even look like a swat team to me (class A pants, no swat identifiers, etc), probably just a bunch of cops who happen to be on-duty at the time of the call, some who may be on a swat team also.

@Linode: Thanks for the information. Most Internet companies would not have even bothered to track down a security problem like this or disclose it to their customers. I think this response was appropriate and justified. Keep up the good work! We all know Linode is a highly visible target.

-My “credentials” include 21 years of IT Security at a multinational ISP, 9 years on a SWAT team.