The anatomy of a security breach, and how to do good in a bad situation

On Tuesday, iThemes posted an announcement that they had suffered from a security breach of their website and servers. The attackers had reached the servers which stored customer information, including email addresses, IP addresses, full names, and yes, passwords.

iThemes was quick to notify customers via their blog, social media, and their full customer email list about the breach. Approximately 60,000 users were affected. They warned that passwords were vulnerable. In the second update, posted today, they gave more information about passwords, in response to many questions from users.

It turns out that passwords were stored in plaintext on iThemes’ server. That is, obviously, very bad practice.

Why Would You Store Passwords in Plain Text?

This is how the membership software we started using in 2009 did it. There are a number of factors for this, none that will make much of a difference at this point or make anyone feel any better about it, myself included.

Know that it’s not because we did not value your data. As an organization, we have been working on a very large migration process that has required us to interlink legacy systems with the latest technologies. Anyone that has ever gone through that process understands the complexities and challenges.

Frankly put, it’s been something we identified as a potential risk and are working rapidly now to rectify this issue as fast as humanly possible.

It’s also worth noting that their customer database and iThemes.com users were affected, but customers that use their Sync product to manage their own websites were not. So if you use iThemes Sync, and utilized your site passwords to connect, those accounts and passwords were not part of this breach.

aMember and legacy membership platforms

The membership platform that Cory highlights in the update is aMember, a membership management system that’s been around for many years. aMember only introduced encrypted passwords in version 4, which was released in November of 2011.

I discussed aMember and plaintext passwords with some other folks that have a significant history with the membership platform, and there are some significant problems that anyone using aMember have experienced.

First, most folks heavily using aMember aren’t using it out of the box. At the time, most membership sites were doing significant customizations to aMember to achieve desired functionality. So when the v.4 update came out, it was a very difficult update procedure for people to take advantage of the features.

iThemes would even tell you that their current version of membership software doesn’t look much like aMember at all.

I discussed aMember at length with Pippin Williamson. He has done a lot of work on his brother’s membership site, CGCookie, which also used aMember until 2012, when he did a huge migration of tens of thousands of members to a new platform.

At the time, Pippin notes that aMember did not disclose passwords were stored in plaintext, so CGCookie had no idea that their users were vulnerable until they learned of the Tuts+ hack, wherein they put a planned migration “into hyperdrive.”

The problem with iThemes’ situation is that they knew of the plaintext passwords and didn’t address the obvious security vulnerability.

All in all, the migration for CGCookie took months to perfect and significant juggling of priorities by their team.

Ticking time bomb

Speaking with Pippin, migrating from aMember was not an easy task. Paypal’s IPN handlers (a payment notification system) were tightly linked to aMember and preventing customer accounts from being disconnected from the membership site took weeks of engineering. Additionally, simply upgrading to the newer versions was also terrible.

Many other WordPress companies have used aMember in the past as well, storing plaintext passwords just like iThemes today.

So, aMember has definitely been a problem before now, but iThemes has absolutely slacked in their prioritization of the issue. Simply put, it’s inexcusable to put users into long term risk if you know their passwords are stored in plaintext.

That said, we should consider the potential consequences — though it doesn’t make it excusable. Because iThemes doesn’t store credit card details, emails and names and passwords are about the worst things at risk. Still, a tiny percentage of the general population uses responsible passwords, so getting a list of names and emails and passwords is a treasure trove of information for third party (as in other websites like banks and email providers) to steal identities and break directly into user accounts.

This should also be a significant lesson for any user to always use different passwords for every website where you keep an account. Tools like LastPass and 1Password make account password management easy. Additionally, learn about 2-factor authentication and use it whenever the service enables it.

Owning mistakes

This is clearly a horrible situation for iThemes to be in. Users have a right to be mad. In 2014 we should be able to expect that our passwords are encrypted, and that even if a server where our information is stored is hacked, that certain valuable details like credit cards or passwords are not exposed. Thankfully, iThemes doesn’t store credit card details, but those passwords should now be assumed stolen, and users have to pay that price.

But where I have enormous respect for iThemes is how they have owned it. Especially in the update post, Cory Miller — iThemes’ founder and CEO — took ownership of the issue and told the full story of the breach.

I realize this will generate a lot of concern. Again, I am deeply sorry for this mistake and how it has affected you.

Let me say: we have made mistakes in the past at iThemes … and as humans will make mistakes in the future. To make a promise otherwise would be absurd and misleading.

But my promise to you, our customers, is this … and it’s the same promise that I’ve held to since January 2008 when I started iThemes in my home:

We will identify mistakes as best we can. You have helped us with this and appreciate that accountability.

We will own up to our mistakes. Again, we’ve done this in the past, we did this yesterday and this post is another example of us living this value.

We will fix the mistake as fast as humanly possible. A number of priority issues have been unearthed, shone a hard light on, but we are working to resolve them.

We will learn and grow from it and be better for it and for you.

Additionally, as the founder and CEO, the leader of this company, I want you to know: the buck stops with me and me alone.

At the end of the day, I am responsible for our company, iThemes, and the work we do. I’ve often tried to defer credit for the great work we’ve done to our team, but as for the mistakes we make, that credit belongs solely to me.

Not every company that’s been breached can say the same. Even huge organizations (I’m thinking of Home Depot as the latest) have done a horrible job at responsible and honest disclosure to their customers.

iThemes could have made this much more quiet and kept it from being a priority to their users or a big deal in WordPress news-land. But they didn’t. They owned it, they apologized, and they’ve been rewarded for that.

In the comments of the security update post, users were appalled by the lack of responsibility on the security priorities, but the honesty paid off in the sense that users were verbally thankful for it so that they could accept the problem and deal with it, when the alternative was to be in the dark.

While this breach has cost iThemes some credibility, and some trust, I believe those things are recoverable. Had they obfuscated the hack itself, my post and users reactions (had they found out) would be very different.

What iThemes is doing now

iThemes has learned a hard lesson this week, as many other companies have before them. Now they must react, and react strongly. In a situation like this, nothing should take priority over getting this issue fixed.

Until they get their password storage issue fixed, they still are storing passwords in plaintext, even the new ones. Thankfully they’ve already updated password restrictions — which previously limited passwords to 20 characters — to now accept up to 255 characters. But storing them in plaintext means that they are still as vulnerable today as they were last week if someone gets into the server again. So they must be absolutely vigilant to protect their servers through this migration.

What we can all learn

Security is not a sexy industry. Too often we don’t consider it until it’s a problem. “Going on offense” should be our default when we consider security actions, but more often — even with some of the best companies and people in the tech space — we react to issues and aren’t appropriately proactive.

That said, our job is not done either. All of us should consider our own security situation and how we can improve it. Especially if you have users of your own (and many of my readers do), consider that state of your own security, and make sure it’s a priority in your business.

When we think about what pays the bills, it’s not security awareness and proactive security investments. But we should consider security breaches and vulnerabilities as risks that must be managed; meaning investment into security priorities should be a significant part of any business.

iThemes, Envato, and WooThemes are all “big” businesses from a WordPress business perspective. Each of them has fought a website security battle. Envato and WooThemes before iThemes have recovered and maintained their users and the community’s trust. I think iThemes will too. But before you, or me, decide “it won’t happen to us”, consider that it can happen to anyone, and our investments into security measures and best practices are not only worthwhile, but the only responsible thing to do.

And for users, use different passwords on every site. Use LastPass or 1Password. Enable 2-factor authentication. Be vigilant. Do not use the excuse that it’s too hard or a burden. What’s much more of a burden is dealing with a hacked email or bank account. Protect yourself and learn how to protect your accounts. It’s easy to adjust to and modern tools make it a much simpler task to manage.

Let’s all — web professionals and users — learn with iThemes here, and do better.

Now I don’t know anything about security but I do know you never store passwords in plain text. How is it I know that and they don’t? Or maybe they do but decided to say “screw it”.

Now here’s another thing. I’m an themes customer. I specifically paid for two plugins that I can us on all client sites. Backupbuddy and ithemes Secruity Pro. Now I’ve got to wonder if theme security pro actually does anything. How can I know?

At the end of the day, iThemes have been negligent and breached their duty of care to customers.

If someone built a site for a client with such a major fault, they’d be sued and likely bankrupted for their negligence.

I agree with Scott. It’s pretty unbelievable that people are using words like “honesty” and “transparency” – 6 years is a long time to know about this.

The extreme irony about them selling a professional security plugin is a bit too much.

I will still use iThemes and hope this event both gives them the unpleasant reminder to get their act together, and for other companies who may have that legacy system floating around still to get it sorted NOW.

Thanks for putting this in context, Brian. Incidentally, Gmail suddenly flagged this post in your newsletter as containing content that is usually considered suspicious — odd, eh?

Miller did admit and accept a lot more blame than others in his position have in the past, so we’re spared the dance of denial, but what else could he do? He didn’t quite come out and say this was the result of a fully conscious choice to keep doing business so long with a system he knew was bad for customers and bad for his business. He didn’t say this bit of kicking the can down the road was rationalized through denial and “what they don’t know won’t hurt them,” or “we’ll get to it,” and “what are the chances?” The fact is they got comfortable with the unconscionable.

We can’t count on others’ good sense or ethics; there needs to be vigilance, and a permanent record, and people calling a spade a spade to do the work of “afflicting the comfortable.”

“Simply put, it’s inexcusable to put users into long term risk”
– wow, you can say that again. But perhaps with the additional superlative ‘absolutely’ before the word ‘inexcusable’… I sincerely hope some of the other WP-related companies, should they have similar security holes festering, fix them RIGHT NOW!

Transparency (admittance of mistakes) and common decency (public disclosure) is all one needs to win back trust, even when you’ve been negligent (not doing anything about a big security concern you know exists). Better yet, control the narrative by reporting before anyone does, because that would look a whole lot worse.
[/cynical thoughts]

Its a bit more complicated then just “negligence”, I think. A key piece of iTheme’s business infrastructure, which they were planning on migrating/updating from if I understand their communications correctly, was vulnerable and attacked. It’s a real shame the passwords were stored in plain text but apparently that vulnerable business system used to only do plaintext passwords until 2011. So if you started a business in 2008 and you’re busy trying to make money and build up your company, its easy to see how these business systems get pushed to the back as far as time and resources go because they don’t directly make the money the business needs to survive and grow.

I think what this episode really illustrates is to make sure your business infrastructure is easy to replace if needed and to make sure to invest a portion of your business time and resources in updating and maintaining your business infrastructure.

That is the definition of negligence. Maybe the original decision to use aMember was made without due diligence, and iThemes didn’t realize the security problem. Later, once the security issue was known, the decision not to rectify it with reasonable haste looks willfully negligent, especially since they modified aMember so extensively they had no upgrade path, but these modifications did not include a method of storing passwords securely.

I am surprised this record was laid out so clearly by Miller, because these are the facts and timeline that would make up the argument for anyone suing for damages. Openness is not usually rewarded after mistakes have blown up in your face. Where openness may have mattered most is in the original software. aMember is a proprietary application; I wonder how much that fact might have played into its prolonged insecurity.