As many of you know, last week we experienced a major outage, affecting multiple email and database servers for just under 5 days.

It was, without a doubt, the worst technical issue we’ve ever experienced as a company.

Thankfully, all systems were brought back up without any loss of data. Although we’re happy that we could restore all your data, we know the duration of the incident, as well as the incident itself, is unacceptable.

To make certain that something like this never happens again, we launched a full investigation to determine the root cause of the issue, outline the steps we took to handle it, and help us take preventative measures against this type of problem in the future.

I want to share this report with you, both to satisfy the curiosities of our more tech-savvy customers, and to illustrate the amount of time, work, and research it took to resolve this difficult issue. I also want to share the steps we’ll be taking to prevent this issue in the future, which are included at the end of the report.

So, here is the post-issue follow-up my system administrators delivered to me this morning. I think you’ll find it informative, and I hope it sheds some light on the mess that was last week:

Incident Name: Storage Outage – sas3

Incident Date: 2014-03-02
Report Date: 2014-03-14

Services Impacted:

Incident Root Cause:

The affected SAN is comprised of an array of drives in a RAID 50 configuration that consists of two RAID 5 groups (one parity group consisting of the even numbered drives and one parity group consisting of the odd numbered drives). The array can handle two disk failures at the same time as long as they are not part of the same parity group. In the case of our outage, drive 6 failed and drive 10 was added to the RAID to rebuild the group. During the rebuild process, drive 0 failed causing us to lose the even-numbered parity group. This occurred just before 4AM EST on 3/2/2014 and caused the RAID to go into an unrecoverable state. We contacted our hardware vendor’s support line before acting, because there was a large potential for data loss, and were escalated to their engineering team. Total call time was 10 hours.

Response:

In order to regain access to the data, we had to manually disable slots 10 and 15 (the spare drives) so that the RAID would not attempt to rebuild. Next, we reseated drive 6 which brought it online, but not as part of the RAID. This allowed the entire RAID to come back online in a degraded state with drive 0 active. Because drive 0 was still failing, we knew the RAID was in a very fragile state and that we had to move forward with great care or we would risk losing data.

Our hardware vendor showed us a binding procedure that allowed us to move the affected volumes from the storage system. We learned that if we triggered another failure in drive 0 at any point during this process, the RAID would go offline and we could lose access to the data. With this in mind, we began to migrate the volumes, one at a time (that way, it would reduce the stress on drive, thereby reducing the chance of it failing again). We were methodical and deliberate in the way we approached this and thankfully, we were successful in migrating all data from the storage without triggering another failure in drive 0. The process completed, and all customers were back online as of 3/6/2014 just after 6PM EST. The whole process took almost 5 days.

Timeline:

You can find a timeline of events on our status blog–from the initial outage to the final server’s reactivation.

What We’re Doing To Prevent This:

Improve Monitoring

Currently, our automated hardware checks are set to notify us when a storage system has an issue of any type. While sophisticated, it’s not specific enough to tell us what the actual problem is. For instance, if a drive fails, we get a general notification, rather than a ‘drive x has failed’ message. We are looking into using more specific, granular notifications for individual disks.

Proactive Hardware Replacement

It may be possible to check via SNMP for things like disk errors on specific disks before they actually fail out of the RAID and trigger a rebuild. This will result in less drive failures and less rebuilds.

Switch All Arrays to More Stable RAID

Our RAID currently rebuilds on storage arrays using RAID 50. Although this is standard, it can take more than eight hours to complete a rebuild. This is an 8-hour window where we risk losing two drives from the same parity group. We can decrease this risk by moving to a significantly faster RAID 10 setup, which can rebuild in about 3 hours.

Thanks for reading and again, we’re so sorry about this inconvenience. If you have questions, feel free to ask them in the comments.

Conclusion

Along with the steps we’ll be taking to improve our hardware, we also need to work on improving our speed and accuracy when it comes to identifying affected services and affected customers. This information should then make it to the blogs quickly so the customers know we are aware they are affected, and that we are working to fix it. The faster we can do this, the easier it is to pinpoint problems and implement fixes.

During this current outage, we took too long to get detailed technical information (especially ETAs) communicated to you. We think it’s better to get some good information out – even if we have to admit that they’re rough estimates and it’s the best we have right now – than to delay providing any real information. Many of you agreed.

Now that we understand the details of this issue, we can sleep a little better knowing that this won’t be likely to happen again and we’re all on the same page about what went wrong. We’re incredibly grateful for your trust, and we will continue to work tirelessly to regain and keep it.

For most of us, it’s a time of making promises to ourselves that get seem to get harder to keep as the year goes by… so instead of that, I resolve to focus on what makes happy times even happier. (Hint: it’s not alcohol.)

It’s gratitude.

I’m not recycling an old “Thanksgiving” email about being thankful here. I will make this new year the happiest it can possibly be and I’m taking you with me.

This new year comes with a bunch of awesome new stuff; new products we’ve worked really hard on like our Domain Club that will save you money, great projects that will continue to improve our hosting services (I even hear whispers of a new hosting control panel, “zzzz…. cPaaahhnellll…. zzzz” in the ice-cold winter breeze), and our new account control panel, which you may have already seen.

So, my ambition for 2013 is to be happier. Happy people make better partners, parents, business owners, thinkers, and well, people. The way to get there is to be grateful for everything you’ve got and to constantly search for things to be happy about. If you can spend your time focusing on the positive around you, you will influence your world to be more positive toward you.

I hope you start searching for all the good in your life until it just becomes obvious on its own and all your struggles appear smaller by comparison. There is a lot to be grateful for, in anyone’s life, and it is all too easy to forget.

We’re incredibly grateful to have you as our customer and partner in doing better things with the Internet.

I hope this year will be a happy one for you.

What’s your New Year’s Resolution? Any specific ones in regards to your business or website?

—FathiHappy IX Leader

PS: I’m really interested to hear what you guys think. Put your answers in the comments and we’ll try and answer each one – especially if you have any confusion about what you can do to reach those goals with your site.

PPS: So, check this out. I got myself this really cool Christmas gift called “prism glasses.” They’re awesome and totally weird, but I love them. Who knows, you might like them too!

Also, if you want to read more about gratitude and happiness – I recommend you check out “The Happiness Advantage” book. I will be using my new prism glasses to read it in bed, but don’t judge the book by that. It’s actually good and has inspired me in many ways!

So, yesterday my mailman brought me yet another one of those Domain Registry of Amer*** letters. These letters are seriously annoying. Take a look (click below to see bigger image) :

If you didn’t already know, these letters are a notorious marketing ploy. They try to convince you to renew your domain name with them by sending you an official-looking notice, by postal mail. If you follow the directions, not only will you pay an arm and a leg for a year of registration, but you’ll also be transferring your domain registration to another company. I often get letters from customers, who ask me, “How did these scumbags get my snail mail address?” It’s actually really easy. And sort of scary, too.

Your domain name’s ownership information (it’s also called the “WHOIS information”) is available in a public database – which can be accessed by anyone. See for yourself right here.

This is the same reason you usually started getting boatloads of spam to the email address associated with your domain.

Obviously this is bad. Only for specific reasons would you want to share your info (and I’ll tell you about those in a couple of weeks). But as a general rule, it’s better to hide your contact information from public view.

So how can you hide your contact information?

Until now, there hasn’t been a lot you can do about it. But because so many of you have been asking for it, we’ve added a sweet new feature called Domain WHOIS Privacy. It basically removes your information from the WHOIS database. You still own your domain, but now you won’t have to deal with all the spam, fake mail, and telemarketing calls. Of course you always have the option to turn privacy off at any time, but I don’t recommend it.

I’ve got all my personal names, and even some of my business names, protected with Domain WHOIS Privacy. It works wonders and the spam you get decreases considerably.

Here’s the details on the five-featured knock-out punch for your domains. All five fall into the “you’ve been asking for it forever” category.

You would not believe the amount of emails I get from my customers asking about these features. They’re finally here. Let’s get right to it: (Drum roll please…)

Number 1: Domain WHOIS Privacy

When you register a domain, you fill out your name, address, phone, email, etc., right?

I’m not sure if you know, but this information is PUBLIC. It’s stored in the so called “WHOIS” database, and anyone, including bad guys (like spammers, identify thiefs (or phishers), and other online fraudsters go there to get the goods on domain owners like you and me.)

There’s actually a close-to-home story that I have of how one my colleagues almost got shafted because she didn’t have her domains protected. I’ll tell you about it later this week.

The only way to solve it: Domain WHOIS Privacy – it hides your information and tells the trolls that it’s off limits!

Number 2:.ME and .NAME Domains

Have you ever wanted to grab a personal piece of web real estate for yourself, your children, or grandchildren?

We’re talking .name and .me domain names. Very cool. They’re the perfect way to put your unique, personal stamp on the internet, and get a really sweet email address. Over 14,000 domain names are registered every hour, so act fast if you want to secure your name.

Number 3: Internationalized Domain Names.

Did you know that domain names can be registered in non-English language characters?

If you get traffic from other countries, your should snap up your domain in those languages and redirect the traffic to your site. Right away. Why? Because it will help your site rank higher in Google, and it’s likely that you’ll start getting more traffic.

If you don’t know how to spell your domain in the language you’re looking for, simply visit http://translate.google.com. Plus, it’s easier to copy/paste than try to figure out how to make a ¨, Б, or 买 using your keyboard.

Number 4: Premium Domains

I’m sure you’ve run into this problem before. You think of your perfect domain name. Alas, it’s not available anymore. Just like all of the other good domain names – all taken. But really, a lot of them are for sale! And with any luck, the name you really want is too.

They’re called “Premium” (or pre-registered) domain names, and they’re a great way to get the exact domain name you want.

But, like all good things, they’re not cheap. They can get pretty pricey – but you get what you pay for. Premium domains are often a big source of immediate direct traffic, built-in SEO rankings, and a reservoir of pre-existing backlinks (and all that stuff is really good for ranking your site higher on search engines.)

Take a look here to see if there is one you like, or if your perfect name is actually available!

Number 5: Easy Domain Transfer

For years, both my customers and my customer service specialists have been beating me over the head with this little tidbit: It’s a pain in the neck to transfer domains into our system.

So finally, we created something new: An automated tool for domain transfers that lets you extricate your domains from the grip of your former registrar, and transfer them quickly and easily to your account, directly in your control panel.

You can do it fully automated, and it doesn’t require any phone calls to our team. So transfer away!

That’s it for now. We’re all very excited about these enhancements. I hope you will be too.

We’re starting the first of a series of book reviews. Today’s book is one that is meant for those that have the desire to start their own revenue generator but have not had that much experience with their own online business before. It’s meant for people who have the drive to turn an hour a day and every day experiences into six-figures. Specifically, it’s about creating a profitableinformation product.

This book hits many points that I started into in my email to you about PayPal last week. In short summary, this book walks you through the process of creating an “information product” that will solve other people’s problems (as it perhaps solved yours), creating a website around that product and then marketing that website so people know you exist and actually buy from you.

Keep in mind, this book is for the development of a profitable information product, but the process he discusses for marketing to your buyers is critical to any business. If you’ve never done email campaigns, news and sales letters, autoresponders, “launches,” pay-per-click marketing, etc., this book might just be a very valuable learning tool for your business.

Jon has decided to give a bunch of bonuses to those who purchase his book on Tuesday, September 21st. The biggest bonus is that they’re giving away a free seminar where he and the co-author, David Lindahl will answer your questions, discuss topics from the book and give you the opportunity to network with other like-minded people. The seminar is just outside of Boston on Friday-Sunday, November 5-7. If you’d like more information, just shoot an email over to questions@sixfiguresecondincome.com.

I had my fiancé read it first because she’s a really fast reader and extremely opinionated, but when I saw it was taking her longer than one sitting to read it, I thought something was wrong. It also always seemed to be opened, face down next to her rather than…well, finished, so I started to worry that Jon and David had produced a dud.

Instead, I saw that she was working on many of the points Jon had written about with her own business ideas and was pausing only to be inspired. This is exactly what you want out of a book.

We have both read many similar books, but none were as realistic and thorough. This book literally walks you through the simple process of making money online without working that hard, but doesn’t make the mistake of promising that by the last page, you’ll be a millionaire.

You definitely can become a millionaire following his exact steps and Jon and I know many who have, but it’s not realistic for everybody. It is, however, important to note that many of the bonuses these guys are offering include audio interviews and advice from the guys who have had 7-figure+ success with this process, so no one wants you to think that it’s impossible.

“I just hope you don’t have some moral or religious aversion to making money without shedding any blood, sweat, or tears. (Page 14)”

Contact Us

We're Social

We're Always There When You Need Us The Most!

Your Dedicated Support

At IX, we take care of our customers. And dedicated support is one of the ways we prove to you again and again that we are here to help you every step of the way, regardless of your skill level. With IX dedicated support, you get a support technician personally assigned to assist you. You get their name, number, email, social media connections, and work schedule! It's just one more facet of our service which proves our deeply rooted belief that being a great hosting provider requires more than just cutting-edge technologies, but the best in support and service.

The Hello Bar is a simple notification bar that engages users and communicates a call to action.