Editorials

Game Over: Our 1&1 Nightmare That Shut Down the Site

Starting Monday night, and through all of yesterday, a few of you may have noticed the site was down. In fact, I received numerous complaints, and to say that I wasn’t thrilled with this development would be an understatement. This post is to explain, in as clear of a way as I can, what happened, how it was fixed, and how I intend to make sure that never happens again.

For more than a week I had been able to sustain great traffic on the website. Traffic that wasn’t originally expected on my original domain name server. We were starting to notice that some people were complaining about timeout errors, and that the website was loading slower than we’d like. This, of course, is not what we want to see, so we decided it was time to upgrade to a better domain name server. This is a process that’s actually incredibly easy. It should take almost no time, and the downtime expected should be a few minutes tops if at all. Any domain name registrar wouldn’t accept less. Except 1&1, it seems. You can read a lot more about the nitty gritty here.

1&1 was the former domain registrar for all Armed Gamer LLC properties. I had a few domains on there, but none were really active accept Armedgamer.com. My “web guys” as I call them, or really the guys who have made the website look and run as well as it does (they’re called UberMotif and they really are the best I know of) made the change at approximately 10:30PM EST on Monday night. The website almost immediately went down, and I couldn’t access the back end to write more content just as no one could access the front end to read anything.

As a company whose primary source of income is adsense, this was particularly damaging. I had almost 30 hours of downtime which was 30 hours without income. Perhaps more damaging was the complete halting of momentum and the penalization of Armed Gamer links on Facebook. Unlike many sites out there, a majority of Armed Gamer’s traffic comes from social: specifically, Facebook. When all of my links were dead, and people were arriving to a website that was no longer existed for all intents and purposes. This resulted in Facebook’s algorithms kicking in, and not serving content to readers at the rate it normally would. Honestly, that’s actually a good function of Facebook: it serves as quality assurance to stop spamming pages from being well… spammy. It also kind of screwed me over during the 29 hours of downtime.

After more than 12 hours of downtime yesterday I took to Twitter to lodge my complaints. Uber Motif chimed in to help out as well, and also published the very useful blog linked above. After receiving a few retweets, 1&1’s Twitter started responding, asking how they could help. I sent them my account number, but for the most part experienced radio silence beyond a “we’re looking into it.” At about 20 hours of downtime, I received an email from 1&1 support letting me know they had seen the post on Uber Motif’s blog, and they were still looking into the issue, and that I could expect a survey to be sent to me within 24-48 hours. Quoting the email, “It appears the CloudFlare name servers have been set within the 1&1 Control Panel and are currently in the process of updating. Typically DNS changes occur quickly, in less than 3 hours, but in some cases can take up to 24 hours or more. I will be forwarding a case to our DNS Administrators to have this change further investigated to see if there is any way we can push it along to attempt to decrease the downtime for the website. Once we have an update from our Administrators, we will update you via email. I apologize for the lengthy delay in the DNS propagation for armedgamer.com and appreciate your patience in this matter.”

How I feel after yesterday.

The thing is, with any other domain registrar DNS changes take almost no time. Earlier today I changed the DNS of a website on Namecheap. The change propagated within the minute and no downtime was incurred. This exact point was made on Joe’s post on Uber Motif’s blog in his summary. Aaron Nye, someone who handles customer Care for the US at 1&1 responded to Joe’s blog via the comments section, and an email exchange took place before Aaron finally called Joe. A brief exchange took place where it was agreed upon that the DNS changes would go through no later than 4AM (a total of 29.5 hours downtime), but honestly the best thing 1&1 could do for us was to push through the transfer we requested to get Armed Gamer’s domain onto Namecheap. At 3AM (28.5 hours of downtime) the domain transfer officially went through and Aaron Nye sent an email notifying us along with the line “Again, apologies for the service issues that you experienced. I hope that we can win your business back at some point.”

After what I’ve seen of 1&1’s services this week, I can honestly say I do not believe I would recommend them to anyone. There’s no excuse for a website to go down like Armed Gamer did, especially for not so long of a time. Their incompetent handling of something as simple as a domain name server has severely impacted my business, and it will take some time to get it all back on track.

I apologize to you, the readers, because I know this has very little to do with videogames, but I felt you were owed an explanation for the downtime from yesterday.

Fun Facts!

Apparently calling support for a company isn’t a way to get answers, but making a decently large stink on social media is.

If you make enough stink about one service-based company, expect competitors to come out of the woodwork to see what they can do to help. Major props to AIT for being ballsy enough to actually call me.

Don’t settle for anything less than what you want from a company if you’re a paying customer. I expect my domain registrar to not be the source of any downtime, and when one last as long as 28.5 hours, it’s time to find a competitor with better service.

I honestly would have been lost if it wasn’t for the incredible services of Uber Motif. They know how to take care of their clients, and they really know their stuff.

I will give credit to Aaron Nye and co. for coming to a resolution. They did come to a conclusion that was beneficial, though I wish their services were faster so none of this would have happened. 1&1 was apologetic and honest, though more transparency would have been helpful, especially if it was provided sooner.

About Stephen Crane

Stephen was hooked by the NES at a very young age and never looked back. He games on a daily basis and is currently trying to climb his way up the ranked ladder on League of Legends!
Outside of the video game world he actually likes running and owns a rapidly growing collection of toed shoes. Stephen Crane is the owner of Armed Gamer.