How our Server Crashed When HN Traffic Kicked in

A couple of days ago I posted the article 1000 days on Hacker News, a brief post sharing my experience with being an active member on that site for the past 2 years+. Since it was talking about Hacker News, I posted it on the said website as well and my submission got quickly upvoted. Until it reached close to the top of the list, faster than I was expecting. Traffic is always good, but this was clearly out of control. Uh Oh.

After submitting it, I was watching with Google Analytics the real time traffic going to PandoraLive, and that was really fun. I rarely get more than 5 people reading PandoraLive at the very same time, and here the numbers were jumping to much, much higher levels.

It started fairly nicely. My post was on the 16th or 17th position on the first page of HN, and while you do get some traffic, it was still manageable, with 23 users simultaneously browsing the article. But, as the post got more upvotes, the numbers starting going up, very fast. I messaged my friend in charge of the server administration, to let him know we were facing increased traffic, just in case.

More than 100 people at the same time. That was unexpected. This is something I knew could happen, I had seen other stories about the kind of traffic you can expect when you get on HN’s front page, but I had no idea that my post would become popular in that sense. At best, I was picturing it making the bottom of the first page, but that’s it. Clearly, the situation was out of control as my post jumped to top 5 spots on HN’s front page.

A quick look at the map showed that the traffic was really worldwide. One of the cool aspects of HN, it reallyis international and not just US-centric.

At that stage, I knew that the risk was clear: I had no caching installed on my WordPress setup, and sooner or later, if the connections go beyond a certain threshold, the server was going to crash. Now, the question was when…

Well, the limit was apparently around 150 concurrent viewers. Then the post went up to the 3rd place on HN and things were over.

The server went down and stopped answering requests. Flat line.

We tried to come back online as is, thinking we could at least get enough time to activate caching, but that was not the case : traffic was still hitting our server like crazy, so as soon as we would come back online, the server crashed again.

What we did is that we disabled WordPress and rebooted the server. WordPress was not accepting any traffic and that way I could add the caching plugin, and then we restarted it. By the way, we used the W3 Total Cache plugin and activated most of the caching options (of the free version).

By the time we came back online, the traffic was still high, more than 100 concurrent connections, and within a dozen of minutes it went back to 130+ simultaneous viewers, since it was still at high visibility on HN. This time we were watching closely what was happening. The cache plugin seemed to work just fine. CPU was clearly under control (around 12/15% usage, with some peaks at 50% now and then), and I was checking on the side how fast the pages were loading now and then. The effect was quite dramatic as you could not feel any of the congestion that was there before, when the Cache plugin was missing.

This was clearly a mistake to have published a story on HN without being ready for the potential traffic. But at the same time, it was a good real-life experience on how far we could expect our server to sustain the traffic.

So, if you are using WordPress, I’d definitely recommend you get this cache plugin, or at least put a static page, and get ready for the case you need it one day. Thanks to all the HN readers who visited our link, it was great to see so much interest in a short time.

This was clearly a mistake to have published a story on HN without being ready for the potential traffic

This was not your mistake. Your mistake was utilizing WordPress.

150 simultaneous users should have been just a walk in the park for what is otherwise a static blog, not a full hard crash incident. In fact, for what is a static blog, you should have been able to withstand 10x that amount with no problems.

For future reference you might consider utilizing the unpaid options of Cloud Flare. it’s excellent for these sorts of traffic spikes. Heck I’d enable it anyways for those unexpected moments, it won’t affect anything on a regular basis and it’s free anyways! No, I’m not a salesman and I don’t work for them. I’ve utilized them in the past and the results were quite dramatic, and as I am a proud pandora owner/supporter I thought it appropriate to mention.

Post navigation

What’s This ?

GiantPockets (Previously Known as Pandoralive.info) is a blog dedicated to the Pocket Linux Handheld computers (such as the Open Pandora and the upcoming DragonBox Pyra, but also non-gaming handhelds like the GPD Pocket). These handhelds can be used for gaming but for a bunch of serious uses as they can run full Linux distributions and are typically equipped with physical keyboards. You will find here news regarding their latest software offering, hardware updates, development stories and testimonies of users.