add a third Stack Overflow server to the HAProxy rotation to handle steadily growing site traffic

push our static content (sstatic.net), which is shared amongst all Trilogy sites, to five different servers

take advantage of our datacenter’s ability to deliver dual switched connections to the internet

The last part is in reference to an unfortunate recent switch outage at PEAK, our datacenter provider. It was handled swiftly, but now there is no longer a single point of failure upstream of us at PEAK — we’re being served by two different switches on two different internet connections. Kudos to PEAK for not only handling this rapidly at the time, but also more permanently fixing it in a way that makes it less likely to happen next time.

What motivated a lot of this was a little scare we had about a week ago. The single server that hosted our single HAProxy instance and the sstatic.net content experienced some kind of bizarre, one-time, and still-unexplainable (edit: now explained) networking issue. This sounds minor, and it should have been, but it was a major bummer in actuality because having that one server become unavailable on the network effectively took out every single site we run with the exception of Careers. We sure lost the server lottery on that one.

Not good, obviously.

We redoubled our efforts to become more redundant. We recently added a sixth 1U web tier machine so our server rack is now completely full. We have way, way more server power than we need. That’s 6 very capable web tier boxes and 2 beefy database tier boxes all told. Each ready and willing to dynamically share whatever load we want them to. We just had to figure out how to do it.

We’re big fans of HAProxy, which the guys at Reddit turned us on to. It has been working flawlessly for us in load balancing Stack Overflow between two — and now three — servers. We currently use IP-hash based routing to determine which server visitors to stackoverflow.com end up on. This helps improve local server cache hit ratios, as users “stick” to the same server for as long as they hold the same IP address. (No, we don’t use a shared server farm memory cache like Memcached or Velocity quite yet.) While this works, it does lead to some slightly imbalanced loads, particularly when single-IP whales like Google thunder through your neighborhood. We found that going from two to three servers produced a surprisingly large improvement in server load balance, even with our less-than-optimal IP hash routing choice.

HAProxy is a proven solution for us. But we needed some way of using two HAProxies — on two different servers.

Geoff did some research and came up with Heartbeat, which is a part of Linux-HA. This works by “sharing” one IP between two machines (in this case, two Linux virtual machines). It dynamically switches the IP from one server to the other when the heartbeat times out. We now have this set up and it works brilliantly; shut down one of the two VMs and within two ping -t cycles, that IP address is automagically and seamlessly switched to the other VM. There is a very brief interruption of service during this switchover, but it’s no more than a few seconds.

We also moved our sstatic.net shared content from living on a single webserver, to living on five different webservers. Our dual HAProxy instance is also responsible for routing sstatic.net traffic — as visitors request sstatic.net files, those requests are served in perfect round-robin fashion by whichever of the 5 webservers HAProxy sees as alive at the moment.

Given that these HAProxy instances are super-critical not only to Stack Overflow but every site in the trilogy (because they serve shared static content), we need constant reassurance that they’re healthy. We already use Pingdom as an external monitor and alert service for our existing websites, so we added our HAProxy instances to the mix with an aggressive ping schedule. If something bad happens to either HAProxy instance, we should know about it within a few minutes.

There might have been some momentary interruptions in service while we set this all up, so our apologies for that. But the net result is a more resilient, more reliable Stack Overflow and friends for you!

I’m surprised you don’t have sstatic.com running off a content delivery network such as Internap – serving static files quickly is what a CDN is perfect at (Internap, which we’ve purchased via Softlayer, has the advantage that it does ‘server pull’ – you just tell it which servers it can fetch the original images/files from when it needs them, change the DNS to point to the CDN account and that’s it – no backend changes needed!).

I’ve found that not only is a CDN faster and more resliant to failure (as they tend to have around 20 geographical nodes around the world), but they are also quite cheaper than most datacenter prices for bandwidth.

Just thought I’d through that out here and to satisfy my curiosity if there was a reason you haven’t gone down the CDN route…

cheap, easy, free, works great as a 512mb (could maybe be smaller, even) VM “device” on your network via Hyper-V. It also works in front of the boxes so it’s completely transparent to them, and easier to troubleshoot as a different networking layer instead of being intermixed with all your windows configuration.

> satisfy my curiosity if there was a reason you haven’t gone down the CDN route

CDNs, even the “cheap” Amazon one, are very expensive relative to the bandwidth we get bundled into our existing host’s plan.

The least I could pay is $1k/month based on Amazon’s CDN rates and what bandwidth we use.

Have you guys run into any troubles using Hyper-V for your Linux-based VM’s? I seem to recall reading that running VM’s without the guest additions on the same host as those with the additions can bring down performance for all guests.

nice article! In order to monitor haproxy, have a look at “monitor-uri” and “monitor fail” in the doc. It allows you to perform remote tests to ensure it is working, and you can even condition the response by the number of available servers behind it. That way, if one haproxy sees no server and the other one sees them all, you can ensure that the one with the servers will be used. I’m not sure on how to combine that with heartbeat however, though it’s trivial with keepalived. Anyway, it could at least be used to alert the team in case of major network failure.

I found Amazon’s CloudFront and S3 storage solutions to be quite expensive, and not as responsive, as a “proper CDN”. If you are pushing 2Tb of data, then Softlayer’s Internap CDN solution would cost just $240 and if you let them know how much you are pushing, I’m sure they can work out better pricing (we’re paying a “pay as you go rate” of $0.20/Gb: our normal DC would charge $2/Gb over our 600Gb/month limit).

Laptopheaven: I know we’re using an NFS mount to a network storage system to keep non-database content in sync, I would expect SO to use something very similar.

Great work – i’ve only ever used hardware based load balancing, but will add this to the bat belt for sure.

I like your usage of VM’s as to start with i thought you weren’t using them.

Are the VM’s hosted on SAN storage – or just on RAID disks on the machines themselves? IE what is the process to migrate the VM’s between hosts for SO?
Is it:
- bring machines down
- copy VHD’s over the network
- start them up
or do you have some sexy quick migration setup?

Of cource the database servers have public ip adresses – what other way do you see to replicate these databases accross the globe (in general)? Being in the public space doesn’t mean that you can connect when you know the ip, there are firewalls that restrict the direct access to only specific ips/subnets, you know.