Posted
by
Soulskill
on Saturday January 19, 2013 @07:01PM
from the i-will-not-be-able-to-answer-any-questions-next-week dept.

hydrofix writes "The Wikimedia Foundation is preparing for the transition of its main technical operations to a new data center in Ashburn, Virginia. This is intended to improve the technical performance and reliability of all Wikimedia sites, including Wikipedia. The current target windows for the migration are January 22nd, 23rd and 24th, 2013, from 17:00 to 01:00 UTC. Since 2004, Wikimedia sites have been hosted in the main data center in Tampa, Florida, a location chosen for its proximity to Jimmy Wales at the time. In 2009, the Wikimedia Foundation's Technical Operations team started to look for other locations with better network connectivity and more clement weather. Located in the Washington, D.C. metropolitan area, Ashburn offers faster and more reliable connectivity than Tampa, and usually fewer hurricanes."

Since I moved down to NoVA over 2 years ago, I've seen a hurricane larger than Floyd and an earthquake and just missed a massive snow storm. The trees decided to have a massive orgy leading to one of the worse allergy seasons ever and now we've got a pretty bad flu season. Nothing surprises me anymore.

I'm just down the road from Ashburn and we were fine...no major power outages from last summer's Derecho or the earthquake other than traffic getting even worse. It was Maryland that got FUBARed from storms. They lose their power a lot across the Potomac but Virginia is solid.

Also most of Ashburn is fairly new construction compared to other locations nearby and most of its power lines are buried... that probably contributes to it keeping power on when older communities in Fairfax and Maryland don't. I suspect also the choice of location may have something to do with proximity to others. UUNET (or whatever they are these days), AOL, Verizon, etc., etc.

My employer's data center is in Ashburn and when the Derecho event happened, we didn't lose access to our resources. It's a modern data center; reliable backup power was part of the decision to house servers there. Not infallible, but proved itself this time. We have more issues with our ISP having failures and configuration errors.

Not sure what to tell ya...our power almost never goes out here while my coworkers who live in Maryland have to shower at the gym at work. Power's funny like that... I bet if you looked at the power grid across the USA it's look like a gnarly mess of spaghetti...

I've got the distinct impression that most power cuts are a result of problems with the distribution wiring (especially if the local distribution wiring is overhead) on a fairly local level (in britan the 6.6KV, 11KV stuff and the 240/400V low voltage stuff, not sure what typical voltages are for similar systems in america). Afaict the 33KV and up stuff has much greater clearances and is on much strudier supports so it's rarely affected by bad weather even if it's routed overhead (which it usualy is because

Tampa hasn't been directly hit by a hurricane since 1921. Not to say it couldn't happen, but I just don't get the 'weather' argument. I remember the reassurances from Amazon Web Services last year when the 'Frankenstorm [forbes.com]' headed for Virginia.

No, but Pasco County in the Tampa area is considered one of the highest lightning capitals in the world. During monsoon season hail and dangerous lightning occurs practically every other day. THat can knock out data easily. I would not be surprised if lightning hits around the facility at least 30 or more times for the summer months.

Virginia gets them too but not so much as it is surrounded by warm 80 degree water on one side in the summer, but not 88 degree water on all 3 sides spawning these thunderstorms

California obviously would be the best for inclement weather., but has an extremely expensive costs for land, rent, and labor with high taxes and earthquakes.

It's not just earthquakes. Taking the state as a whole, California is subject to every natural disaster known to man save for volcanic eruptions. Anybody moving facilities there, when the intent is as close to 100% up-time as possible, is bloody freaking nuts.

No, but Pasco County in the Tampa area is considered one of the highest lightning capitals in the world. During monsoon season hail and dangerous lightning occurs practically every other day. THat can knock out data easily. I would not be surprised if lightning hits around the facility at least 30 or more times for the summer months.

Virginia gets them too but not so much as it is surrounded by warm 80 degree water on one side in the summer, but not 88 degree water on all 3 sides spawning these thunderstorms.

California obviously would be the best for inclement weather., but has an extremely expensive costs for land, rent, and labor with high taxes and earthquakes.

Most lightning maps draw a corridor right up I-4 from Tamps to Orlando. The second-highest probability zone is, well, just about the entire rest of the state. Lightning doesn't occur "every other day" in Summer, it occurs almost every day. People routinely get killed or injured.

Still, we know how to handle lightning and storms that take down power. Data centers with smaller budgets than Google are routinely built with motor-generators, backup batteries, power conditioners, etc., not to mention buildings cap

Lightning (especially Florida lightning -- higher frequency AND higher currents) makes Florida special for two reasons. #1, though we can mitigate power failures, it's not a risk-free operation; increase the rate of power failures, and sooner or later there will be a mistake. #2, if the lightning actually hits your facility all bets are off, and lightning strikes buildings at a decent rate (my parents' house got struck every couple of years when I was a kid -- it was on high ground) and hit my in-laws in

No one, really. Just the CIA, the Pentagon, a lot of the Federal Bureaucracy (although much of it is moving back into D.C.), The FBI Academy, lots of Military Bases, most of the Government Contractors, etc.

NASDAQ's backup data center. Huh, maybe that means that Ashburn, VA is a reasonably good location for data centers. Spooks can install taps for local snooping in any data center, but getting connectivity to the rest of the world is expensive.

No one location is safe from natural or human disasters. So, I'd rather hear that they were going to a more distributed architecture and that they'd be able to sustain a complete loss of one data center.

They do maintain servers in other data centers, in Amsterdam and San Francisco, and use them for offsite backups and read-only Squid caches. They don't live-replicate DBs to them, though, I believe due to the decreased normal-case reliability and performance that you get when trying to replicate DB servers between data centers on different continents. The architecture of centralized DB with worldwide caches performs a lot better and more reliably. But if the VA servers were offline for an extended period of time, they could fail over to the Amsterdam cluster.

I think they do replicate DBs to Florida (the former main data center), because they wrote it's a hot failover [wikimedia.org]. The other data centers are just caches, because that's most of the requests.

Virginia has had 90% of all requests anyway already, they are serving bits.wikimedia.org (JS, CSS,...), upload.wikimedia.org (images and media) and I guess also Squid+Varnish. The "only" thing missing is the actual mediawiki software, databases and things like memcached. Here's the checklist [wikimedia.org]

It is never a measure of safe, it is a measure of safer. So not risk versus no risk but risk versus lesser risk. Of course major snow storms versus hurricanes, can be a tricky balance of risk. Governors would be above ground power and comms versus below ground power and comms. In this case of Wikipedia access to volunteers would also count, more possible volunteers versus fewer, especially could be tied to the number of Universities in relatively close proximity. Of course land values could also be a major

I haven't done much Googling, but it appears that Wikimedia's data centers are all located in the US. Is this still correct? Shouldn't Wikimedia be thinking of expanding its essential operations to other relatively "free" countries like Sweden, as a safeguard against possible natural disasters or human-induced server shutdowns?

A lot of Wikimedia wikis' policies are based on United States law. The applicable laws differ from country to country. Case in point: Copyright terms for some works are longer in Sweden than in the United States. For example, copyright in any work published before 1978 and more than 25 years before the author's death expires in the United States before Sweden or other EU countries. Putting a datacenter in Sweden would affect which images could be declared public domain on Commons based on its practice of us

...I have absolutely no idea why anybody outside of the Federal Sector would want their data center in this area. We get Severe Weather (Tropical Storms and Snow Storms) on a semi-regular basis, and traffic tends to jam with a slight dusting of snow or a moderate rain to the point where it can take 3 hours to drive 2 miles, and the Utility Companies are not always the greatest at keeping the power running during these times. Neither of these things can be good for maximizing uptime and minimizing downtime.

I recommend Quincy Washington. There are a few data centers here already. The power is cheap. The thermal management is good. http://www.coloandcloud.com/editorial/quincy-wa-big-data-centers-leverage-abundant-inexpensive-renewable-energy/

A surprisingly large number of key data centers and control points have been relocated to locations in Northern Virginia near CIA HQ. AOL is there. The Iridium satellite control center is there. (It used to be in Schaumburg, IL, near Motorola HQ) Ashburn alone has four Equnix colo facilities, two AT&T data centers, two Net2EZ facilities, and a few other major centers.

A few miles away in Vienna, VA, even closer to CIA HQ in McLean and less than a mile from "Liberty Crossing" (Homeland Security HQ) there are six more big data centers.

For those familiar with downtown Tampa, it's the building with the gecko on the side, 10th floor, second cage on the left as you come in. I spent 3 weeks working a few cages down and got to chat with the Wikipedia tech.