Posted
by
ScuttleMonkeyon Wednesday October 10, 2007 @06:12PM
from the where-is-my-underwater-data-center dept.

johannacw writes "Would you house a data center in a diamond mine or an old chapel? These organizations did, with great success; many of these facilities offer the latest in cooling and energy technology, among other advances. 'If you want an even more hardened environment for your data, you might look at the aptly named InfoBunker in Boone, Iowa, about an hour outside Des Moines. [...] The 65,000-square-foot, five-story site is dug deep into the ground. No one gets in without passing though the 4.5-ton steel door and then a three-step process. A scanner uses radio frequency to read the would-be entrant's skin as a biometric identifier. He then needs to use a keycard and enter a code on the keypad. This three-tier security is standard for high-level military installations, McGinnis explains.'"

For the trifling sum of 1.5 million dollars you too can be lairing it up in style...

"The Missile Base consists of 57 acres of real estate. The center secured portion of the property is protected by the original barbed-wire-topped chainlink fence. There is a paved road leading into the property with dual entry gates.

Above ground is the original 40 X 100 shop building, two concrete targeting structures, two manufactured homes, two 8 X 8 X 40 storage containers, and the silo tops of the three missile silos, two antenna silos, one entry portal and a few other misc structures.

Below ground is a huge complex consisting of 16 buildings and thousands of feet of connecting tunnels. The major underground structures are:

I mean, honestly, is it just me or are all these "exotic" data centers just a way to boost your CIOs ego at gatherings? Is it really necessary to have military security? Do your competitors care that much? Furthermore, would they be willing to risk criminal charges to try and steal a few thousand hard drives full of potentially useless data?

Basements with backup power, secured doors, & a good fire system in my opinion. Then again, I'm not a CIO. Once I become one though, well, I imagine MY data center will have a golf course. And blackjack. And possibly hookers.

A friend of mine bought an old missile bunker in the UK to use as a data center back during the late-90s boom. It had redundant power-grid connections, lots of cooling, and raised floors, so it cost a lot less to condition the space for data-center use than if he'd started with a basic warehouse shell like many of his competitors, and it was close enough to London for latency not to be a problem but far enough that the real-estate costs were cheaper.

U.S. geography isn't always that cooperative - most of the missile bunkers were out in not-even-flyover parts of the country like North Dakota and eastern Montana, where there was almost no telecom infrastructure nearby and it was tens of milliseconds away from SF, NYC, or even Chicago.

And Canada has their own problems - even though most of the people live within 50 miles of the US border, the Canadian government has been doing things like offering tax incentives to put call centers in remote areas to deal with unemployment - former fishing ports in Prince Edward Island, etc. - where there's not enough local telecom infrastructure to get high bandwidth connections or diverse routes. Too bad, since they've got a pool of educated people who speak good English and something that passes for French and could use the jobs.

I will build mine on the bottom of the sea, a data center where the artist would not fear the censor, where the scientist would not be bound by petty morality, and the great will be unconstrained by the small!

Flooding is no problem if you have a double-wall, pumps and redundant tubing downstream.
Top floor datacenters are no longuer very desirable after certain planes crashed in certain towers. Plus it makes it easier to break into.

I mean, honestly, is it just me or are all these "exotic" data centers just a way to boost your CIOs ego at gatherings?

You say that like it's a bad thing. Really, is making the CIO feel cool all that much worse than whatever the datacenter is doing anyway? It's probably calculating stock prices, keeping track of financial information, caching web pages, or whatever. It's all just the mental masturbation of modern society anyway. Might as well feel cool doing it, then some concrete good will come of it.

Pseudo-security is a bad thing, because it gets people to let their guard down. When they think that some magical talisman they bought (or in this case a bunker) makes the server super-extra-uber-secure, then the next thing that happens is that they cut the funding for real security.

Think of the dot-com era, really. How many times have you heard companies going "we're secure because we use 128 bit HTTPS! See that padlock icon? It means we're secure!" and then they forgot to check rights in their web site and/or just leave internal files around in the web server's directories or on some public FTP directory? Or leave their web server, some active ftp daemon, and God knows what else run with the default admin password? I can think of a couple which cheerfully left text files with user data and credit card numbers available for everyone. But, hey, they have 128 bit HTTPS, so they're secure.

Or I know of at least one corporation which bought all sorts of expensive appliances to scan all JMS messages and SQL statements for malicious stuff... but then noone actually configured rules for those. They used them effectively as some magical talisman that makes them secure just by being there, no extra work required. And some of them were bogus talismans anyway, pure snake oil that couldn't even have done the job right.

_That_ is the problem. When someone is as disconnected from reality as to think that security means preventing teams of ninjas from physically breaking in, something tells me that they probably didn't have thought much about actual security. And will think even less about it in the future.

It isn't the CIO's job to worry about security, though, he's got IT people for that. He's got other people's money to burn, and he doesn't want ninjas attacking his datacenter. I don't really see a problem with blowing a bit of the money on cool shit. It's not like a few extra thousand dollars would make all the stockholders trillionaires.

Speaking as someone who actually works with system that transfers hundreds of billions in assets and securities on an annual basis (yeah, hundreds of billions, that'll keep you awake at night from time to time), physical security is a very real, very important consideration. There is nothing pseudo about it. In fact, after 9/11, federal law requires companies like ours to have certain levels of physical security which are surprisingly stringent (on top of redundancy at widely separated physical locations an

Let's put it like this: the very same institutions "where a disruption will affect global markets and everything that follows" have, about a dozen times in the last year alone, copied sensitive data on some sales-guy's laptop and it got lost. Some of the very same institutions had got pwned and had zombies. Some of the very same institutions have offshored that kind of data to places where it's entirely out of their control, just because it was a couple of dollars cheaper. (And I don't mean just to India, b

You've got the scale all wrong. A relatively small breach of data (account data on a laptop or whatever) is a far cry from bringing down a major piece of the global financial market infrastructure (and yes, we really do process enough activity to warrant that description).My company lost two of our four major data centers in the 9/11 attack on the WTC. Completely independent of the many other effects of 9/11, the loss of those data centers and the related downtime and confusion and disruption of service had

The briefly required travelers to surrender toothpaste and bottled water. I travel by air fairly often and haven't encountered any such difficulties recently. Long lines to get through security is about the worst of it now.9/11 is relevant, as it demonstrated to my company a need for rather more extreme physical security measures. Our data centers were in fairly stout, very secure buildings -- and one was completely destroyed, and the other was nearly destroyed (along with major portions of our headquarters

Dude, no offense, but I see business decisions taken every day that are 100%, pure, unadulterated, _show-business_, and serves no other purpose than to boost some PHB's ego or make him feel like he's doing something (cool.) And I'm not even counting the ego-masturbation kind where some clueless PHB takes some "strategic decision" based on knowledge and expertise that he just doesn't have. (The ego-masturbation there being either pretending that he's savvy enough to even know the implications of what he deci

Data centers can be in strange locations, before the advent of data centers companies just put their computers anywhere. Two strange locations I know, a closet in the womens bathroom at a company and a closet in the another companies machine shop. One had access issues, the other had massive metal dust issues. One large company for some strange reason put all their printers in their computer room, talk about paper dust issues. One other one I remember was under the companies staircase.

Basically, yes, they're there to boost some manager's ego. I haven't even heard of a recent data loss or theft that involved a team of ninjas breaking in and stealing hard drives. The ones I did hear about, offhand, involved stuff like:

- pissed off admin exports the customer database and sells it to a spammer

- a hired rent-a-coder working at home is given an export of the fucking productive database, just so he can work out the report formatting. So he asks for help in a forum and attaches a zip file of said productive database. Just so, you know, others can try their hand at formatting that data too. (And if you think that's a one-off thing, at a recent consulting job I've seen exactly that happen, with the dumbass PHB's blessing. They exported the productive database, installed it on a test machine, then let the external contractor -- not me, but the guy whose neverending mess I was supposed to help fix -- copy it all on his private laptop too. And since he was not supposed to connect an external laptop to the internal network, the PHB cheerfully supplied an USB stick to transfer the data with. Made me cringe. But, hey, he was cheaper than doing it in-house.)

- productive data, complete with customer names and personal data, is copied on some salesman's laptop, because god forbid that you inconvenience the sales guys in the least bit, even by making them log in to a web site. Plus, I'm sure he thinks he's a wizard with Excel and God knows what ad-hoc graphs and reports he might need to generate on the spot from that data. Then said laptop is forgotten on the airport or stolen. (I can remember a dozen or so instances of this in the news without even googling.)

- social engineering and/or lax security standards (As an extreme case, I've actually worked for a dot-com back in the day, who told their 1st level support to give anyone an admin account who calls in and asks for one. It's easier than just creating one for the regional managers -- although I'd debate whether those need one in the first place. Nah, just tell them to phone in and ask for one. Eventually after a year they realized that they have a few thousand admin accounts and nobody knows who those people are.)

- pwned machines on the internal network that haven't been patched since Jurassic. I remember one touching story about IIRC Slammer, where a company got hit hard because they were running with completely unpatched workstations, since apparently installing any service pack broke one of the internal applications they were using. And, of course, they'd rather save money than fix the stupid application.

- pwned machines on the internal network because some dumbass PHB or marketter figured out (or bribed an engineer for the knowledge) how to open a tunnel from inside to his home machine and leave it on, so he can access the company network from home. So when his unprotected, crapware-ladden home machine got pwned, it was connected to the intranet.

- pwned machines on the internal network because just about anyone is allowed to plug their laptop in

The last three are especially nice if everything is one big network zone.

- pwned machines because some dumbass programmer would rather argue that SQL-injection and cross-site-scripting are just hype, instead of fixing his freakin' application. I'm still suprised at the number of people who don't even know how to quote a string for use in a web page or in the database. Or better yet, to use prepared statements and/or some template/framework that handles that kind of thing for you. And, yes, I remember at least one article linked even on Slashdot where the idiot was arguing that cross-site-scripting vulnerabilities are inevitable and harmless.

- pwnage via any of the above methods (including social engineering or dishonest employees) because noone bothered setting productive database passwords more creative than the same as the app name, and/or using more than one account for a whole department. Or indeed for the whole company. It's too much work

I haven't even heard of a recent data loss or theft that involved a team of ninjas breaking in and stealing hard drives.

Ofcourse you haven't heard of it. They're ninjas. They sneak in, replace your HD by a death one and leave without you knowing it. How else would you explain HD crashes? So next time you hear the sound of a death harddrive, you'll know there is a ninja nearby...

You forgot about outsourcing with insufficient control on the handling of data. You may work at a bank, for an example (but it could be medical data for a hospital or insurance company) but the person on minimum wage cleaning data who works for another company doesn't give a fig for the data.

Thanks for the correction, I'll try to remember it. Still, what with not being a native English speaker nor in an English speaking country, if that's the worst blunder I've made... I'll take it as a compliment:P

Well, you know that saying that if full tilt nuclear war breaks out that the intertubes will still function? There is a reason for that...ARMED GUARDS to kill off the zombies and pr0n starved nerds of course!!

IN all honesty though, a freaking suspended glass NOC? Does it really matter whether you can see the server racks or not?//my data center will have an armory, and slides that go everywhere. (yes, those kinds of slides)

Interesting. Sounds competently done. Other folks try for the same outcome and can fall short.I've had occasional reasons to visit some of our "secret" government offices, usually multi-agency installations sort-of gathered under a DHS umbrella. The last one I went to had no sign outside, no sign on the door, no number on the door. How did I know I was standing at the right door? In this non-descript office building in a ho-hum industrial park, there were 50 cars parked outside with US government plate

That reminds me of a story (maybe apocryphal) about how some foreign country with which there had been recent tensions knew the US was about to attack because one night at the Pentagon they observed a ton of people working after hours and a steady stream of late night pizza deliveries.

The way I heard it, the "something's up because of all the pizza deliveries" story had something to do with activity at the White House during Watergate. I looked it up at Snopes and couldn't find anything.

The RFID reader was for access to the big door for customers/staff. There's a whole lot more behind door number one. It's not like you walk straight into the racks or anything. Mantraps, guards, etc. before you get to the fun stuff.

You are going to see a lot more systems use a "two out of three" approach. I actually thought, at one point, that this was going to be a requirement for Vista. I guess not.

The system in TFA requires all three: what you are, what you know, what you have. While requiring three out of three might seem a little nuts, it will seem less nuts in a few years when everyone has to have at least two out of three in order to do basic things like log onto their computer.

1) What is your name?
"Cowboy Neil"2) What is your quest?
"To fix the bricked file server"3) What is the Emacs key binding for going to the previous line and decreasing the indent?
"What? I don't know tha.. AARRRRGGGHHHHH!"

In most of the three-tiered identification methods I've seen, #2 and #3 provided all the real security and #1 was only able to make any kind of decision if it had #2 and #3 to back it up.
Also, #3 also tells the system who you are unless you have given your password to someone else. If you give it away voluntarily, you are an idiot. If you give it away at gunpoint, then likely they would have found a way to drag your biometrics along with them.

I'm not sure that you have thought through this very carefully. There are many, many solutions to your questions.

1. If you know that someone might need to access your account in advance, you can register their fingerprint with your account. The combination of your password and their fingerprint is two out of three.

2. You can leave a USB key somewhere hidden in your office / bedroom / etc. In the case of an emergency, you call up your business partner / family member and tell them where the USB key is

Disasters come in many forms. Having more than one center is probably more important than extreme security at one site.

The sites should be separated by physical distance and political jurisdictions. Data lost isn't limited to physical problems. It can come in the form of a legal scavenger hunt. Both can put you out of business.

I have done quite a bit of research on using them. I had the idea to use it for hot sites, data storage and other DR related.
One of the main problems is environmental. Old Titan II silos are FULL of asbestos and other carcinogens (PCB's). There is a very large cost to cleanup, drain, and refurbish the infrastructure. Much more than the purchase price. I found one in eastern Washington near major fiber optic lines, power and transportation that was ideal (with LOTS of work and $$$). If I had a 10-20+ milli

The InfoBunker, the Iowa site mentioned in TFA, is one of a number of cold war missile and/or communications facilities being used as data centers. The PJM Interconnection, which runs the East Coast power grid, is setting up a data center in a Pennsylvania site [datacenterknowledge.com] once used for White House-to-Kremlin communications during the Cuban Missile Crisis. The Bunker [thebunker.net] in the UK is in a former Ministry of Defense command-and-control center. Ask.com is building a major data center in the Titan building in Moses Lake, Wash

File server, print server, dual tape loaders, UPS, all setting on shelves, mounted above the level a suspended ceiling, with a mirrored fail-over setup at the opposite side of the building, also above ceiling-level.

It was a medical office and they were floor-space constrained so 'going up' seemed the logical solution (there was an absurd amount of space up there.) They'd had the electrician in to put outlets up there, the shelves were reinforced and had a lip added so nothing accidentally slid off (there was even a strap with a buckle to make sure nothing ever dropped down.) The hardest part was lifting the hardware up into place.

It was a complete "you've got to be kidding!" scenario when I first saw it, but I had to admit for a crazy location it was a sweet setup and worked great for their needs.

Installing computer equipment above the ceiling would be a violation of building codes in many cases, especially if the above ceiling space is used for air return. The national electric code prohibits such installations by banning the use of flexible cords above the ceiling:

400.8(5) Flexible cords shall not be used where concealed by walls, floors or ceilings or located above suspended ceilings.

Generally, the only time receptacles can be installed above a ceiling is the provide the receptacle requir

No one gets in without passing though the 4.5-ton steel door and then a three-step process. A scanner uses radio frequency to read the would-be entrant's skin as a biometric identifier. He then needs to use a keycard and enter a code on the keypad. This three-tier security is standard for high-level military installations, McGinnis explains.

There are problems with playing "military installation" when you're not a government. What do you do when someone shows up at your front door with apparent legal author

That's not just a bank vault door, it's a blast door. If it's designed to resist nuclear attack, no redneck with a backhoe and some dynamite is going to be able to do much to it.

A blast door is not necessarily resistant to burglary. It's installed from the outside and can generally be removed from the outside. I saw the blast door at the Titan missile base in Arizona, and while it's an impressive chunk of steel, it has some glaring differences from a bank vault door. Such as only a single locking point (

In Frankfurt am Main, Germany, there is a datacenter in the old bunker at the airport. With all the security and technical infrastructure available at an airport, and the prime connection to every other airport of the world this just makes sense.;)

Yeah it might be cool to put a data center in an abandoned missile silo, salt mine, catacomb or crypt. But the practical site administrator should look for low real estate costs, reliable low-cost green electricity. Companies no longer have to put their IT centers where their employees are. If you put data centers in places such as Silicon Valley, Bangalore, Beijing where real estate and electricity demand are so high that you're paying 2-10x per square foot 2-10X per watt what you'd be paying elsewhere.

There are a lot of great places to put datacenters, not just for novelty reasons but because of the natural advantages of the site. Where I live there are lots of old brick mill buildings that would be easy to cool, and could be powered at least partially by water power. The problem is that they're all many miles from any kind of existing backbone link.

I was in Italy this summer and when I was in Portofino, they were getting ready to film a movie at Castello Brown. As I was touring the castle, I came across a little greenhouse outside and couldn't help but notice that they (the movie people, apparently) had set up a small data center inside it.

I dunno, seems like an awfully strange place to be setting up a data center.

Security lesson #1. Can you spot the security problem in this high-security datacenter? Sure, it's got a vault door and armed ninjas. But you didn't count on the night janitor sneaking in a wireless router and plugging it into the network.

I for one am glad they are sending cold air to the backs of the racks. It doesn't matter quite so much in this case as there is so much crap in the plenum that the air flow is already killed before it has a chance to get to the out vents. Oh ya, one more gripe... people will bring trash, dust, liquid, and a potential of pressing shiny buttons and pulling on cords.