Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

mariushm writes "According to the Register, the Chicago-based colocation datacenter C I Host was attacked by armed intruders recently, making it the the fourth time in two years that armed thugs have made off with data. According to a letter C I Host officials sent customers, 'At least two masked intruders entered the suite after cutting into the reinforced walls with a power saw ... During the robbery, C I Host's night manager was repeatedly tazered and struck with a blunt instrument. After violently attacking the manager, the intruders stole equipment belonging to C I Host and its customers.' Aggravating the situation, C I Host representatives took several days to admit the most recent breach, according to several customers who said they lost equipment, all the while reporting the problems as 'router failures'."

is that it was reported as "Router Failures" instead of the real cause.

And if they have been robbed before - why not increase the security? Four times? - That's some kind of record. Maybe it's time to check if the localization of the whole thing is incorrect and move it to a better location where it's less likely to suffer from this kind of incident?

The employee wasn't on the premises when the door lock was jimmied open (no, they didn't "cut through a reinforced wall" - they just forced a door open according to the police report and people who went to visit the site the next day to check on their equipment)

Fat chance getting a concealed carry permit in Chicago.

Illinois is one of the few states that has no provision for the concealed carry of firearms by citizens. Open carry is also illegal, except when hunting. When a firearm is being transported, it must be unloaded and enclosed in a case.

Only cops and military get CCW permits.

CI Host doesn't want to spend the money on secure facilities, instead replacing that with a rent-a-dumb-warm-body. Dumb, because taking this job w/o a permit is just begging to be thrown in jail, and if you have the permit, you can sure as heck do better than CI Host.

You can't GET a carry permit in Chicago, unless you're a cop or military.

CI Host didn't invest in proper facilities. Contrary to the article summary, the robbery was made by people forcing open the door lock to the office, when nobody was there, and an employee "just happened to show up later in response to the alarm". And the crooks "just happened to have tasers" instead of guns. And the crooks "just happened to steal all the non-existent video surveillance cameras".

And the walls are not "reinforced" - they're plain ordinary office walls. Unless you want to count a new coat of paint as "reinforcement".

They didn't bust through a wall this time - they forced the lock on the front door of the office "suite", according to a customer who went there the next day to check on his equipment.

The "reinforced walls" exist in the same universe as the "router outage".

There were no employees on duty at the time of the break-in. One employee showed up and got himself tazered, AFTER the door had been forced, in response to an alarm.

This was the 4th break-in in 3 years. That alone is suspicious. Taser? Sure, can't kill a co-worker, right?

Security cameras? There's now a question as to whether they (security cameras owned by CI Host) existed in the first place. The only cameras anyone has seen are a few owned by other businesses in the building...

One of the best design/implementation guys I know is a high school dropout. He was too smart and too much of a wiseass in school so he had problems with the administration, in frustration he dropped out. Now he makes six figures and travels the country as a consultant. He's probably been in charge of the design and implementation of over a thousand systems. While I would want my DBA to be a college graduate due to the applicability of CS concepts there are many positions including network administrator that

"at least two masked intruders entered the suite after cutting into the reinforced walls with a power saw,"

In what way was that wall reinforced? Dual layer of sheetrock? If it was sufficiently reinforced it would have delayed the intruders long enough for the police to get there (unless the police chose to not respond). If I was insuring that company I would drop the insurance dead by now due to lack of sufficient protective measures. If the measures were approved by the insurance company I would recommend all other clients to change insurance company.

Anyway - maybe it's time to weave in copper mesh into the T-shirt of all datacenter employees to protect against tazers.

And notice from a comment to the article that any so called man trap doesn't exist - and the security seems to have been far too relaxed. Just a fine example of how not to do things. A good datacenter is located where almost nobody knows where it is - preferably underground in a nondescript location in the countryside. A set of optical fibers will take care of all the traffic. And very few persons shall have physical access to the hardware. Think about how the military handles their datacenters.

Time to move, I actually live in a rural area, but due to location there was a particular bank that could not be defended, robbed eight times in two years, they finally closed the thing, and voila no more bank robberies in town. Once these people figure something out that works they will become accustomed to more income, they will come back until they get caught, or the easy pickings disappear.

Or Maybe arming some of the guards - but then again Chicago has some very restrictive gun laws, including a complete ban on handguns, so this may not be possible without relocating.

But, that's unpossible! Everyone knows that once you ban guns in an area, crime immediately ceases and the criminals turn to a life of petting puppies and painting rainbow butterflies. Sheesh. To hear you talk, one would think that the criminals would (gasp!) exploit a legally-imposed tactical advantage or something.

So can anyone here be bothered to explain to the ignorant what a datacenter is? I'd never had need to think about this concept until I RTFA. Is it really a place that people rent real estate to put their equipment?

It's not about the real estate, it's about redundant everything and lots of backup systems. If an idiot with a backhoe cuts their uplink, they have a couple more so that you don't lose connectivity. If someone flys a kite into the power lines, they have generators to keep things running until power is restored. If a squirrel eats a hole in the air conditioner, they have another one so things don't melt.

Or is it for third-party security, which really makes this bad.

Computer equipment is expensive and a data center has a lot of this, so their security compared to wh

if you don't know what a datacenter is, you're probably nowhere near interested in this entire field.

The fact that he is asking at all proves that he is interested in the field.

You must have very poor memory, nuzak, if you can't recall what it was like learning your trade.

When I was in high school, I knew I wanted to work in the computer field, yet I didn't know what a datacenter was. If I had met a pretentious jerk like you, I might not be managing a datacenter, today. Luckily, I met people who answered my questions instead of people like you. I also asked questions on slashdot, and learned quite a lot from this place.

And to answer the original question: A datacenter is a room full of computers with experts monitoring and maintaining them 24/7. It has special air conditioning, wiring, and security. Sometimes people rent servers or space there. Sometimes they contain only the computers of the company that owns the datacenter.

In that forum, it was posted that the hosting provider had posted a job application for somebody willing or able to carry a gun. They were hiring at minimum wage or just above minimum wage if you had experience with guns.

So, they're looking to hire people that carry guns that are willing to accept a job at minimum wage. That should tell you something right there.

Illinois is one of the most anti-gun states, and Chicago has even more strict rules on top of that. It's almost as bad as Britain. About the only way you're going to have an armed security guard in Chicago is if he's actually a sworn law enforcement officer or you have *really* tight political connections to those in power.

In a more, uh... "free" state, yes, armed security is a realistic proposition. However such states usually have less violent crime too, so you don't need them as much.

More like, you underestimeate the American public's willingness to side with thieves. There's a reason file sharing is accepted in polite society. Nothing Joe Public likes more than a good heist, not to mention a monkey driving the van.

It's not so much siding with thieves as it is not caring when somebody else gets ripped off, but let me tell you, that indifference disappears fast when Joe's bank account is flattened. The problem nowadays is that so many different data aggregators are keeping information

I agree that it is probably the same guys doing it over and over again. The relatively advanced method of attack also implies something over and above the crackhead level of intelligence. There are inside jobs and inside jobs though. It could just as be a relative of boyfriend or even friend of a friend or relative.

All they need is the knowledge of what is inside combined with some knowledge of the defense systems. You can get that for a couple of beers if you ask the right person in the right way.

Well, the companies finances won't be doing as well now that news of 4 seperate break-ins has gone public...Also, if thieves found it easy the first time and didn't see any significant improvements being made it makes sense from their perspective to go back. They already know the layout of the place.

Great intuition:) In the forum thread, linked from the story, they mention it seems like an inside job multiple times, including once where they claim the detective on the case stated it as well. Are there other data centers in Chicago that get robbed? I have never heard of this kind of thing happening before. I work for an ISP / data center in the upper midwest (not in Chicago or even IL).

It most likely was an inside job. A little while back, I was working for a company that was installing some VoIP phones for CI Host and the list of employee phone numbers kept changing from visit to visit - "Oh that guy? No, he doesn't work here any more."

A friend of mine that used to work there said that "being in jail was a fairly common excuse for missing work there". The employees seemed to hate working there, to put it mildly.

And the cokehead that owned the company loved to fire employees at a moment's notice, left and right. I highly doubt there's any employee loyalty there.

So in short, you've got highly unhappy employees that get fired at an amazing rate, with some seriously negative employee loyalty and they're surprised when stuff gets stolen?

Ditto. I switched off them a year and a half or two ago. I don't remember what the final rub was, I think it was reliability (website or email going up or down seemingly randomly?). I've been on Dreamhost since, and been pretty happy (note: referral link at the bottom of my website).

I agree with the other posters. They lied. They obviously have no security (or they are facing an inside job). Four robberies in two years?

I'd switch off 'em real fast if I heard this news. I like Dreamhost but if I heard this about them I'd probably switch off them fast too. How can I trust a hosting company that can't even secure their own premises?

I'm a happy DH customer, and while when I first signed up there were some disconcerting outages, the general experience has been rock solid, and from my own anecdotal experience I can definitely say they at least hit 99% uptime. This may not be good enough if you're trying to run an online service that demands 24/7/365 uptime, but it's certainly good enough for $10/month.

DH practices overselling to an extreme degree, I totally agree, but in my experience they're capable of handling the aggregate load. You

After the first robbery, I'd seriously consider moving my data. If my data is still there after the second robbery, I feel stupid. If my data is still there after the third robbery, I should lose my job. If my data is still there after the fourth robbery, I need to promoted to executive management.

The entire purpose of off-site storage is disaster recovery, and prevention of major disasters like this. Why are these guys still in business?

I'm a Dreamhost customer, and the past couple years they've had a few issues, and some people have taken it as an opportunity to bash the hell out of them. Having used many hosts over the years myself and for customers, I've found them to be on par with, if not a little above many hosts. The biggest difference is DH is HONEST about their issues, on their status blog. When they fuck up, they say so. To me, that's more valuable than a host that makes it self look like it's more stable by lying, such as the pa

True, honesty has as much, if not more, value as measured reliability.Haven't used DH, but from your description it'll go into my list of potential hosting providers now for whenever I need one.One of my last bad experiences with hosting was with actadivina - they just crashed in silence, and customers were left to speculate what happened to their site and their data, etc.At least for me (it was a hobby site), the lack of follow up was the worse part. Customers found their host provider went out of business

That was a truly "professional" operation. It's happened four times. They're specifically targeting this datacenter, and management's reaction is anomalous. If I were a police investigator, I'd start probing. Maybe there's nothing to find, but I'd bet money that there is.

But consider, if you were a thief and you robbed a datacenter... And the reaction to your robbery was very weak, no improved security etc...Would you target them again, knowing that their security was still weak and knowing the layout of the building and their security protocols etc? If i was these thieves, i'd keep hitting the same place over and over so long as they weren't doing anything significant about it. 4 times in 2 years is quite a reasonable rate, spaced far enough apart that it's not worth it fo

Well, if they actually care enough to try to prevent these attacks, I can see three solutions, any of which should be highly effective:

Deadly force. If you are being robbed at gunpoint on a regular basis, your employees can legitimately say that they fear for their lives, and thus, purchasing of firearms is legally and morally justifiable. Perhaps a couple of guards posted at the entrance with semiautomatic rifles, plus three or four in appropriately concealed locations within the facility (or more if the facility is large enough). Criminals (armed or not) will think twice before attacking.

Electrical interference. Hook a 230 kV transmission line directly to the rebar in the walls. Anyone who tries to cut their way in will likely spontaneously combust, or at the very least, be knocked several meters. Such an attack won't happen twice.

Oxygen deprivation. You probably already have halon fire extinguishers. Assign everyone emergency oxygen masks and a red button remote. In the event of an attack, press the red button and put on your oxygen mask. Assuming you dump enough halon, it will bond with all the free oxygen in the room, incapacitating or killing the intruders in seconds. Assuming they survive, they should still be unconscious when the police arrive to arrest them.

Armed guards are probably legal, and using the fire suppression system you could probably get away with if you didn't specifically give orders to use it as a weapon, but the electrical booby trap is almost certainly illegal. If nothing else it almost certainly contravenes local health and safety laws.

......It was certainly not intended to allow Joe Sixpack to wander around the US with a firearm tucked underneath his jacket......It says "the right of the PEOPLE to bear arms shall not be infringed". It doesn't say the right of the "militia", army, government, corporations etc, but "people". Maybe in your eyes and many other liberals, "Joe Sixpack" doesn't come under the classification of people. If "people" doesn't mean ordinary humans, than what does it mean? Maybe you and the other liberals ought to lo

A 230kV transmission line would probably blow holes in the concrete and arc to any metal in the building. You couldn't get within 10 feet of the wall without the electrical fields being strong enough to make your skin tingle...

Not to mention once it faults the building would probably explode.

What I'd recommend instead would be a fine mesh of metal connected to an *ungrounded* 480V feeder line.

Deadly force. If you are being robbed at gunpoint on a regular basis, your employees can legitimately say that they fear for their lives, and thus, purchasing of firearms is legally and morally justifiable. Perhaps a couple of guards posted at the entrance with semiautomatic rifles, plus three or four in appropriately concealed locations within the facility (or more if the facility is large enough). Criminals (armed or not) will think twice before attacking.

If the criminals REALLY want to get in it's not a good idea to arm the employees anyway. Specially assigned guards maybe, but the employees of a data center - no... Just imagine if someone has a bad day... Shooting out at the boss, servers and everyone else in sight. Or if the criminals know about it they will shoot first and check later.

Electrical interference. Hook a 230 kV transmission line directly to the rebar in the walls. Anyone who tries to cut their way in will likely spontaneously combust, or at the very least, be knocked several meters. Such an attack won't happen twice.

Assuming that it's concrete walls... But it's a good idea until the maintenance guy comes in to drill a new hole for a cable.

Oxygen deprivation. You probably already have halon fire extinguishers. Assign everyone emergency oxygen masks and a red button remote. In the event of an attack, press the red button and put on your oxygen mask. Assuming you dump enough halon, it will bond with all the free oxygen in the room, incapacitating or killing the intruders in seconds. Assuming they survive, they should still be unconscious when the police arrive to arrest them.

Halon use is outlawed, at least in some countries since it has a bad effect on the ozone layer. Carbon Dioxide is almost as good, and has the same effect. Of course - you may use any non-oxidizing gas like pure nitrogen or helium instead. As long as it lowers the oxygen level in the compartment. A much more evil way is to use carbon monoxide instead of carbon dioxide. In this case the survivability is even lower, but if it's released by accident it will be much more nasty.

But it seems that the datacenter hasn't taken action as it should and moved the servers to a different more covert location. The daily operation can remain at the same location, but since the servers aren't there anymore the criminals will have to leave empty-handed. This requires that the persons running the night-shift doesn't know about the real location of the servers unless they also are relocated.

Hmm, the last company I worked at had a halon system in the data storage/backup room for fire protection. This was in Florida about 2 years ago. Obviously no federal law outlawing it in the US.

IANAL, but it may be illegal to install new Halon systems. There may be a grandfather clause for existing Halon systems.

Plus (IIRC) Halon isn't actually lethal, dispite what you read in BOFH, it merely makes you dizzy and light-headed - which still isn't necessarily a bad thing, as it might help partially incapicate the thieves until they are apprehended (at which point the adrenaline may kick in, partially counter-acting the effects of the Halon.

Actually, 220 Volt is plenty, and when the rebar gets cut, you're quite likely to notice it popping a fuse. And you don't have to risk the lives of your electricians running 230 kV through poorly electrically isolated rebar.

I've used them for years, and was an avid supporter of CI Host even while they were enduring constant negative publicity.. I was initially a client of their shared hosting, then upgraded to a dedicated hosting package, and never had an issue aside from the typical short downtime every now and then.. nothing crazy.. so a startup I was working with put a colocated server with them earlier this year and in around 6 months we endure an outage for numerous days, numerous BS excuses, then one day "Oh yeah by the way your server was actually stolen, and good luck finding the real thieves!" So now we come to find that this has happened 3-4 times in the past 2 years, the detective (and even a worker there I talked with) told me they believed it was an inside job. Obviously I am cancelling all of my accounts and taking my business elsewhere. I will proudly do my best to spread the word and tell EVERYONE I know to NEVER use CI Host for *ANYTHING*

'At least two masked intruders entered the suite after cutting into the reinforced walls with a power saw... During the robbery, C I Host's night manager was repeatedly tazered and struck with a blunt instrument.

Good on them for hiring the disabled! Although perhaps the night manager position is not one suited to someone so deaf that they can't hear a Sawz-All cutting though the wall...

And especially if it's near a few racks of servers, AC, etc. All that pumping of cooling air through a real data center is very, very noisy. It's why I often wear earplugs or good ear protectors in such environments, and recommend them to staff I work with.

At first, I thought WTF???? Why would someone break into a data center.Then, I thought about it, and being that a datacenter has more supposedly valuable stuff that you can pick up and leave with _and_ you have an easy time to sell it, well, I'm only surprised that this is not a daily occurrence.

In the "information age", what is more valuable than information? And the price/pound or volume makes information orders of magnitude more valuable than gold, art or even money itself.

You are right, datacenters are a prime target for criminals.
That is why good datacenters have military-grade security. The last data-center i worked for had tripple barbed fences(NATO-standard compliant)and reinforced concrete blocks to prevent ramming with explosive-filled trucks. The inner layout of the datacenter placed the offices at the outside walls of the building and the machines at the center core of the building.
It would have taken several men with heavy machinery to get into the center, and t

Last November I had ALL my servers stolen there. Now over all the years I have had servers I have backed up data, upgraded servers, clean installed servers, etc., then that fateful day in November after being stonewalled for days I finally went over to the data center only to find ALL my servers stolen. All my data. Yea I had some offsite backups, etc, but.... stolen. I guess they thieves really liked my XServes. They were nice and shiny in a sea of beige and black.
Anyway, they gave me some free hosting after that, so, I said ok.. big mistake... about a month ago two of my three servers were stolen. Thankfully I had them a bit spread at the datacenter.
Well, can I sue? What can I do now? Same bs, promising me servers and nothing.
Last year I lost a lot of clients,,, granted all but two were mostly very small time hosting accounts. This time I now lost a big client even though I got them back up and running asap... get a server, install configure, read the files from the backup server, etc.
Anyway it was truly the worst feeling I have had in many a year.
It is definitely bs. And what do the Chicago PD do? Well your guess is as good as mine.. maybe they are out ticketing the thieves truck as their meter runs out.
Any lawyers out there that can help?

The question here is if there was a change in security protocol after the first incident. If not - then you should have pulled out and changed to another datacenter, or even considered hosting it yourself.

The problem that the Police has is that as long as it's property lost and no person was physically assaulted they tend to decrease the priority of the case rather quickly. Unless it's the RIAA, MPAA or similar organization that claims loss of billions in intellectual property. A stolen computer is the

I'll bet a big, multi-port Cicso router might be a better target, pound-for-pound, than a dell server. So the hosting company might have been telling the truth. "The router failed because, ummm, it's no longer connected!"
My second thought on this-- it seems like a lot of work to go to (and huge legal risk) for a few dual Xeon servers. I wonder who (or what) was hosting on those boxes. Cutting through the walls and roughing up the security guard will add a lot of years to a conviction. Maybe there's more t

Such distinctive equipment as an expensive Cisco router is tougher to sell. But hot-swap hard drives are easily sold on the grey market, as are modest 1U and 2U servers: few people would bother to carefully register them. Moreover, if there was credit card data on some of the servers, that's another pool of potential profit higher than a Cisco router.

But 4 times? That's ridiculous, and screams of inside job, or covering for a "Patriot Act" raid.

Yes, the servers are expensive peices of kit, but I think frankly, its more likely that there is specific data for a specific site that is being stolen rather than just some hardware, and if thats true, they shoudl be hiding the friggin servers rather than leaving them sitting round "ooo rob me rob me rob me!" You gotta ask, who's the target?

I've actually been in this datacenter. Tried to host some boxes there for a while... and when I finally gave up on their shenanigans, I was not near Chicago, so I just abandoned them there (cheaper than shipping).

First, this datacenter is literally two blocks from what is left of the infamous Cabrini-Green projects. Tough neighborhood, so it's not entirely impossible that it is an outside cracked-up scheme.

There was none of the double-man-trap doors or whatever there. The one staffer was in the back playing a Playstation. The couple of customers in the center exchanged cell numbers, so we could call each other to get let back when we needed to use the toilet.

Where are they going to fence them. The average geek has no need of 16 core Xeons, no matter what game they play. If they were dells, (IF) they are going to have TAG #s and it wouldn't be hard to see Dell doing a trace on em, ie very hot property.

Thirdly, no legit business, at least any I have worked in, would touch (some) state of the art servers at half price, no support, from a questionble source with no history. Same goes for all the cisco kit. Bet they end up abroad.

Not so much $50,000 worth of servers, as gadzillions worth of potential data. Think of how many credit card numbers/SSNs fit on $50,000 of servers... Depending on who they host (if they host any e-retailers or such) what was on the hard drives could be worth far more than the servers.

This is also why all small-time etailers should NEVER keep any pertinent data on the server. I run a small arts and crafts online store for my parents, and the most we keep are customer names and shipping addresses, so that they don't have to type it all back in each time they visit. Credit card info? Processed then immediately discarded. Passwords are all properly hashed. While I pray that my server never gets stolen like this, at least I know that my customers will not be in danger of identity theft (reas

Seriously, cutting through a reinf. concrete wall is not trivial, if it was indeed just that. By code, the minimum thickness of a concrete wall is 6" and most used for loadbearing in anything but the cheapest residential construction are 8". You aren't cutting that with a reciprocating saw (aka Sawzall). Second, reinforced concrete walls are required (in order to be considered "reinforced" by code) to have steel bars equal to 0.0014 x wall area in both directions at a spacing no greater than 18". That typically works out to a 1/2" steel bar at 12" on center or a 5/8" steel bar at 16" o 18" on center both horizontally and vertically.

Now, this is a non-technical publication, so "reinforced" may mean anything - like a 1/2" bar at the top and bottom, and around jambs. Also, this is Chicago, known far and wide for severe corruption in the building inspection process.

Still, anything close to a RC wall is going to require a diamond blade and a gas powered saw for any kind of efficiency at all, and the cut rate is going to be measured in single-digit (or fractional) inches per minute. Most also require a water source for cooling. You'd have to be utterly incompetent not to catch these guys before they got in.

To be fair, my Sawzall will go through brick easily enough. If you can find an area secluded enough to not have people nearby, you might be able to get in without a problem. If you can then get to the night manager with tasers before he can call anyone, you've got all the time you need to move things onto a big van. Maybe he was tasered multiple times because he was waking up before they were done.

1) Reinforcement can be added to walls. Make a sandwich of OSB | sheet Aluminum | OSB | galvanized steel | OSB. The layers will make it difficult to use a chain saw or an abrasive blade. Glue and screw the sandwich.