Dirty IT Jobs: Grime and Punishment

Dirty IT jobs don't always look so dirty at first glance.

Dressing up like Tom Cruise in "Mission: Impossible" and breaking into a secured facility sounds like a blast -- until you're trapped for two hours in the freezing rain waiting to be rescued. Think writing sexy games would be fun? Imagine poring over endless photo sets of explicit anatomical closeups.

Whether you're trying to squeeze big data into tiny spaces, moderate arguments between angry geeks, or hack code that's so old it qualifies for Social Security benefits, you're doing a dirty but necessary job.

This fifth installment in our Dirty Jobs series features tech jobs that can be physically challenging, mentally debilitating, or just plain irritating. Be thankful that these people are doing them -- otherwise, you might have to.

Dirty IT Job No. 1: B&E Artist

Dressed in black camo, hiding in the woods in the dead of night on the edge of a Pennsylvania mountain; it's not your typical IT job.

But that's where Matt Neely found himself more than a year ago. As vice president of consulting for SecureState, an information security management consulting firm, Neely's job is to test the physical security of his firm's clients, which include large federal agencies, major retailers, energy plants, and even entire countries. Trained in the art of lockpicking by his previous employer (a bank), Neely uses his breaking-and-entering skills so that organizations can find holes in their perimeter and fill them.

On this cold December night, Neely and a colleague were asked to break into a mining facility just past midnight and steal "trophy data," while two other SecureState penetration testers social-engineered their way in via the front gate. The coal mine was concerned about environmental activists breaking in and tampering with its SCADA systems, causing the mine to shut down. They had good reason to worry.

According to Neely, the mine's external security was so porous that he and his partner were in and out in 10 minutes, or about two hours and 20 minutes less than he'd bargained for. The area around the mine was so remote there was no cellphone coverage, so he had no way to reach the other SecureState team. He and his partner had to hunker down for two hours in a freezing rain before they got picked up.

Neely always carries a "get out of jail free" card on his jobs, listing the names and numbers of company personnel who've authorized the security test. So far, he's never had to use it -- even when caught red-handed breaking into a power plant by the cops. Fortunately for Neely, SecureState's policies prohibit performing tests on facilities where the guards carry weapons.

The dirtiest part of his job? The ensuing paperwork, he says.

"The biggest difference between a criminal and a penetration tester is that a criminal doesn't have to spend half his time writing reports," he jokes. "You get a job breaking into facilities to steal data and you think you'll be doing all sorts of cool crazy stuff all the time. Instead you wind up spending hours looking for a security hole, there's a couple of minutes of excitement when you get to break in, and then you spend the rest of your time writing about it."

Dirty IT Job No. 2: Big Data Backup Slave

Stephen Roubelet has seen the ugly side of big data firsthand. As IT/IS manager for engineering design firm RDK Engineers, part of his job is to manage 8TB of data over five locations. In fact, RDK had so much data there wasn't time to do a full backup every weekend.

"We had built a small server network so that we could spool our data to disc and then to tape," he says. "We'd start on a Friday night and not finish until Tuesday or Wednesday. With the backups running so long, there would invariably be problems, so I and my staff were constantly checking in over the weekends to make sure the jobs were still running."

Eventually Roubelet began doing one full backup a month and nightly incrementals, hoping none of the tapes in the series got damaged or the entire set would be lost. But about 15 months ago RDK adopted Actifio, a protection and availability storage platform that allowed the firm to virtualize its data management and simplify the backup process.

"Being the backup guy is like the hazing program for new IT workers," says Ash Ashutosh, CEO of Actifio. "No one gives you credit for doing a good job, but screw up once and you're fired. And when you bring that process to a big data environment people usually just give up. The data slave puts on a brave face and claims he's keeping the data secure, but he's hoping and praying no one ever asks him to recover something."

The dirtiest part of being a big data backup slave? Searching for old tapes when you need to do a restore, says Roubelet.

"The first question you have to ask: Is it on tape?" he says. "If so, what tape is it on? Did it spool over two or three tapes? If so, you have to find the whole sequence. Is it stored offsite? You'll have to contact the storage facility. Did the backup fail on the particular weekend when the file was saved? Our backup software may have crashed and restarted four times that weekend, and maybe one server didn't fully restart. Inevitably that's the server where the files you need were stored."

Dirty IT Job No. 3: Sexy Games Producer

You spend all day staring at pictures of naked women and talking dirty in three languages. Sounds like dream job, right?

Not exactly, says Patryk Bukowiecki, a game producer and manager who worked for a maker of "sexy" J2ME mobile games in the late 2000s. There he produced games like Lessons of Passion Blackjack, in which game players engaged in simulated sex talk with scantily clad models. Bukowiecki's job was to comb through 300 explicit photos of each model, pick eight of them to use in the game, write "sexy" chat for each girl, then translate it from English into Polish and German.

"I had to make sure the girls talked dirty enough so that guys would play as long as possible and then buy other editions of the same game," he says. "The texting part was actually pretty fun."

But those hot models? More sad than sexy, says Bukowiecki, especially after you've seen their passport photos, which each model had to provide to prove she wasn't underage and was doing this of her own free will.

"I had to approach my job kind of like I was an ob-gyn," he says. "You have to get some distance on these things or you will go mad with all the ideas running through your head. You might be tempted to try and find these girls and save them. Don't do it."

The dirtiest part of this job? Telling people what you do for a living.

"You sit in an open space in the office looking at these pictures on your computer surrounded by 20 people," he says. "They're all passing by your desk, sneaking peeks at what you're doing. Then you get home and your wife or girlfriend asks, 'What did you do at work today, dear?'"