Steve Barnet is hiring, but not for an ordinary IT job. His ideal candidate "will be willing to travel to Polar and high altitude sites."

Barnet, interim Computing Facilities Manager for the Wisconsin IceCube Particle Astrophysics Center (WIPAC) at the University of Wisconsin, is looking to fill what may be the coolest Unix administrator job opening in the world—literally. Plenty of IT jobs exist in extreme and exotic locales, but the WIPAC IT team runs what is indisputably the world’s most remote data center: a high-performance computing cluster sitting atop a two-mile thick glacier at the South Pole.

The data center has over 1,200 computing cores and three petabytes of storage, and it's tethered to the IceCube Observatory, a neutrino detector with strings of optical sensors buried a kilometer deep in the Antarctic ice. IceCube observes bursts of neutrinos from cataclysmic astronomical events, which helps to study both "dark matter" and the physics of neutrinos.

That mission demands a level of reliability that many less remote data centers cannot provide. Raytheon Polar Services held the National Science Foundation’s Antarctic programs support contract until April. As Dennis Gitt, a former director of IT and communications services for the company puts it, a failure anywhere in the Antarctic systems could lose data from events in space that may not be seen again for millennia.

Running that kind of IT operation at one of the most hostile and remote locations in the world creates a whole set of challenges few in IT have ever experienced.

The few, the proud, the cold

Ralf Auer, left, and Steve Barnet, right, with James Roth of the IceCube team, dressed for a walk to the IceCube data center.

A trip to the Amundsen-Scott South Pole Station is as close to visiting another planet as you can get on Earth, with “a palette of whites, blues, greys and blacks,” Gitt says. In summer, it feels like two o’clock in the afternoon 24 hours a day, which “can do interesting things to your diurnal cycle,” says Barnet. In winter, the outside world is lit only by moonlight and the Aurora Australis.

With a maximum population of 150 at the base during the Austral summer, South Pole IT professionals-in-residence are limited to a select few. And they don’t get to stay long—most of the WIPAC IT team only stays for a few months in the summer, during which they have to complete all planned IT infrastructure projects.

The rest of the year, WIPAC's US-based employees remotely walk the wintering members of the IceCube team—usually chosen for their physical science credentials, not for their IT skills—through tasks over satellite calls. Systems are monitored over a multiplexed Iridium satellite connection. “You just try to collect as much information as you can and do what you can remotely,” says Ralf Auer, WIPAC’s South Pole systems administrator.

The wintering-over team “can do a lot of the physical maintenance,” says Barnet, but routine IT tasks can sometimes feel foreign to physical scientists. The result is that Auer, Barnet, and the others back home have to visualize what the over-winter team is seeing in order to walk them through fixes. (Barnet compares it to being an air traffic controller talking down an airliner flown by a flight attendant.) So part of the team’s job is to make these hands-on tasks as simple as possible for the scientists, and to handle as much as possible remotely over whatever bandwidth they can get.

South Pole Station does have other IT support, but it’s not on WIPAC’s payroll. The general IT and communications support team for Amundsen-Scott peaks at 5-8 people during the summer and shrinks to 4-5 during the winter, according to Gitt. More stay at McMurdo Station, the main logistics support base in Antarctica: 35-40 at the peak of the research season in the summer, “depending on the projects,” he says. Smaller stations run by NSF may have only two or three IT and communications people total. (Lockheed-Martin just took over the NSF contract, but the award is currently under protest.)

The IceCube Lab at Amundsen-­Scott South Pole Station—home of the world's southernmost data center.

There’s plenty of work to go around. During the summer, Gitt says, “You may have close to 1,000 scientists rolling into different stations getting their science done.” During the peak of the research season, McMurdo Station can surge to a population of around 1,100 people, some staying for only a few weeks based on their projects and funding. Every one of them wants to make sure they can get the IT they need when they need it—often all at the same time, since they have to reconfigure schedules around what the weather allows.

The low headcount and the urgency of just about every project—where the millions spent on a grant for a particular project may be lost if data can’t be collected or transmitted—demands that the few IT pros on hand in Antarctica have not just depth of knowledge, but skills across other fields as well. “Typically, they’re at an engineer level as opposed to a technician,” Gitt says. “They could fix almost anything.” One technician had to diagnose a problem with the Windows software package needed for a project on a researcher’s laptop—and the program was written and documented in French.

The IT teams in Antarctica don't operate in complete isolation; while Raytheon provided support, the Antarctic equivalent of a help desk was in Centennial, Colorado, for instance. But communications back to the US from the South Pole are akin to communication from space—in fact, South Pole stations use the same satellite network for broadband communications as does the International Space Station.

The most reliable form of communication available is the Iridium satellite network. Individual Iridium connections aren’t exactly blazing—they support a data rate of only about 2,400 bits per second. But according to Gitts, Raytheon did a lot to coax as much bandwidth as possible out of Iridium, including multiplexing Iridium connections, doing compression to shrink the size of e-mails—even adding a wireless server with file-sharing and e-mail services to containerized Iridium ground stations to support some of the smaller field stations.

“We even dropped the package size down further in size and put it on some of the traverse tractors," Gitt says—providing a data lifeline to expeditions crossing the frozen continent.

The higher-bandwidth access is limited to about 10 hours a day of broadband coverage from NASA’s Tracking and Data Relay Satellite System (TDRSS) and the GOES-3 Satellite —a weather satellite launched in 1978 that lost its weather imaging capabilities and now provides 1-megabit per second data transmission for eight hours a day. TDRSS provides the most bandwidth, with transmission speeds of up to 150 megabits per second.

But catching signal from those satellites at 90 degrees south latitude requires some serious work. Two of the IT support team for the South Pole Station are a satellite engineer and a technician, because keeping communications up often requires more than one set of hands. “With TDRSS," says Gitt, “they are actually working four or five contacts a day, where ground station is swinging over the dish to catch the next satellite as it comes over the horizon.”

And they don’t get very high above the horizon; ground station dishes are almost always pointed nearly parallel to the ground.

So....where do I sign up? I know I am nowhere near qualified, but stranger things have happened. Plus, it's only 3-4 months in the summer of the Antarctic, then rotated off. It could be an annual thing, maybe?

That leads to some interesting medical prerequisites even for the relatively healthy. Before he was cleared for travel to the Pole, Barnet was told he had to get his wisdom teeth extracted. “And if you’re staying for the entire winter,” says systems admin Auer, “they prefer to have your appendix removed. They had a case in 2001 when they had to medevac someone because of appendicitis.”

On the morning of 29 April 1961, Rogozov experienced general weakness, nausea, and moderate fever, and later pain in the lower right portion of the abdomen. All possible conservative treatment measures did not help. By 30 April signs of localised peritonitis became apparent, and his condition worsened considerably by the evening. Mirny, the nearest Soviet research station, was more than 1,000 miles from Novolazarevskaya. Antarctic research stations of other countries did not have an aircraft. Severe blizzard conditions prevented aircraft landing in any case. Rogozov had no option but to perform the operation on himself.

The operation started at about 22:00 on 30 April with the help of a driver and meteorologist, who were providing instruments and holding a mirror to observe areas not directly visible, while Rogozov was in a semi-reclining position, half-turned to his left side. A solution of 0.5% novocaine was used for local anaesthesia of the abdominal wall. Rogozov made a 10–12 cm incision and proceeded to expose the appendix. General weakness and nausea developed about 30–40 minutes after the start of the operation, so that short pauses for rest were repeatedly needed after that. According to his report the appendix was found to have a 2x2 cm perforation at its base. Antibiotics were administered directly into the peritoneal cavity. By about midnight the operation was complete.

Doesn't pay enough. $50k for SysAdmin is about normal. Should be adding another $50k just for "danger pay" on top of that.

Ragnarredbeard wrote:

who? wrote:

Wow. I never thought of adding "My appendix has been removed" to my CV as a major selling point..I wonder what the male:female ratio is down there? (then again, for us nerds, it just reduces our chances from almost zero to almost zero.)

Doesn't pay enough. $50k for SysAdmin is about normal. Should be adding another $50k just for "danger pay" on top of that.

Yeah, but one nice perk is free room/board for 1/4 the year. For a single person, that's worth several thousand dollars if you can sublet your place.

Still not even close to enough. I was (and still am) thinking that it'd be a neat experience, but 50k per year is a laughable amount of money considering the sacrifices. This is under the assumption that it's a full time posting. If it's something like a few months of actual work on-site, it'd be better.

I am sure it probably was in C++. But when all your variable names and text fields are in French, its gets rather hard to derive the context of what the author was trying to do. The variable TextFieldOutput probably doesn't look anything like that in French. Then you have all the slang that may happen because of the difference in the two languages.

Doesn't pay enough. $50k for SysAdmin is about normal. Should be adding another $50k just for "danger pay" on top of that.

Yeah, but one nice perk is free room/board for 1/4 the year. For a single person, that's worth several thousand dollars if you can sublet your place.

Still not even close to enough. I was (and still am) thinking that it'd be a neat experience, but 50k per year is a laughable amount of money considering the sacrifices. This is under the assumption that it's a full time posting. If it's something like a few months of actual work on-site, it'd be better.

But remember: it's all in the name of science.The contractors for NSF who are full-time IT support make more. But most of them are in it for the adventure.

I am sure it probably was in C++. But when all your variable names and text fields are in French, its gets rather hard to derive the context of what the author was trying to do. The variable TextFieldOutput probably doesn't look anything like that in French. Then you have all the slang that may happen because of the difference in the two languages.

Reminds me of a argument against allowing people to name local variables in the Linux source using their native tongue. I think one of the lead devs worried about getting patches from Eastern Europe that had variable names that looked like they were street signs from R'yleh.

Doesn't pay enough. $50k for SysAdmin is about normal. Should be adding another $50k just for "danger pay" on top of that.

Yeah, but one nice perk is free room/board for 1/4 the year. For a single person, that's worth several thousand dollars if you can sublet your place.

Still not even close to enough. I was (and still am) thinking that it'd be a neat experience, but 50k per year is a laughable amount of money considering the sacrifices. This is under the assumption that it's a full time posting. If it's something like a few months of actual work on-site, it'd be better.

Not to put too much emphasis on it, but the job listed in the article is for Northern hemisphere support, and there is some possibility of a shorter summer trip. However, the number of people we can get down there in a summer is pretty small and so those slots are highly prized. And note that the $50k is a minimum. ;-)

“You get a very strong appreciation for the sound dampening in commercial aircraft,” says Barnet, describing the trip. “But the food is better these days on the National Guard flights than it is on most airlines.”

Heh, the food is better on the LC-130 flight than most commercial flights. That's pretty sad and pathetic I say. I wonder if they simply hand them an MRE for the flight and say "enjoy".

“You get a very strong appreciation for the sound dampening in commercial aircraft,” says Barnet, describing the trip. “But the food is better these days on the National Guard flights than it is on most airlines.”

Heh, the food is better on the LC-130 flight than most commercial flights. That's pretty sad and pathetic I say. I wonder if they simply hand them an MRE for the flight and say "enjoy".