NMCI, now being phased out, is the world's biggest intranet, and its biggest target.

In 2012, Iranian hackers managed to penetrate the US Navy’s unclassified administrative network, the Navy Marine Corps Intranet. While the attack was disclosed last September, the scale of it was not—the attack gave hackers access to the NMCI for nearly four months, according to an updated report by The Wall Street Journal.

Vice Adm. Michael Rogers, who is now President Barack Obama’s choice to replace Gen. Keith Alexander as both NSA director and commander of the US Cyber Command, led the US Fleet Cyber Command when the attack came to light. Rogers' response to the attack may be a factor in his confirmation hearings.

Iranian hackers attacked NMCI in August of 2012, using a vulnerability in a public-facing website to gain initial access to the network. Because of a flaw in the security of the network the server was hosted on, attackers were able to use the server to gain access to NMCI’s private network and spread to other systems. While the vulnerability that allowed the attackers to gain access in the first place was discovered and closed by October, spyware installed by the attackers remained in place until November.

Officials said no e-mail accounts were compromised and no data was stolen in the attack. But it cost about $10 million to repair the damage done to the network’s systems—a process that included taking the whole network down twice for upgrades to systems and removal of malware.

The attack on NMCI is not exactly surprising. The unclassified network, which encompasses approximately 70 percent of the Navy’s IT operations, is the largest corporate intranet in the world, supporting over 800,000 users at 2,500 sites. It also hosts a vast majority of the Navy’s public-facing websites, making it a huge target. And while the network has had the benefit of frequent updates and fairly consistent security by corporate intranet standards, it’s also been the target of criticism (and sometimes ridicule) almost since its inception. Early growing pains caused sailors and Marines to complain that NMCI stood for “No More Computer Information.”

Launched in 2000, NMCI was the Navy’s grand experiment in outsourcing. The network is a managed service, originally deployed as the result of a 10-year contract to EDS (now Hewlett-Packard Enterprise Services). Part of the justification for NMCI’s managed service approach was security. The managed service contract was supposed to make it easier to support and maintain the network—and enforce a common level of anti-malware measures, network security, and user training. Before NMCI, nearly every Navy shore command had its own network, often operated by sailors and Marines with an inconsistent amount of training.

But security remained a headache for NMCI, largely because of the number of legacy applications that EDS had to deal with, which often used custom TCP/IP ports and protocols. While the Navy claimed in 2006 that it had “never suffered a root-level intrusion and has thwarted attacks that penetrated other DOD systems on several occasions,” the lack of visibility into what was happening on the network gave the Navy very little to work with in terms of understanding what kinds of attacks might have been possible. All the network operations data belonged to EDS.

The security measures that have been taken to secure NMCI have also made it a lot less of an "intranet" and more of a collection of smaller, standardized networks. There's so much segmentation of the network that even e-mail attachments couldn't be effectively shared between a Navy user and a Marine Corps user, ostensibly on the same infrastructure. It didn't help that EDS lost nearly $3 billion on the original deal—which may be part of why the company ended up being acquired by Hewlett-Packard.

Even after NMCI was fully rolled out, the Navy had over 500 other separate networks. In 2009, as the contract was nearing its end—and with concern about “cyber threats” against military networks on the rise—the Navy was still planning the acquisition of a follow-on network that would give it greater control over its operation and defense and help it to roll in even more networks. At the same time, it continued to move more of its administrative users over to NMCI in the hopes of improving security and greasing the skids for consolidation with a new intranet called Next Generation Enterprise Network (NGEN).

But as the original NMCI contract expired, the Navy found itself in a bind. Since Hewlett-Packard owned all of the data about the network’s operation, simply shifting to a self-run network wasn’t going to happen by waving a magic wand, and the planning process had been hampered by a lack of information—information the Navy had to buy from HP. So the Navy pulled together a sole-source contract for HP to continue to support NMCI past 2010 while it figured out what to do next.

Now, four years later, the Navy is finally in the middle of the first phase of getting off NMCI and moving to NGEN. The winner of that contract, which could end up totaling over $3.4 billion dollars, was HP. The side contract to maintain the existing NMC, which is expected to last at least another year, could bring in as much as $6.1 billion before it is done. And that doesn’t include the millions the Navy will spend to buttress network security overall through its Fleet Cyber Command.

I hope they at least learned some lessons and the new contract has terms that require the delivery of adequate documentation. Also, given the way security works in practice, I think it would make sense to give bonuses for time passed with no exploits and penalties for hacking incidents.

This is embarrassing, but not quite as embarrassing as the guided missile frigate USS Taylor having been run aground and rendered inoperative during a special deployment to the Black Sea monitoring the security situation in Sochi.

Officials said no e-mail accounts were compromised and no data was stolen in the attack. But it cost about $10 million to repair the damage done to the network’s systems—a process that included taking the whole network down twice for upgrades to systems and removal of malware

How do they get $10m? I really want to know. It seems like they should have been doing most of that anyway, I.E. updates/patches. The only thing I could see being a pain is the malware and the fact they don't get to plan the upgrade now, it's got a reactive update.

Quote:

and the planning process had been hampered by a lack of information—information the Navy had to buy from HP...The winner of that contract, which could end up totaling over $3.4 billion dollars, was HP. The side contract to maintain the existing NMC, which is expected to last at least another year, could bring in as much as $6.1 billion before it is done

So not only did the USN get blackmailed by HP, they then awarded them a new contract on top of the other contract. Roughly worth $9.5 billion. Which if history tells me anything will probably run a couple billion over and behind schedule and still have issues. I hope the USN had a way better contract this time that included documentation and what not.

Maybe I've got a jaded point of view, but it seems like there a lot of these contracts that are always over time, budget, have issues and the U.S government keeps handing out more contracts to the same companies. If I didn't ever complete a project on-time and then told my boss I needed more money, I don't think I'd have a job much longer.

The U.S. really needs to get it's cyber security shit together, or Dragon Day is not an 'if', but a 'when'.

That drone was not hacked. It does not use GPS for navigation, for one. It uses inertial guidance. GPS is just used to check the accuracy of the INS system. And it certainly does not use GPS as you know it. Military GPS is encrypted, and if Iranians cracked that we'd know it by now big time. Worst possible thing Iranians could have done to it, and that's a stretch, is jam its command and control link back to US operators and cause it to go autonomous. It's anyone's guess what happens then. Probably uses INS to finds its way around and follows a preprogrammed flight path. But no they did not force it to land by spoofing GPS that's laughable.

My personal guess is the drone had a problem, and made a hard landing in iran and was heavily damaged. They put it back together, repainted it, and showed it off. Assuming they didn't just make a fiberglass replica, and the real drone is in 100,000 pieces. It's a well established fact that iran exaggerates their military capabilities. Google their stealth jet fighter press conference.

EDS's policy was to underbid (ok, lots of contractors do that), but with a computer spec that was fairly decent (at time of bid). Nobody else could match that spec at that cost.

But EDS didn't actually have to deliver that spec for a couple years. And which point, what was a decent spec at bid time, was "stuff collecting dust that wasn't really selling".

So they ~should~ have made a lot of money, at least on hardware.

The problem they ran into, however, was that while that hardware spec they delivered (which, let's face it, was on the low end of 'ok' by that point) was what they bid on, it wasn't spectacularly capable for the software that was then available. (I.e., OS and apps had moved on, and needed more.)

So EDS had to do some mid-term upgrades to systems, adding memory - which was about all they could do.

Then they ran head on into some government policies on network gear that said, "If the network switch is not Common Criteria certified, you can't use it". This wouldn't have been a problem... Except the switch manufacturer had forked the firmware, and all the lower end switches fell into the side of the fork that didn't maintain CC status. Which was an unexpected cost that EDS had to eat (because their contract said such). Replacing several thousand network switches tends to eat up a decent chunk of profit.

EDS's policy was to underbid (ok, lots of contractors do that), but with a computer spec that was fairly decent (at time of bid). Nobody else could match that spec at that cost.

But EDS didn't actually have to deliver that spec for a couple years. And which point, what was a decent spec at bid time, was "stuff collecting dust that wasn't really selling".

So they ~should~ have made a lot of money, at least on hardware.

The problem they ran into, however, was that while that hardware spec they delivered (which, let's face it, was on the low end of 'ok' by that point) was what they bid on, it wasn't spectacularly capable for the software that was then available. (I.e., OS and apps had moved on, and needed more.)

So EDS had to do some mid-term upgrades to systems, adding memory - which was about all they could do.

Then they ran head on into some government policies on network gear that said, "If the network switch is not Common Criteria certified, you can't use it". This wouldn't have been a problem... Except the switch manufacturer had forked the firmware, and all the lower end switches fell into the side of the fork that didn't maintain CC status. Which was an unexpected cost that EDS had to eat (because their contract said such). Replacing several thousand network switches tends to eat up a decent chunk of profit.

If the government changes their requirements, shouldn't the contractor get the opportunity to update their budget?

I've been with the Navy for nearly 12 years, and I've never heard anyone here say that NMCI stands for "No More Computer Information".

Now... "No More Computing Infrastructure"... that one has been pretty common, especially among the civilian personnel who used to support all the stuff that EDS took over.

When I was writing about defense things in the government market, five separate Navy people (civ and mil) used the "computer information" line. It was because of glitches with email and access to legacy applications, which were port-blocked.

We are too busy spying on our own people to recognize the real threats. Our focus is not on terrorism but on what American's are doing. If Benghazi has taught us nothing else. Is that Obama is bored with defending America. His sole focus is on American's and controlling their health care, retirement and incomes. He has no interest in defending freedom or liberating others who want it so desperately. \Our military is at its weakest point in many years. Probably since Regan years. To be hacked for so long and not be discovered is embarrassing to say the least. Most likely worse then anything Snowden did.

EDS's policy was to underbid (ok, lots of contractors do that), but with a computer spec that was fairly decent (at time of bid). Nobody else could match that spec at that cost.

But EDS didn't actually have to deliver that spec for a couple years. And which point, what was a decent spec at bid time, was "stuff collecting dust that wasn't really selling".

So they ~should~ have made a lot of money, at least on hardware.

The problem they ran into, however, was that while that hardware spec they delivered (which, let's face it, was on the low end of 'ok' by that point) was what they bid on, it wasn't spectacularly capable for the software that was then available. (I.e., OS and apps had moved on, and needed more.)

So EDS had to do some mid-term upgrades to systems, adding memory - which was about all they could do.

Then they ran head on into some government policies on network gear that said, "If the network switch is not Common Criteria certified, you can't use it". This wouldn't have been a problem... Except the switch manufacturer had forked the firmware, and all the lower end switches fell into the side of the fork that didn't maintain CC status. Which was an unexpected cost that EDS had to eat (because their contract said such). Replacing several thousand network switches tends to eat up a decent chunk of profit.

If the government changes their requirements, shouldn't the contractor get the opportunity to update their budget?

The government didn't change the requirement, EDS didn't know about the requirement, and neither did the government people who let the contract. Then, later, when someone who worked in IT, not contracting, realized the problem, EDS had already committed to deliver.

How do they get $10m? I really want to know. It seems like they should have been doing most of that anyway, I.E. updates/patches. The only thing I could see being a pain is the malware and the fact they don't get to plan the upgrade now, it's got a reactive update.

Some possible sources of this cost: motherboards and hard drives that had to be replaced due to BIOS or MBR infection. Labor costs for replacement parts installation and other malware removal tasks. Extra security scans specialized for known attack vectors. In a network of 100k+ machines, $10m is less than $100 per machine, which isn't very high at all if hardware replacement was needed. And since the article mentions over 800k users, it's possible the number of systems on the network is even higher.than the 100k ballpark.

I agree that routine security patches shouldn't be included in the cost estimates for the intrusion, but once the network is infected, the activities required to actually remove the malware can quickly add up. Whether or not these costs were billable under the contract or were absorbed by HP is not documented in the article.

Quote:

Quote:

and the planning process had been hampered by a lack of information—information the Navy had to buy from HP...The winner of that contract, which could end up totaling over $3.4 billion dollars, was HP. The side contract to maintain the existing NMC, which is expected to last at least another year, could bring in as much as $6.1 billion before it is done

So not only did the USN get blackmailed by HP, they then awarded them a new contract on top of the other contract. Roughly worth $9.5 billion. Which if history tells me anything will probably run a couple billion over and behind schedule and still have issues. I hope the USN had a way better contract this time that included documentation and what not.

Maybe I've got a jaded point of view, but it seems like there a lot of these contracts that are always over time, budget, have issues and the U.S government keeps handing out more contracts to the same companies. If I didn't ever complete a project on-time and then told my boss I needed more money, I don't think I'd have a job much longer.

I'll tread carefully in responding to your characterization of this as "blackmail" (see my disclaimer below), but I will note that a government contractor cannot provide free services outside the scope of a contract. Otherwise those services might be considered a gift given with the purpose of influencing future contract awards (bribery). So if the network logs were outside the scope of the contract, HP was required by law to charge fair market value.

Also note that the article states that "EDS lost nearly $3 billion on the original deal." So from the Navy's point of view, this was a fairly balanced contract - the vendor had to absorb a significant portion (most?) of the cost and time overruns, to the point where they lost significant money on the deal. The fact that the time and cost overruns existed in the first place is a question I will pass over, because that brings me into the realm of defending or publicly criticizing the predecessor of my employer based on very limited information and differing points of view. If you Google "GAO report on NMCI contract performance", you can see the official reports on this topic.

Note: I am an employee of HP ES, but the views here are my own and are not based on any internal knowledge of the NMCI contract or systems. I have not supported NMCI at any time in my career with HP ES and don't have any internal knowledge regarding this breach in any case. In particular, the discussion of possible cost factors related to the intrusion is pure speculation on my part.

Sounds like we need to develop something like Tor node appliances that can connect via a variety of protocols. Create an extensible VPN by adding more nodes.

The crucial part (and hence appliance) is that the nodes self update and perform P2P validation of each other so that if one is compromised it can be rejected, short of every other node being compromised simultaneously.

This was an non-classified public-facing network? Then I'm not seeing the issue, particularly. If they'd hacked internal TS/SCI networks or similar then that would have been an issue. I'm not all that excited by this, news as it may be. I'm sure the *real* pros at running attacks on US military networks are still busy exfiltrating untold terabytes of data regardless, all the way back to Moscow and Beijing...

Now, what I'd like to know is what "Rogers' response to the attack may be a factor in his confirmation hearings" refers to. General Keith Alexander is on public record as saying aggressive attacks are an appropriate response to foreign hacks so my money would be on the Iranians being on the receiving end of a cyberattack or even a kinetic one in response, that being US policy. The US isn't going to just take punches like this, even light ones, from a country like Iran.

EDS's policy was to underbid (ok, lots of contractors do that), but with a computer spec that was fairly decent (at time of bid). Nobody else could match that spec at that cost.

But EDS didn't actually have to deliver that spec for a couple years. And which point, what was a decent spec at bid time, was "stuff collecting dust that wasn't really selling".

So they ~should~ have made a lot of money, at least on hardware.

The problem they ran into, however, was that while that hardware spec they delivered (which, let's face it, was on the low end of 'ok' by that point) was what they bid on, it wasn't spectacularly capable for the software that was then available. (I.e., OS and apps had moved on, and needed more.)

So EDS had to do some mid-term upgrades to systems, adding memory - which was about all they could do.

Then they ran head on into some government policies on network gear that said, "If the network switch is not Common Criteria certified, you can't use it". This wouldn't have been a problem... Except the switch manufacturer had forked the firmware, and all the lower end switches fell into the side of the fork that didn't maintain CC status. Which was an unexpected cost that EDS had to eat (because their contract said such). Replacing several thousand network switches tends to eat up a decent chunk of profit.

If the government changes their requirements, shouldn't the contractor get the opportunity to update their budget?

The government didn't change the requirement, EDS didn't know about the requirement, and neither did the government people who let the contract. Then, later, when someone who worked in IT, not contracting, realized the problem, EDS had already committed to deliver.

The above is speculation, but I'd be surprised if I was wrong.

Actually, I think it was more: EDS knew about the requirement (Common Criteria), but they didn't expect that the switch manufacturer would fork the firmware and not certify both sides of the fork.

I don't really want to give too much cover to EDS here, but I don't think that was something that they legitimately could have forseen.

Now... that they put off the upgrade until the last possible moment, and ended up paying a ton of OT to get it done before the deadline... that was preventable.

I've been with the Navy for nearly 12 years, and I've never heard anyone here say that NMCI stands for "No More Computer Information".

Now... "No More Computing Infrastructure"... that one has been pretty common, especially among the civilian personnel who used to support all the stuff that EDS took over.

When I was writing about defense things in the government market, five separate Navy people (civ and mil) used the "computer information" line. It was because of glitches with email and access to legacy applications, which were port-blocked.

Out of curiousity, were they all East Coast or Gulf people?

I ask because they've had the unfortunate position of being the unofficial beta testers for the entire run of the NMCI contract, in a sort of, "the closer you are to Washington Naval Yard, the earlier you get the roll-out" policy.

Whereas, by the time it hit the West Coast, the most obvious of the bugs have generally been worked out of the system. (I should note that obvious is not the same as annoying.)

The U.S. really needs to get it's cyber security shit together, or Dragon Day is not an 'if', but a 'when'.

That drone was not hacked. It does not use GPS for navigation, for one. It uses inertial guidance. GPS is just used to check the accuracy of the INS system. And it certainly does not use GPS as you know it. Military GPS is encrypted, and if Iranians cracked that we'd know it by now big time. Worst possible thing Iranians could have done to it, and that's a stretch, is jam its command and control link back to US operators and cause it to go autonomous. It's anyone's guess what happens then. Probably uses INS to finds its way around and follows a preprogrammed flight path. But no they did not force it to land by spoofing GPS that's laughable.

My personal guess is the drone had a problem, and made a hard landing in iran and was heavily damaged. They put it back together, repainted it, and showed it off. Assuming they didn't just make a fiberglass replica, and the real drone is in 100,000 pieces. It's a well established fact that iran exaggerates their military capabilities. Google their stealth jet fighter press conference.

I totally agree with you, but the DoD likes to say that Sentinel was hacked. Possibly to get more funding. Now Iran claims they hacked it, but...well you know. There are lies, damnable lies, and military press conferences.

EDS's policy was to underbid (ok, lots of contractors do that), but with a computer spec that was fairly decent (at time of bid). Nobody else could match that spec at that cost.

But EDS didn't actually have to deliver that spec for a couple years. And which point, what was a decent spec at bid time, was "stuff collecting dust that wasn't really selling".

So they ~should~ have made a lot of money, at least on hardware.

The problem they ran into, however, was that while that hardware spec they delivered (which, let's face it, was on the low end of 'ok' by that point) was what they bid on, it wasn't spectacularly capable for the software that was then available. (I.e., OS and apps had moved on, and needed more.)

So EDS had to do some mid-term upgrades to systems, adding memory - which was about all they could do.

Then they ran head on into some government policies on network gear that said, "If the network switch is not Common Criteria certified, you can't use it". This wouldn't have been a problem... Except the switch manufacturer had forked the firmware, and all the lower end switches fell into the side of the fork that didn't maintain CC status. Which was an unexpected cost that EDS had to eat (because their contract said such). Replacing several thousand network switches tends to eat up a decent chunk of profit.

If the government changes their requirements, shouldn't the contractor get the opportunity to update their budget?

LMAO, almost NEVER does the contractor get to request additional funding. It's call requirements creep or scope creep and is the single biggest reason (although definitely not the only one) for contractor overruns.

EDIT: Not saying that this is what happened, but I have seen multiple instances of vague government requirements that half way through the program they say - oh, we really meant this. Any assumptions you made at the time are brushed off as irrelevant. Contractors will usually eat the cost because they want the next contract. This usually happens on R&D type contracts, rarely on production or maintenance contracts because the scope is much better defined.