Navy finds billions of dollars in under-the-radar IT expenses

Jared Serbu, DoD reporter, Federal News Radio

More than a dozen years after the Department of the Navy awarded a contract to
build what it has touted as the military's first true large-scale enterprise
network, the Navy is discovering hundreds of networks and tens of thousands of
applications that have been operating for years under the radar, outside the scope of the Navy-Marine Corps Intranet and beyond the oversight of the maritime
services' headquarters.

NMCI, which was first conceived as a way to
eliminate information technology stovepipes and reduce the number of disparate,
purpose-built applications in the naval services, hosts more than 700,000 users
across the Navy and Marine Corps. The DoN is expected to award NMCI's successor,
the Next Generation Enterprise Network (NGEN), later this year.

But ever since the Navy decided it was time to cut its IT expenses by about 25
percent two years ago, its IT leadership has been making interesting
discoveries about what the service is spending on technology as it scours the
system to tally its costs and look for efficiencies. In fiscal 2011, the Navy
estimated its overall IT budget was around $7 billion, but as the Navy chief
information officer's staff began poking into nooks and crannies and developing
better "all-in" estimates, it figured last year the number was more like $9
billion.

This year, the estimate is $11 billion, said Janice Haith, the Department of the
Navy's deputy CIO who oversees IT for the Navy side of the department (as distinct
from the Marine Corps).

"That's not because the budget is getting bigger, it's because we're making people
report, and we're finding out where money is being spent. We had no idea before,"
she told an AFCEA Northern Virginia luncheon in Vienna, Va., Friday. "Both [former
Chief of Naval Operations] Adm. Gary Roughhead and [Chief of Naval Operations]
Adm. Jonathan Greenert have directed us to get a handle on this and control it."

Legacy networks getting shut down

In the effort to centralize control, the Navy has come across vast swaths of IT
infrastructure the CIO's office never knew about — servers, data centers,
applications and software licenses. Haith said the Navy, until recently, had been
operating under the mistaken presumption that it already had consolidated most of
its IT spend under NMCI.

Janice Haith, deputy CIO, Department of the Navy

"Turns out we had 302 legacy and excepted networks still sitting out there. Nobody
looked at them. They decided they didn't want to be bothered with them, but now
the CNO has said, 'Get rid of 'em. You've got until March 2014. They've gotta go.' We've started doing that, and boy, has this been pain," she said. "I have never seen so many people tell me, 'I can't move to the enterprise environment because I'm special. My data doesn't belong in the mass environment, and so you have to keep it.'"

Those arguments come before a new IT governance panel called the Navy Enterprise
Information Technology Guidance Board. Each month, it convenes to review pleas for
special dispensation from Navy IT managers who still don't want their networks to
be assimilated into NMCI.

"And guess what? We're not approving any more networks," Haith said. "We're not
even allowing waivers. We're bringing them all in to the NMCI environment. And if
we can't get them all migrated by March of 2014, they will be migrating into
NGEN."

Haith said the problem of IT operations running off the Navy's broader grid wasn't
limited to networks — data centers to support them have popped up around the world.
As part of the federal data center consolidation initiative, the Navy has been
trying to cut down its number of server facilities. First though, the service had
to determine how many it actually had. Two years ago, it did not know.

The CIO's office tasked the Navy's Space and Naval Warfare Systems Command with
counting them and determining which could be shut down.

"We did the boots-on-the-ground thing. We sent SPAWAR out in teams to go out to
all these sites and tell us what we've got, then come back and report it," she
said. "Wow. We thought we only had 6,000 servers and maybe 100 data centers. Turns
out we have 32,000 servers that we know of today, and we have 210 data centers. We
were like, 'Where did all this come from?' We have a problem. We never recorded
it. We never knew how much money we were spending on it."

Just like Gmail

The Navy decided it would try to scale back its number of data centers to just 25 and Haith said, as with the network consolidation effort, there was a rebellion in the ranks.

"They said, 'You can't take my data center. My data has to be right here with me.'
We had to say, 'Do you know what the cloud is? Do you have a Gmail account?'
That's basically what we're talking about," she said. "You don't have to have your
data with you. Education for us has been enormous in this area, particularly with
people in my generation and older, who think data has to be sitting right next to
you in order to do your job."

Haith said the perceived need to have data stored locally wound up leading to a
massive underutilization of servers the Navy already owns.

"People don't virtualize. They just don't do it. They're like, 'We'll just buy
another one.' We keep buying new servers over and over again," she said. "Our
hardware procurements, we think, are almost $500 million, just on servers, every
year."

Many of those data centers sprang up to support specific applications that Navy
components determined they needed in order to meet their individual missions. Two
years ago, the Navy said it had about 15,000 distinct software applications, and
that consolidating and eliminating them would be a huge challenge.

But once again, that turned out to be only the tip of the iceberg.

"We started doing some checking, and found out we have 42,000 applications just in
our (research, development, test and evaluation) community. Then it turns out we
have another 27,000 that are not in programs of record that need to be migrated.
And we still haven't figured out how many we have on the warfighting side," she
said. "And a lot of those applications have problems. They're not compliant with
current regulations. They're in some sort of sustainment mode, or if we get rid of
one, it will break something else. It's a big challenge for us."

The next challenge, she said, is consolidating software licenses.
Last year, the Navy signed its first big enterprise-wide licensing agreement. It
was for Microsoft products, and all Navy components now are required to use it if
they want to buy anything the Redmond, Wash., company sells. Next, the Navy
expects to sign a deal with Oracle, and 17 major providers are next in line.

And as in other areas of the Navy's IT spend, Haith said at the headquarters
level, officials still don't know what software licenses the Navy owns today, nor
whether it's using what it's already paid for.

"Not only do we not know what we've got, we can't even get anybody to tell us what we need for the future. That's also because we don't have an IT asset management tool, but we're working toward that," she said. "We've also been asked by many of the software vendors to do this, because they're tired of dealing with 23 different Navy budget submitting offices, everybody with different requirements, and then the individual program offices below that. There's big dollar savings for us there, and we expect it to happen soon."

Investment needed upfront in some cases

Even if the centralization of authority over IT spending will bring the long-run
savings the Navy's hoping for, Haith said many of the IT modernization efforts the
Navy wants to undertake will require some up-front investment. While that's a
tough sales pitch at a time when every corner of DoD is being told to cut back on
operation and maintenance spending immediately, she said it's a pitch the CIO's
office needs to continue to make.

"We have a challenge with our leadership in getting them to understand that in
order to get where they want us to be, they have to give us some money to get
there first," she said. "All of the other services have the same problem. We
cannot get them to understand. If it's going to cost us close to $50 million to
close data centers and the return on investment is $200 million, you've got to
give it to us up front. We can't cut everything right now, because we'll never get
there."

But the Navy's IT leadership bears some blame for failing to articulate those
possible returns on investment, because it hasn't been able to make concrete
assessments of precisely where the money's going, Haith said.

"For instance, on data center consolidation, we didn't look at the full gamut. We
only looked at hardware, software, and people," she said. "But there was an energy
cost, and unfortunately, we're only starting to put smart meters on our buildings,
so we weren't sure what the [energy] cost was. We didn't think about the security
in the buildings, we didn't think about tech refresh, we only looked at the here
and now. We didn't think about all the people to take care of the applications,
because we didn't even know how many we had."

In the short term, amid the Pentagon's uncertainty about the 2013 budget, the Navy
has ordered another 25 percent reduction in IT spending in this year alone. In
response to a question from Federal News Radio, Haith said the initial guidance
from Jonathan Greenert, the chief of Naval Operations, appeared to be based on the
assumption that the Navy was incurring exorbitant expenses on day-to-day office
technology such as mobile devices and printers, and that the spending could be
quickly scaled back.

"That's not where we're spending the money," she said.

The spending, she said, is on servers and other IT infrastructure. In response to
Greenert's guidance, which is intended to help the Navy conserve as much cash as
possible in anticipation of both a full-year continuing resolution and budget cuts
under sequestration, Haith issued a memo last week, telling Navy components that
many big IT purchases were to be put on hold.

"I was surprised my car was still out in the parking lot after that memo," she
said. "But I have a $32 million purchase request for servers sitting on my desk
right now, and I'm like, 'Why? Why do you need $32 million worth of servers for
this capability? We're putting a restriction on it. It's got to be mission-
essential in terms of loss of life to our sailors if we don't do it, or we can't
deliver a critical milestone in a program of record. Otherwise, it's going to sit
in a hold pile until we get some resolution in this budget. We're scrutinizing
everything."