How VA's $36 million move to the cloud evaporated

By Adam Mazmanian

Jun 04, 2014

The Department of Veterans Affairs canceled a $36 million cloud computing deal with HP in May 2013 after a dispute between the CIO's office and the agency's inspector general over how long emails should be retained and concerns over system security, FCW has learned.

When the deal was announced in November 2012, it was one of the most ambitious cloud migrations of any federal agency, covering the VA's entire 600,000-strong workforce. HP Enterprise Services was selected as the integrator to deploy the system, which was set to go live with a March 2013 pilot for 15,000 users, involving calendar and email apps.

But the pilot didn't get off the ground because of "serious concerns" about the system's 90-day retention period for emails. Those concerns were raised in a Feb. 20, 2013, memo signed by acting Inspector General Richard J. Griffin that was obtained by FCW in a records request.

VA emails in the HP cloud, as explained in Griffin's memorandum, would be retained for 90 days before being automatically purged. The loss of emails, Griffin wrote, "raise[s] major concerns about our accountability and transparency in VA, VA's ability to defend itself in litigation, and, in particular, the OIG's ability to conduct our statutory independent and objective oversight of VA programs and operations."

More from FCW

The March pilot was going online just as then-CIO Roger Baker was preparing to depart, and Steph Warren, currently the head of the Office of Information and Technology (OIT) at VA, was preparing to take over on an interim basis. The 90-day retention order was put on hold, pending a study of the issue by a group of stakeholders, including OIG, the VA's general counsel and the National Archives and Records Administration, which sets government-wide policy for the storage of records.

By May 24, 2013, the deal was scuttled. The OIG wanted new contract language inserted into all VA cloud contracts designed to facilitate access and visibility into the system, preserve emails and increase the security rating under the Federal Information Security Management Act. There was pending guidance from NARA on records retention that would affect the disposition of email storage. It was determined that the necessary changes were out of scope with the HP contract, and it was terminated.

"The contract was awarded before the unique VA OIG requirements were fully elicited by the organization," Charles De Sanno, executive director of enterprise systems engineering at VA, wrote in a memorandum terminating the deal with HP.

Baker and Warren received an email – the senders' name was redacted in the FOIA request -- in August 2011 as the system was being drawn up, advising that they include language covering access for audit and investigation purposes in any contract for cloud services.

It's not clear from Warren's responses to Griffin's February 2013 memo whether the emails in the HP system were to be permanently expunged or automatically archived. The request for information put out by VA in 2011 suggests that all emails would be archived physically and retrievable via "rehydration," according to the contracting document. The 90-day limit, a VA spokesperson told FCW, "was the time for materials determined to be non-record" to live in the cloud.

Whatever the reason, the 90-day limit was not determined by capacity. The cloud-based system provided for 25 gigabytes for each user account. The average mailbox size under the local Exchange server system was about 150 megabytes, according to the OIG.

Personally identifiable information

The memo traffic on the cloud issue expanded to include other oversight issues regarding OIG's access to agency email.

The move to require personal identification badges to access VA computers had the effect of encrypting all email. In order to access email for oversight purposes, OIG investigators had to request decrypts from OIT, and requests quickly piled up, creating a weeks-long backlog. Eventually an interim solution was found, and since that time a vendor was identified to perform email decryption on behalf of internal VA customers with oversight responsibilities. That application is in development.

More from FCW

OIG was also frustrated by the inability or unwillingness of IT executives to comply with a request for any email aliases used by senior officials. While the OIT eventually supplied a computer-generated accounting of aliases in the VA systems, they did not provide "a list that was responsive to the request about senior leaders," said James O'Neill, deputy inspector general for investigations.

Finally, OIG was concerned that the security rating for the system was not high enough, considering that personally identifiable information might be moving across the VA cloud. The VA had contracted for a system rated "moderate" under the Federal Information Security Management Act regulations. The OIG wanted a rating of "high" because of the possibility that personally identifiable information could be at risk.

The FISMA ranking continues to be a sticking point. The OIG hasn't moved off of its determination that the VA's cloud should be "FISMA high". The VA's information security and IT operations experts recommended that "FISMA moderate was appropriate for this particular contract because VA's IT system is not officially a Privacy Act system of records, and because VA's email systems are not to be used to transmit sensitive information without encryption," a VA spokesperson told FCW.

According to documents, about $870,000 was obligated to the contract, but the dollar amount of sunk costs into the defunct contract are likely far higher, considering the staff time that went into the contracting process.

The VA still faces an email crunch. As Warren noted in a March 22, 2013, memorandum, the current VA email system dates back to 2006. The VA is maintaining the old emails and a "voluminous" number of attachments at "significant" cost," Warren wrote, adding: "If VA can migrate to technology such as cloud email and come to agreement on a reasonable retention period for email, the cost savings to VA will be considerable."

At this point, VA is retaining all emails indefinitely. "VA will revisit its email retention policy once [NARA] completes revisions to its guidance in this area," a spokesperson said. There are no current plans to put out a solicitation for a cloud migration.

HP had little to say on the matter.

"HP understands the Department of Veterans Affairs elected to terminate the Microsoft Exchange contract while it broadly re-evaluates its requirements to potentially move to a cloud-based solution with Microsoft Office 365. HP looks forward to continued work with the VA to address the agency's cloud security and privacy requirements," the company said in an e-mailed statement.

OPM is partnering with CSID to try to manage the fallout from a massive breach of some 4 million federal personnel records.

Reader comments

Mon, Jun 30, 2014

"Many of these SaaS providers do NOT use backup tapes because of their high-availability (never needing to restore because data is in 10 places). This being said, if a user deletes an email and it was not properly legally-held and/or retained under a records management program, after 90 days, that file/email is gone forever. "
Exactly. People need to stop confusing backup, which is an IT operations function, with records management/retention, which is a policy/business practice.
And there are many third party email journaling vendors that work with Office 365 - including, if i'm not mistaken, Autonomy which HP acquired recently.

Mon, Jun 9, 2014
Eton Klatu

Baker tried hard, but the yapping careerists did not want to change and locked arms with their bungling contractors to ensure no change. VA is a major monument to botched IT systems, botched IT acquisition processes, and highly unproductive IT operations.

Mon, Jun 9, 2014
Twenty-five year federal IT professional
DC

For OIG at VA to request a "High" rating, there has to be more to the story than PII. Moderate includes PII for identity theft, public humilation and blackmail potential. Quoting the NIST guidance on protecting PII:
Harm to individuals as described in these impact levels is easier to understand with examples. A breach of the confidentiality of PII at the low impact level would not cause harm greater than inconvenience, such as changing a telephone number. The types of harm that could be caused by a breach involving PII at the moderate impact level include financial loss due to identity theft or denial of benefits, public humiliation, discrimination, and the potential for blackmail. Harm at the high impact level involves serious physical, social, or financial harm, resulting in potential loss of life, loss of livelihood, or inappropriate physical detention.

Thu, Jun 5, 2014

What irks me most is that the Federal Government is big enough to build their own cloud. Realistically a cloud is no more than a central storage and retrieval facility. The Government is a unique entity in that it is required to provide certain information to the public because the public is, after all, the owner and collect certain information from the people for the purpose of governance. This is not the business of private industry. Data collection in this country use to be very straight forward; the government only collected information that was necessary to perform its mission and it would protect the information that it did collect. Perhaps CIO's and Agency Executives should not only pay attention to the privacy act but also read the paperwork reduction acts. Here's an idea; take the storage facility that NSA built, purge all the data that was illegally collected and place the Government cloud there, then archive everything in the cloud like the wayback machine does. Stop making the simple complex and this will translate into the Government saving money, if for no other reason, because they stop wasting money.

Thu, Jun 5, 2014

I believe the VA was planning a move to Office 365 for government. Many of these SaaS providers do NOT use backup tapes because of their high-availability (never needing to restore because data is in 10 places). This being said, if a user deletes an email and it was not properly legally-held and/or retained under a records management program, after 90 days, that file/email is gone forever. This has nothing to do with the contract verbiage and it's just what comes canned with Office 365. It would be like putting in the contract "Microsoft word shall be able to make phone calls." The VA OIG's concern over encrypted email not being able to be decrypted quickly is more a business function from their IT department and has little or nothing to do with Cloud. yes, the email internally may not need to be encrypted, but once it goes out, it should be. Their FISMA high seems crazy... but who knows, maybe there IS information on veterans that, if spilled, could cause loss of life. If this is the case, they have NO cloud options at the moment. But what this does raise is the need to include OIG in agency cloud requirements elicitations.