New Details, Lessons in White House E-Mail Debacle

Some answers help to explain how millions of e-mails became lost -- with some key archiving tips for enterprises, as well.

If enterprises can learn anything from the White House's e-mail problems now under investigation, it's that the task of safely preserving messages shouldn't be taken lightly.

New details -- along with new lessons for businesses -- are coming to light concerning the loss of millions of White House e-mails. According to recent congressional documents, the data loss may have been tied to a platform migration from IBM Lotus Notes to Microsoft Exchange, beginning in 2002.

Earlier news had indicated that the Bush administration's e-mail archiving system also had not been up to par, with e-mail backup tapes being recycled.

Since then, a White House IT staffer has suggested that the data loss took place in part due to oversight lapses during the e-mail server migration -- in spite of admins' worries about just such an incident.

"There was a great deal of concern about proceeding with the migration to Outlook/Exchange without having an adequate e-mail records management solution in place," wrote Steven McDevitt, an IT admin in the Executive Office of the President (EOP), responding to inquiries from the U.S. House of Representatives' Committee on Oversight and Government Reform.

"The process by which e-mail was being collected and retained was primitive and the risk that data would be lost was high," McDevitt wrote in his letter to Rep. Henry Waxman (D-Calif.), the committee's chairman.

"By early 2003, an entire year had been spent trying to identify a solution that would support the e-mail records management requirements," McDevitt said.

He added that some EOP personnel had been concerned that "system problems would create a public perception that the EOP was unwilling or unable to retain records that were required under current law."

Despite the concerns, McDevitt said the White House failed to implement an audit system following the switch to Exchange, which would have ensured e-mails were archived and preserved.

He added that the staff took steps to address the problem in late 2005.

In addition to those shortcomings, McDevitt also said that the staff found potential security holes relating to the Exchange .pst files containing stored e-mail.

"In mid-2005 ... a critical security issue was identified and corrected," he wrote. "During this period it was discovered that the file servers and the file directories used to store the retained .pst files were accessible by everyone on the EOP network."

McDevitt added that he had not been involved in managing the White House's backup tape policy, and couldn't shed any light on whether the tapes were indeed recycled, potentially resulting in additional data loss.

Not surprisingly, storage industry players and IT analysts are warning enterprises to take steps to avoid similar predicaments.

"What this should tell every business is that you have to archive and backup and do it now," said Kieron Dowling, president and CEO of Jatheon Technologies, a Toronto-based company that makes a network appliance for archiving, retrieving and monitoring e-mail and messaging data.

"Politics aside, if the White House can be sued for failure to maintain an adequate e-mail archive, any organization is at risk," Dowling said. "And a growing number of organizations are required to maintain secure, accessible electronic records, per regulations."

Those regulations include the Federal Rules of Civil Procedure, Sarbanes-Oxley, the USA PATRIOT Act, the Health Insurance Portability and Accountability Act, Security and Exchange Commission rules, various state laws and corporate policies.

"Depending on your need, there is the right technology out there," Dowling said. "Something went very wrong with the White House's e-mail archival and backup strategy and it shouldn't have happened."

Often, IT admins and other officials fail to fully appreciate the costs of implementing a bad archival system -- or none at all -- and drop the ball as a result.

McDevitt admitted in his response to Waxman that his department had been worried about the expense of correcting e-mail migration or data-loss problems, as well as the public-image impact that might result should e-mails go missing.

As it turns out, White House officials have stated that reconstructing lost documents from disaster recovery tapes could cost upwards of $15 million.

"The labor costs can be astounding when you're talking about going through thousands of messages and reconstructing e-mail boxes," Dowling said. "It's like finding a needle in the haystack in recreating the messages."

"Obviously, the White House didn't have an adequate system, and that's a serious issue as there are plenty of products out there for good archival and retrieval," he added.

Despite Dowling's assertion, McDevitt said in his letter to Congress that finding an adequate product had proven difficult.

"By early 2003, an entire year had been spent trying to identify a solution that would support the e-mail records management requirements of the EOP," he wrote.

Additional security holes could be opened when IT admins and other officials simply fail to understand the differences between a simple backup process and secure archiving, Dowling said.

Ensuring they adopt a full-fledged archival system could help mitigate problems even in situations such as the exposure of the EOP's .pst to its entire network. In such a case, a secure archived system may face less opportunity for tampering.

"The thing many people don't realize is that while archiving can protect e-mail from being tampered with once captured, backup files can be accessed, with e-mails even deleted, with no way to prove the tampering took place," Dowling said.

Other industry professionals suggested that e-mail migrations can be made be far simpler.

"There a few pitfalls of moving from one to another platform, and it's in your best interest to download all of the mail to another local mail viewer before re-pointing the mail to a new provider or location," Adam Schwam, president of IT consultancy Sandwire, told InternetNews.com. "Be as prepared as possible, and with the right people on task, the process should be almost seamless."

Similarly, Dowling added that "what a good solution will do is take something that's very complex and make it very simple."