Rating and Stats

Document Actions

Share or Embed Document

Description: On September 25, 2007 Citizens for Responsibility and Ethics in Washington (CREW) filed a lawsuit against the Executive Office of the President, the Office of Administration and the National Archiv...

On September 25, 2007 Citizens for Responsibility and Ethics in Washington (CREW) filed a lawsuit against the Executive Office of the President, the Office of Administration and the National Archives and Records Administration in the District Court for the District of Columbia. CREW's action challenges as contrary to law those parties' knowing failure to recover, restore, and preserve millions of electronic communications created and/or received within the White House. The lawsuit stems from the millions of e-mails that were improperly deleted from White House servers and exist on back-up tapes, if at all.; FOIA Request: CREW versus EXECUTIVE OFFICE OF THE PRESIDENT ET AL (Lawsuit); Holder of Document: CREW; Producing Agency: Executive Office of the President; Date Received: 4/1/2010;

There are three sets of files requiring backup that are associated with the ECRMS s~n. '

These are:

• Source PST files that exist on_

• System files and ECRMS application files that exist on the ECRMS servers

• ECRMS data

Eo{) lOA/ 0(10 IIStT

Both the source PST files located on SFEOPOI and the ~stem and application files located on the ECRMS servers are backed up as part of the standarflS&T backup process. This process extends over a 14-day period, with a full backup, incremental b ups, and a level, I rollup followed by additional incremental backups. For reference, is of the ECRMS server' directories that are included in the standard IS&T backup and a provided in Appendix A.

tJ.

The focus of this document is the backup and restore of the ECRMS data. With ECRMS being

the integration of three COTS products, MDY's FileSurf, Symantec's Enterprise Vault and

The main requirement of EOP for this backup and restore process is data integrity. The integrity of the data after a restore is much more consequential to EOP than, for example, the time it takes to perform the restore itself. Regarding the backup and restore timeframes:

• The goal is to minimize the backup window in order to maximize the ECRMS operational time, specifically to import the daily emails and legacy emails. There is a large backlog of legacy emails, so the more time that can be allocated to the import of these emails, the shorter the timeframe to complete the processing of the entire backlog. However, this goal cannot compromise the data integrity requirement.

reducing the backup window. It is acceptable to EOP if the data restore takes a couple days. This is the case since the number u ers: f the system is very small and the need to interact with the system (for searches, expo~ and rrco ""management) is not on a consistent basis.

is Wfti t r(,\'\ll~ . Note that the backup of the source PST files (on SFEOPO allows a second lelel of recovery in the event that there is all, iS6ye with the ECRMS data b kup/restore proces! The source PST

files can be reprocessed from wherever the ECRMS data . This means taking the time to] (cvlu51~

process PST files that were processed in the past, but in some circumstances, this may be the ~ Wi M4Ih,~

4. Backup of the ECRMS full text indexes by copying these files to another disk location.

5. The files resulting from steps 2, 3 and 4 will be written to tape as part of the standard IS&T backup procedure.

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGE.

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044386

j. ECRMS BACKUP PLAN

h (N'e Mlv.M In tc\

I - Replication of the ECRMS rchive em~ccur between the Primary C ~tera and a secondary offsite Centera. Th repji tio'U be scheduled to occur during certt: time periods.

2 - SQL Maintenance Plans will be created to back up the transaction logs for the ECRMS databases. During ECRMS operational hours, the FileSurfDatabase (ECRMSDB) logs will be backed up hourly, while the Enterprise Vault Database (Directory and Vault Store) logs will be backed up every 3 hours. The SQL Maintenance Plans will also be set to delete transaction logs older than 2 weeks.

3 & 4 - A master script will be developed to control the backup of the ECRMS databases and full text indexes, since these must occur 'in sync' and only when the ECRMS services are in a quiescent state. The master script will stop all the ECRMS services, then back up the ECRMS

data from the various sources and then restart the services.}J., .

. /: W(l<t{4 lki~ ~ a. lIb I, S£II(,tu_r rur, dV' (l,\'n (Oh(i.i,W~t~ Two versions of the master sc¢t will be created, one for incremental backups and one for full 1f-sl backups and database Ch~~S:\f. he incremental backup script can be ru.n daily (5 days aweek) . and the full backup run ~ however, the schedule can be tailored as sui1EOP needs. The

differences between the two master scripts are listed below. -to

The Incremental Backup Script will perform:

o incremental backups of the ECRMS databases and full text indexes.

The Full Backup Script will perform:

o full backups of the ECRMS databases and full text indexes

o a database integrity check (DBCC) on the ECRMS databases (after the full backups)

o a full backup ofthe FileSurfand Enterprise Vault system databases (i.e.: master, model and msdb). The master, model and msdb backups will be placed into one file for FileSurf system databases and one file for Enterprise Vault system databases.

The master scripts will stop the FileSurf services first, since FileSurf is a client of Enterprise Vault. There is then a delay to give the FileSurf services ample time to shut down, particularly if they are in the middle of processing a very large email. After the delay, the Enterprise Vault services will be stopped. Then the data backups will occur, after which the Enterprise Vault services will be started and finally the FileSurf services will be started.

The master scripts will execute on _under the dsxs-ecrms login. Remote Service Management to make remote calls to _and backup activity across all 3 servers. Regarding the database ~""-''''''''P'''''''''''V

--:;:'---..----

Maintenance Plans for these backups and for the scripts to initiate these plans

uh II ~ l0'ko.:t t\ar~ If elCfl1 lSi1rl

fdl\omcl !

The following diagram depicts the overall flow of the Master Scripts (Incremental and Full).

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGE:

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044387

ECRMS BACKUP PLAN

ECRMS Backup Flow

Stop

FileSurf Exchange Services

Delay x minutes

Backup FileSurf ECRMSDB Database

, J _

, ,

: . cri tOnly: , : PerfO~BC on FileSurf : , ECRMS atabase :

L~~~~~~r------:

1-------- --------1

, ,

: Full Script Only: :

, Backup ,

: FileSurf System Databases :

L :

Backup

(Full Text) Indexes

Stop Enterprise Vault Services

Backup

EV Directory and Vault Databases

,-------*-------

I I

I Full Script Only: :

: Perform DBCC on EV I I Directory and Vault : : Databases '

'------- -------:

Start Enterprise Vault Services

Start

FileSurf Exchange Services

, J ,

, ,

: Full Script Only: :

I Backup ,

: EV System Databases :

I ,

'- - 1

Diagram Notes:

The backups (databases and indexes) are either 'incremental' or 'full', based on the script being run.

The dash lines indicate additional steps in the Full Backup script.

The scripts can accomplish the above in a serial manner, as long as it does not significantly impact the backup time.

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

GEORGE W. BUSH PRESIDENTIAL RECORD

PAGEl

OAP00044388

ECRMS BACKUP PLAN

2.1 Backup Attributes

The following table lists -the attributes of the ECRMS data to be backed up, the source location, the planned frequency of the backups, how the data is backed up, what controls the backup and finally the location of the backup files.

S is deployed and begins to process both daily and legacy PST files, it will become apparent to how much time is required for the incremental backups. If the daily 4-hour maintenan e window can be shortened, -#ti~ yv ill fillo w more time for processing of PST files.

/'. J

-... lIJlII!,e dIQW'C,\

Also, t e 12AM tart of the maintenance windows may change (e.g., shift a couple hours back or

forth) to ommodate the processing of the PST files, as more experience is gained with

the processing of the daily and legacy PST files. The ECRMS maintenance windows must also be coordinated with the IS&T backups, since they cannot overlap. The files resulting from the ECRMS data backups will be written to tape during the IS&T backups.

Also during the ECRMS Maintenance Windows, the Trend ServerProtect daily virus pattern update and the weekly full virus scan of each ECRMS server should be scheduled. This is highly recommended since the Trend ServerProtect daily virus pattern update has been known to cause

ECRMS SQL errors. \

L uJl\<h\~' ~Ov\oJ3

l~\tJ~~~: ~ \\\#

MARCH 27, ZOOS - ECRMS BACKUP AND RESTORE PUUi

GEORGE W. BUSH PRESIDENTIAL RECORD

PAGES

OAP00044390

ECRMS BACKUP PLAN

*** The Centera replication can be continuous or scheduled (e.g., to occur during the ECRMS

- These .bak and .tm files will be backed up to tape as part of the standard IS&T backup procedure.

2.2 Maintenance Windows

The desired maximum duration of the daily ECRMS maintenance window is 4 hours, thus allowing 20 hours for ECRMS operation. The following diagram depicts a 4-hour maintenance window from 12AM to 4AM Tuesdays through Saturdays inclusive and a maintenance window all day Sunday beginning at 12AM. If needed, the Sunday maintenance window can continue until Monday at 4AM.

EOP may opt to perform full backups on a less frequent basis (e.g., every 2 weeks or every month) than the weekly basis depicted below. This would allow more contiguous days with a 4- hr maintenance window and 20-hr operational window.

/

MARCH 27, 2006 - ECRMS BACKUP AND RESTORE PLAN

PAGEl

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044391

ECRMS - BACKUP PLAN

2.3 Backup Considerations

•

Searches and exports cannot occur during the Maintenance Window. If a search or / ~/{4 + export is in progress at the beginning of a Maintenance Window, the search or export will Y'tsta.\.T{£.t ~ P not complete successfully. For lengthy searches or exports, the incremental/full backup ft

scripts should be postponed until the search/export operation completes. ~fsYt

Note that the Enterprise Vault searches can be started in Read-Only mode during a

Maintenance Window, which would allow searches and exports to occur during the

Maintenance Window. However, in order to place the Enterprise Vault services in Read-

Only mode, they must first be stopped. This stopping of the services will disrupt any

search or export in progress.

•

It is highly recommended that a static database backup be performed periodically (e.g., quarterly) on each ECRMS database, where SQL is stopped and the .mdb file backed up. This will capture updates such as service pack (e.g., stored procedures) and user updates. These backups provide a starting place for a restore of the databases.

•

Advanced backup techniques like snapshots can greatly reduce the amount oftime required for backups, thus shortening the Maintenance Windows. This may bea consideration, as the size of the databases and particularly the size of the full text indexes (expected to be 3TB after 3 years of deployment) increase.

•

Additional backups before and after Enterprise Vault configuration changes and upgrades are highly recommended.

•

flo.ust ~

For scalability and resilience each Enterprise Vault "Vault Store"~ h its own SQL database for all the metadata pertaining to the objects managed wit n that specific Vault

Store. As new Vault Stores are created nrak ~nr€l that the backup afe-configured to ~.:W- (£tv-

backup the new database. Note that FileSurf typically interacts with only one Vault 5 tklV\

Store; another Vault Store may be set up for testing. J ctt!~I{ to'rt.-\

i.e :>(cme ... (J, ~ V~ ~i\Vl..

Each Indexing Servic~tore its indexes in multiple locations. Use the Enterprise ~ \d Uf l

Vault Administrator Console to examine the properties of each Indexing Service to .

determine which folders need to be backed up. Note that in the initial deployment of

ECRMS, there is only one Indexing Service and it is currently set up to store its indexes

to only one location ~:\ECRMSIndexes).

MARCH 27, 20(}6 - ECRMS BACKUP AND RESTORE PLAN

PAGE1iJ

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044392

EeRMS - BACKUP PLAN

\~~r:,~~~

3.0 RESTORE PROCEDURE • ftfJ_II)?"'() ~ • g d'.-' f:#JN so;

_ ~U</.- Jif~~O'

The Restore Procedure for ECRMS will be a series o\:anual\tt~ aided by scripts where ~";;' iL possible. ~~

~-br~L"'--'

Of'~~ ~

1. Stop the ECRMS services.

A script will be developed to stop the FileSurf services, then delay to allow their shutdown, then stop the Enterprise Vault services.

2. Restore all ECRMS databases and restore the ECRMS full text indexes. This can be approached in one of two ways:

a) Restore the last full backup, followed by all subsequent incremental backups, for all ECRMS databases and the full text indexes. Apply the necessary transaction logs for all

the ECRMS databases. defil\L ~ <.j(( ( LLJt CARf!._

~

b) Restore the last full backup, followed by all subsequent increme al backups, for all ~ <c-

ECRM.S databases ~d the ~ll = inde~es. Rep:-ocess the PS fi s that were in process ~) at the time of the failure. This option will result tn orphane CLIP on the Centera,

which EMC can remove if desired.

3. Start the ECRMS Services.

A script will be developed to start the Enterprise Vault services and then start the FileSurf

services.

;/ Is'/4is

Additional tools are available from Symantec to aid in the restoration of data, including: il1f!f1t'Yl"lidfl

ManageIndexReplay - rebuild the full text indexes for a specific Enterprise Vault archive I I1Il~! If

from the emailsstoredontheCentera.This tool is included in the EV software release. i net) I\!.t~

During testing, the rate achieved was 8-9 emails per second. Note that beginning with I to (ll'IY"Pl.ik

EV6 SPI, the utility IndexVolumeReplay will be replacing ManageIndexReplay; this I ~~u.

Test I will be performed on the 'production ECRMS servers' and consist of:

• One full and two daily incremental backups of the ECRMS data (databases and full text indexes) using the master scripts, with transaction logs being backed on their regular schedule.

• After the full backup and each of the incremental backups, additional emails will be ~

processed by ECRMS. / ~... .' 1-!

• Complete Checklist 1, recording all results. <checklist to include various counts and ,.\ .£tvY'

search results> ~ {> v.I-

• Restore the full backup, followed by the 2 incremental backups. Apply the subsequent '('vUf\. ,.I

transaction logs. This restore applies to the ECRMS databases and the full text indexes. j(JJAr\.OA+r-

• Complete Checkli.st 1, recording all results and comparing to the results prior. to the .- 1

restore. ~ /J,K1~ .. ~'(~I~. )M-.L ~ ;r-4f-{~ ~~k <~. ~.

'v--"{)(t ~tJ !;(f:1'·:~~,~f·ef' 1l: (It"!: ~8»C ,. a-cr--

Test II will be performed on the 'production ECRMS servers' alliltllnvolves swi chi g from the f!Jts:i

primary Centera to the secondary Centera. The steps include:

• Complete Checklist II <checklist to include opening a number of emails and the export of some emails>

• Pull the network cable from the primary Centera. Enterprise Vault should then automatically redirect its operations to the secondary Centera.

• Complete Checklist II and compare the results to those prior to the switch-over.

OA'.V)P-.OC\O~ \$-\1: $\S ~

ackups of the 'production ECRMS servers' and restores to 3 'test' servers r the duration of this test.

• One and two daily incremental backups of the ECRMS data (databases and full text

indexes) using the master scripts.

• One full and two daily incremental backups of the 3 ECRMS production servers performed by Legato Networker.

• After the full backup and each of the incremental backups, additional emails will be processed by ECRMS.

~ • Complete Checklist 3, recording all results. <checklist to include various counts and

search results and opening of emails >

rY ~ < '

)\"J~ ..

l'IifARCH 27, 20()$ - ECRMS BACKUP AND RESTORE PLAN

PAGE 12

GEORGE W. BUSH PRESIDENTIAL RECORD

OAP00044394

ECRMS - BACKUP PLAN

• To simulate the loss of all 3 ECRMS servers, take them offline. To simulate a problem with all ECRMS equipment at Dakota, also pull the network cable from the primary Centera. The remaining steps are performed on 3 other servers, made available for this test.

• Load the standard EOP server image on the 3 'test' servers.

• Restore the system files, system state and application files from the full and incremental backups performed by Legato Networker.

'. Verify that the following software has been restored. If there are any concerns, then reload this software, as required, on the 3 ECRMS servers.