Agenda Ø How do you backup/archive courses? l What policies and procedures guide your response to requests to recover a course, a file, an internal ANGEL page, a student upload file? Ø How do you protect your system from various failures, and in what time do you “promise” to have it back online?

Constraints (what we can’t change) Ø Support coverage is not 24 x 7: Central IT (IST) provides extended support for critical systems but not 24 x 7 support. Ø Cannot survive lengthy power outages. Ø Cannot survive some network outages. l Network support is also not 24 x 7.

Backup Processes Ø System data backup l Database (dump of db file), Transaction logs (cut once per day) and Upload files backed up nightly by campus backup service. Ø Course archives l l Long term: Archive courses at end of term. Shorter term: Remove from system after 4 terms. (Note: to offer a course again, copy course rather than reuse same instance).

Recovery Process Ø Recover data to dev system and copy lost data to production. l This can be very complex if the missing data is a quiz that was run, a bulletin board, etc. ! Ø Currently no policies on what to recover, or promise of time to recovery. Requests considered on individual basis.

Vulnerabilities in Current Strategy Ø The ANGEL/IIS or SQL Server hardware, e. g. , system motherboard failure l Don’t have ready back-up machine. • Could temporarily use development system. l Ø Likely a minimum half day down-time. Machine room “fire” l l l All hardware lost. Up to one day of lost data (if 24 hours from last backup). Days of down time!

Providence College (Adam) Ø Like Waterloo, ANGEL has been our LMS since Fall, 2001. Ø Support coverage is not 24 x 7. Ø Cannot survive lengthy power outages or network outages.

PC Backup and Recovery Ø System data backup l l l Ø Course archives l l Ø Back up database and logs to files once per day. Use Tivoli to back up both DB and file system nightly. Creates “backup of a backup. ” Short term: Archive courses 90 days after term end. Long term: Store archives to DVD. Recovery l Like Waterloo, recover Production database in Development environment.

PC’s Future Architecture Ø This Summer: Production System New Server and SAN IBM Storage Area Network Development System ANGEL IIS/SQL (New HP) ANGEL IIS/SQL (Old HP) • Purchase new server and install O/S and SQL Server on local RAID. • Store database and web files on SAN disk. • In the event of Production hardware failure, connect Production disk to Development server with little downtime.

Kelley Direct On-Line Programs, Indiana University Ø Current Data Protection Measures l Backup System Backups • • Full Backups once a week starting Friday night Differential Backups every night around 11 PM Database Backups • Full ANGEL SQL database backup every night at 10 PM. The database backup output files are then backed up by system tape backups for that night. • Transaction log backups every six hours. The backup tapes are then taken to an offsite location.

Kelley Direct On-Line Programs, Indiana University Ø Current Recovery Practices File or Database Restore l • Restore from disk, tape backups, or individual developer’s machines. System Component Failure l • • Replace the faulty component(s) from the spare chassis (test server) or move entire disk array to from production to test server Total System Failure or disk array failure Rebuilt entire system, possibly to alternate hardware. All the ANGEL components will either need to be installed from scratch, or restored from backup tapes. Some system components have to be reconfigured manually.

Kelley Direct On-Line Programs, Indiana University Ø Challenges for KD ANGEL Environment Storage Capacity l • Limited expansion capability Recoverability Single copy of production data on disk. Tape restoration is time consuming and means data loss Availability l • No redundancy built in. Single server design. Any component failure means downtime Growth l • Significant enrollment growth is expected for the programs in the next three years Development Environment l • Developers are coding on own machines. Configurations differ from production environment. Less efficient.

Kelley Direct On-Line Programs, Indiana University Ø Some Questions l l l How can backend infrastructure better support the vision of the on-line programs? How to plan system capacity when progarm changes (such as enrollment growth)? How to better protect student data? What the available options for long-term data retention? How to better meet the requirements for less service interruption? What should we do to ensure a faster ANGEL systems recovery?