The board and senior management should establish a testing
program appropriate for the size, complexity, and risk profile of
the organization and its business lines. They should ensure that
the testing program demonstrates the institution's ability to meet
its requirements for continuity of operations. The board and senior
management should establish clear lines of authority and
responsibility for all parties involved with developing,
implementing, and monitoring the continuity testing program. They
should also review and approve the continuity testing program at
least annually, ensure that appropriate follow-up on test results
is performed, and review test results.

Institutions may employ various approaches for ensuring
coordinated and consistent testing across the organization and
support for various quality assurance activities, including
consistent standards for testing and reporting. For example,
many institutions have created a business continuity oversight
function, under the direction of a senior manager, with
accountability and authority for business continuity planning and
testing across the organization. The business continuity
function is supported by a team of liaisons assigned from within
the business lines and support functions. Some institutions
rely on a steering committee, comprised of representatives from
business and support functions, to ensure a coordinated and
consistent approach to business continuity planning and
testing. Regardless of the approach taken, it is the
responsibility of the board and senior management to ensure that
sufficient resources and qualified staff are allocated to the
business continuity testing effort.

Business Line management

Business line management should have ownership and
accountability for testing continuity of business operations,
including applications and processes. While business line
management has overall responsibility for testing their business
processes and related interdependencies, they should coordinate
with the enterprise-wide business continuity plan (BCP) testing
function and support areas, such as IT and facilities
management. Ultimately, business line management should
ensure that its BCP is continually updated based on test results
and changes in business processes.

IT Function

The IT function should have ownership and accountability for
testing recovery of the institution's systems, IT infrastructure,
telecommunications, and the infrastructure of alternative computing
facilities. Moreover, the IT function has custodial
responsibility for business line data and applications. IT
should coordinate with business line management and staff to
establish test environments suitable for business line testing and
should continue to coordinate throughout the testing process.
Additionally, the IT function should, through effective management
of the test schedule, provide sufficient opportunities for the
various business functions to test the operational consistency of
primary and alternate computing facilities. The IT group is
responsible for maintaining the technology test environment,
including controls such as change and configuration management and
information security.

Crisis Management

The board and senior management should ensure that the business
continuity testing program includes the institution's crisis
management capabilities. The testing program should include
exercises to demonstrate that the crisis management program
effectively meets the institution's objectives for responding to a
crisis situation, including identifying and declaring emergencies,
providing a central point for the management of an event, and
coordinating internal and external communications and human
resource issues.

Facilities Management

The facilities management function should have ownership and
accountability for testing the recovery of the institution's
physical plant and equipment, environmental controls, and physical
security. Environmental controls for data centers and the
facilities that house critical business functions should be
included in the institution's continuity testing program.
When data centers or business functions are housed in vendor
facilities, contracts should specify the requirements of the vendor
for testing continuity of those
facilities.

Internal Audit

The internal audit department, or another qualified independent
party, plays an important role in providing an independent review
of the adequacy of the overall business continuity testing
program. The depth and frequency of audit activities and
reporting should be scaled to the criticality of the
operation. While the scope of audit activities and
deliverables may vary, in all cases they must encompass an
independent and objective evaluation of the effectiveness of the
testing program.

As part of the review of the testing process, internal audit
should determine the reasonableness of the underlying assumptions
that were made in developing the test program. The
reasonableness of underlying assumptions, as well as the adequacy
of test plans, scenarios, schedules, and reports, should be
evaluated relative to (1) the size and complexity of the
institution, (2) the criticality of the business line, and (3) the
risk and impact of a possible business disruption. Audit
should observe test exercises to assess the control environment of
alternative locations, verify the results, ensure that proper
reporting and escalation mechanisms are established and utilized,
and ensure that test plans are updated to reflect prior test
results.

Testing Strategy

Enterprise-wide testing strategies should be developed to
properly validate the BCP. Management will achieve greater
confidence in their testing strategies when consideration is given
to the following elements and complexity issues:

Elements

The test strategy should encompass at least three elements:
staffing, technology (data, systems, applications, and
telecommunications), and the facilities that house the staff and
technology environments.

Staffing-Testing strategies should include
demonstrations of the staff's ability to support business
processes, including the processing and settlement of transactions,
communication with key internal and external stakeholders, and
reconciliation of transactions and books of record.
Strategies may need to address the ability of staff to support
increased workloads resulting from the transfer of processing to
alternate sites for extended periods of time. For
institutions that have implemented split processing business
models, any aspects of the client relationship model that present
challenges or complexities to the transfer of workloads across
sites, and related dependencies, should be identified and
incorporated into testing strategies. In addition, testing
strategies should demonstrate the effectiveness of the
institution's management succession plans.

Technology-Testing strategies for technology should
include the data, systems, applications, network, and
telecommunications necessary for supporting business
activities. In the event system recovery is dependent upon
the retrieval of data files, programs, and other items maintained
at the back-up facility; off-site testing procedures should only
include the use of these back-up items to properly replicate the
loss of any master data files and programs maintained at the main
facility. Back-up data files should also be tested frequently
to assess the integrity of the information, to determine if the
data is being saved in the correct format, and to ensure that
applicable files can be retrieved in a timely manner.
Alternatively, institutions may employ other processes for data
replication, such as synchronous and asynchronous data
replication. Regardless of the data replication process used,
the process for demonstrating consistency of data across different
processing environments should be included in the testing
strategy. In addition, strategies should test processes to
recreate any data lost during a switch to alternate processing
facilities, and periodic reviews of telecommunications services
should be conducted to determine circuit diversity.

Facilities-Testing strategies for business functions
should encompass environmental controls, workspace recovery, and
physical security to ensure continuity of facilities and
environmental systems at primary and alternate processing
sites. Testing strategies should include the adequacy of
back-up power generators and heating, ventilation, and air
conditioning systems to meet business recovery objectives at
operating centers. Workspace recovery test strategies should
include assessments of the availability and adequacy of workspace,
desktop computers, network connectivity, e-mail access, telephone
service, and physical security controls. For institutions
relying on the physical relocation of hardware, software, or data
storage devices to recover the technology infrastructure and
applications at alternate locations, the facilities testing
strategy should address the secure transportation of these
items.

Complexity

Organizations should develop testing strategies that demonstrate
their ability to support connectivity, functionality, volume, and
capacity using alternate facilities. The testing strategies
should encompass internal and external dependencies, including
activities outsourced to domestic and offshore business and
technology service providers.For critical
business functions, test strategies and plans may need to extend
beyond network connectivity and include transaction processing to
assess capacity and data integrity.

Test Planning

Crises management Test Plans

Test scenarios, plans, and objectives should include the
institution's crisis management function to demonstrate the
institution's ability to respond effectively to contingency
events. The crisis management program should be tested, with
particular emphasis on the institution's capability to gather
information about the threat or event, initiate the BCP, and
communicate relevant information to the appropriate staff,
customers, vendors, service providers, regulators and other public
authorities. Crisis management test plans should address the
ability of crisis management team members, and their alternates, to
carry out their designated responsibilities under various event
scenarios.

Test Scripts

Test scripts provide sequential procedures related to testing
specific business or technology functions. Test scripts can
be readily used by employees to test business processes within
pre-established timeframes, and test scripts should include
references to production documentation and procedures. Each
test script should clearly document the test objective and
procedures, including:

Detailed information regarding the application, business
processes, system, or facility to be tested;

Sequential test steps to be performed by employees or external
parties;

Prompts for test participants to record quantifiable test
metrics;

Procedures to be followed for manual work-around processes, if
applicable;

A detailed schedule for completion of the test;

Prompts for participants to record issues encountered with the
continuity plan during the test; and

Prompts for participants to record suggestions for improving
continuity plans and associated test methods.

Test scripts may include steps for rotating staff involved in
specific tests to simulate the inaccessibility of key employees
during a disaster.

TESTING EXPECTATIONS FOR CORE CLEARING AND SETTLEMENT
ORGANIZATIONS AND FIRMS THAT PLAY SIGNIFICANT ROLES IN CRITICAL
MARKETS

The guidance provided in this section describes additional
expectations regarding business continuity testing for those
organizations that perform core clearing and settlement activities
in critical financial markets (core firms) and those organizations
that process a significant share of transactions in critical
financial markets (significant firms). These organizations have
been advised by their regulators that they have met the definition
of a core or significant firm as set forth in the "Interagency
Paper on Sound Practices to Strengthen the Resilience of the U.S.
Financial System" (Sound Practices Paper).

Core and significant firms that are subject to the Sound
Practices Paper should develop verification strategies and execute
testing activities to validate the implementation of the
interagency guidelines. The following discussion is not meant
to limit the testing strategies or activities of core and
significant firms and should be read in conjunction with more
comprehensive guidance, available in the public and private
sectors, to evaluate the scope and test the effectiveness of
business continuity plans.

Verification Strategies

In general, core and significant firms should have a
comprehensive, risk-based approach for testing and evaluating the
effectiveness of all of its internal business continuity
arrangements. It would be appropriate to include documented
strategies and plans to determine whether the core or significant
firm has established the facilities and arrangements necessary to
assure substantial achievement of the recovery objectives and other
expectations set forth in the Sound Practices Paper. In this
regard, the Sound Practices Paper advises core and significant
firms to routinely use or test their individual internal recovery
and resumption arrangements for connectivity, functionality, and
volume capacity. It is also suggested that significant firms, which
have back-up sites within the current perimeter of synchronous
back-up technology or that rely primarily on employees from the
same workforce as the primary site, confirm that their plans would
be effective if a wide-scale disruption affects both sites.

Moreover, in light of the dependencies between core firms and
significant firms and the potential impact that a prolonged
disruption of clearance and settlement activities would have on the
operation of the financial system, verification strategies should
include an external component. This external component should
help the agencies and core and significant firms assess whether
there is a consistent level of resilience across critical financial
markets and whether their recovery arrangements are compatible.

Because of their critical role in the operation of financial
markets, the external verification strategies of core firms should
include ample opportunities for significant firms to test their
recovery of critical clearing and settlement activities from their
alternate processing sites. Significant firms are expected to
test with the relevant core firms from their alternate sites and
meet any testing requirements the core firms establish specifically
for significant firms and for participants more generally.
Significant firms should take advantage of these opportunities to
test their ability to meet the recovery time objectives (RTOs) set
forth in the Sound Practices Paper from their geographically
dispersed alternate sites. Core firms and significant firms
also are encouraged to participate in pertinent market-wide and
cross-market tests (such as the "street tests" sponsored by the
Securities Industry Association, Bond Market Association, and
Futures Industry Association) that test connectivity from alternate
sites and include transaction, settlement, and payment processes,
to the extent practical. Verification strategies should
incorporate lessons learned from prior tests and exercises to
improve their effectiveness in validating back-up
strategies.

Testing Scope

Internal testing activities should confirm that each core and
significant firm has identified all clearing and settlement
activities, as well as the systems that support or are integrally
related to the performance of those activities, for each critical
market in which they are core or significant. These activities
should also be designed to demonstrate the core and significant
firm's ability to complete pending material payments and
transactions, access funding, manage material open risk positions,
and make related entries to books and records in the event of a
wide-scale disruption from alternate geographically dispersed data
centers and operations facilities. Moreover, testing
activities should confirm that such critical clearing and
settlement activities could be recovered within RTOs set forth in
the Sound Practices Paper.

As noted earlier, test programs should address external
interdependencies, such as connectivity to markets, payment
systems, clearing agencies, messaging services, and other critical
service providers. Moreover, test programs should validate the
effectiveness of internal and external communication protocols with
stakeholders. Test scenarios should include a wide-scale disruption
in which primary data centers and operations facilities are
rendered inoperable for some period without notice, making it
necessary to recover critical clearing and settlement activities
from an alternate site. Core firms should confirm that resumption
of critical clearing and settlement activities can be sustained at
alternate sites. Core or significant firms that use the same
alternate sites or whose alternate sites rely on the same employees
as their primary sites should assume that employees at primary
sites are unavailable to clear or settle pending transactions for
several days, or are that some employees are unavailable for longer
period of time.

Supervisory Expectations

Examination and supervisory activities will include evaluations
of verification strategies and test plans in order to assess
whether core and significant firms, which are subject to the Sound
Practices Paper, have achieved the resilience necessary to protect
the financial system from a wide-scale disruption.
Verification strategies should be incorporated into implementation
plans and should have an external as well as internal
component. If a core or significant firm finds it necessary
to make incremental changes in its recovery strategies, it should
modify its verification strategies and test plans to incorporate
those changes. Core and significant firms should perform
robust testing to assess the effectiveness of their recovery
strategies. Verification strategies, test plans and test
results should be documented and subject to a qualified,
independent review, such as an internal or external audit.
The agencies will evaluate a core and significant firm's
verification strategies and test plans, the execution of such
strategies and plans, and the test results.