16.1 Understanding Oracle Fusion Middleware Backup and Recovery

An Oracle Fusion Middleware environment can consist of different components and configurations. A typical Oracle Fusion Middleware environment contains an Oracle WebLogic Server domain with Java components, such as Oracle SOA Suite, and a WebLogic Server domain with Identity Management components. It can also include Oracle instances containing system components such as Oracle HTTP Server, Oracle Web Cache, Oracle Internet Directory, and Oracle Virtual Directory.

The installations of an Oracle Fusion Middleware environment are interdependent in that they contain configuration information, applications, and data that are kept in synchronization. For example, when you perform a configuration change, information in configuration files is updated. When you deploy an application, you might deploy it to all Managed Servers in a domain or cluster.

It is, therefore, important to consider your entire Oracle Fusion Middleware environment when performing backup and recovery. You should back up your entire Oracle Fusion Middleware environment at once, then periodically. If a loss occurs, you can restore your environment to a consistent state.

The following topics describe concepts that are important to understanding backup and recovery:

16.1.1 Impact of Administration Server Failure

The failure of an Administration Server does not affect the operation of Managed Servers in the domain but it does prevent you from changing the domain's configuration. If an Administration Server fails because of a hardware or software failure on its host computer, other server instances on the same computer may be similarly affected.

If an Administration Server for a domain becomes unavailable while the server instances it manages—clustered or otherwise—are running, those Managed Servers continue to run. Periodically, these Managed Servers attempt to reconnect to the Administration Server. For clustered Managed Server instances, the load balancing and failover capabilities supported by the domain configuration continue to remain available.

When you first start a Managed Server, it must be able to connect to the Administration Server to retrieve a copy of the configuration. Subsequently, you can start a Managed Server even if the Administration Server is not running. In this case, the Managed Server uses a local copy of the domain's configuration files for its starting configuration and then periodically attempts to connect with the Administration Server. When it does connect, it synchronizes its configuration state with that of the Administration Server.

16.1.2 Managed Server Independence (MSI) Mode

A Managed Server maintains a local copy of the domain configuration. When a Managed Server starts, it contacts its Administration Server to retrieve any changes to the domain configuration that were made since the Managed Server was last shut down. If a Managed Server cannot connect to the Administration Server during startup, it can use its locally cached configuration information—this is the configuration that was current at the time of the Managed Server's most recent shutdown. A Managed Server that starts without contacting its Administration Server to check for configuration updates is running in Managed Server Independence (MSI) mode. By default, MSI mode is enabled. However a Managed Server cannot be started even in MSI mode for the first time if the Administration Server is down due to non-availability of the cached configuration.

16.1.3 Configuration Changes in Managed Servers

Configuration changes are updated in a Managed Server during the following events:

On each Managed Server restart, the latest configuration is retrieved from the Administration Server. This happens even when the Node Manager is down on the node where the Managed Server is running. If the Administration Server is unavailable during the Managed Server restart and if the MSI (Managed Server Independence) mode is enabled in the Managed Server, it starts by reading its local copy of the configuration and synchronizes with the Administration Server when it is available. By default MSI mode is enabled.

Upon activating every administrative change such as configuration changes, deployment or redeployment of applications, and topology changes, the Administration Server pushes the latest configuration to the Managed Server. If the Managed Server is not running, the Administration Server pushes the latest version of the configuration to the Managed Server when it does start.

16.2 Oracle Fusion Middleware Directory Structure

The following shows a simplified view of the Oracle Fusion Middleware directory structure:

16.3 Overview of the Backup Strategies

File copy utilities such as copy, xcopy, tar, or jar. Make sure that the utilities:

Preserve symbolic links

Support long file names

Preserve the permissions and ownership of the files

For example:

On Windows, for online backups, use copy; for offline backups, use copy, xcopy, or jar. Do not use Winzip because it does not work with long filenames or extensions.

Note that for some versions of Windows, any file name with more than 256 characters will fail. You can use the xcopy command with the following switches to work around this issue:

xcopy /s/e "C:\Temp\*.*" "C:\copy"

See the xcopy help for more information about syntax and restrictions.

On Linux and UNIX, for online and offline backups, use tar.

Oracle Recovery Manager (RMAN) to back up database-based metadata repositories.

If you want to retain your backups for a longer duration, you may want to back up to tape, for example using Oracle Secure Backup.

You can also configure Oracle WebLogic Server to make backup copies of the configuration files. This facilitates recovery in cases where configuration changes need to be reversed or in the unlikely case that configuration files become corrupted. When the Administration Server starts, it saves a .jar file named config-booted.jar that contains the configuration files. When you make changes to the configuration files, the old files are saved in the configArchive directory under the domain directory, in a .jar file with a sequentially numbered name such as config-1.jar. However, the configuration archive is always local to the Administration Server host. It is a best practice to back up the archives to an external location.

16.3.1 Types of Backups

You can back up your Oracle Fusion Middleware environment offline or online:

An offline backup means that you must shut down the environment before backing up the files. When you perform an offline backup, the Administration Server, all Managed Servers in the domain, and all system components in the Oracle instances should be shut down.

Back up the environment offline immediately after installation and after applying any patches or upgrades.

An online backup means that you do not shut down the environment before backing up the files. To avoid an inconsistent backup, do not make any configuration changes until the backup is completed. To ensure that no changes are made in the WebLogic Server domain, lock the WebLogic Server configuration, as described in Section 3.4.2.

You can perform backups on your full Oracle Fusion Middleware environment, or on the run-time artifacts, which are those files that change frequently.

To perform a full backup, you should back up the static files and directories, as well as run-time artifacts.

Static files and directories are those that do not change frequently. These include:

The Middleware home (MW_HOME). A Middleware home consists of a WebLogic Server home (containing the Oracle WebLogic Server product directories), an Oracle Common home, and optionally an Oracle home. It can also contain the user_projects directories, which contains Oracle WebLogic Server domains and Oracle instance homes, which are not static files.

OraInventory

OraInst.loc and oratab files, which are located in the following directory:

(Linux and IBM AIX) /etc
(Other UNIX systems) /var/opt/oracle

The beahomelist file, which is located at:

(UNIX) user_home/bea/beahomelist
(Windows) C:\bea\beahomelist

On Windows, the following registry key:

HKEY_LOCAL_MACHINE\Software\oracle

In addition, for system components, such as Oracle Web Cache, you must back up the following Windows Registry key:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services

Run-time artifacts are those files that change frequently. Back up these files when you perform a full backup and on a regular basis. Run-time artifacts include:

Domain directories of the Administration Server and the Managed Servers (by default, a domain directory resides in MW_HOME, but it can be configured by the user to point to a different location.)

In most cases, you do not need to back up Managed Server directories separately because the Administration Server contains information about all of the Managed Servers in its domain.

All Oracle instance homes, which reside, by default, in the MW_HOME but can be configured to be in a different location.

Application artifacts, such as .ear or .war files that reside outside of the domain.

You do not need to back up application artifacts in a Managed Server directory structure because they can be retrieved from the Administration Server during Managed Server startup.

Database artifacts, such as the MDS Repository.

Any database-based metadata repositories used by Oracle Fusion Middleware. You use Oracle Recovery Manager (RMAN) to back up an Oracle database.

Persistent stores, such as JMS Providers and transaction logs, which reside by default in the user_projects directory, but can be configured in a different location. However, note the limitation described in Section 17.2.

16.3.2 Recommended Backup Strategy

This section outlines the recommended strategy for performing backups. Using this strategy ensures that you can perform the recovery procedures in this book.

Perform a full offline backup: This involves backing up the entities described in Section 16.3.1. Perform a full offline backup at the following times:

Perform an online backup of run-time artifacts: This involves backing up the run-time artifacts described in Section 16.3.1. Backing up the run-time artifacts enables you to restore your environment to a consistent state as of the time of your most recent configuration and metadata backup. To avoid an inconsistent backup, do not make any configuration changes until backup completes. Perform an online backup of run-time artifacts at the following times:

On a regular basis. Oracle recommends that you back up run-time artifacts nightly.

Prior to making configuration changes to a component or cluster.

After making configuration changes to a component or cluster.

Prior to deploying a custom Java EE application to a Managed Server or cluster.

After a major change to the deployment architecture, such as creating servers or clusters.

Perform an offline backup of static files and directories: This involves backing up the static files and directories described in Section 16.3.1. Perform an offline backup of static files and directories at the following times:

16.4 Overview of Recovery Strategies

Recovery strategies enable you to recover from critical failures that involve actual data loss. Depending on the type of loss, they can involve recovering any combination of the following types of files:

16.4.1 Types of Recovery

You can recover your Oracle Fusion Middleware environment in part or in full. You can recover the following:

The Middleware home

A domain

The WebLogic Server Administration Server

A Managed Server

An Oracle home

An Oracle instance home

A component, such as Oracle SOA Suite or Oracle Web Cache

A cluster

Deployed applications

16.4.2 Recommended Recovery Strategies

Note the following key points about recovery:

Your Oracle Fusion Middleware environment must be offline while you are performing recovery.

Rename important existing files and directories before you begin restoring the files from backup so that you do not unintentionally override necessary files.

Although, in some cases, it may appear that only one or two files are lost or corrupted, you should restore the directory structure for the entire element, such as an Oracle instance home or a domain, rather than just restoring one or two files. In this way, you are more likely to guarantee a successful recovery.

Recover the database to the most current state, using point-in-time recovery (if the database is configured in Archive Log Mode). This is typically a time right before the database failure occurred.

These topics include information about configuration files for particular components. Note that the list of files in not an exhaustive list. You do not back up or recover the individual files. Generally, you back up or recover a Middleware home, the domain, Managed Server, Oracle home, or Oracle instance.

For the steps you take to back up your environment, see Section 17.3. For the steps you take to recover a component, see Chapter 18.

16.5.1 Backup and Recovery Recommendations for Oracle WebLogic Server

The following sections describe backup and recovery recommendations for Oracle WebLogic Server:

This section describes the Oracle WebLogic Server data that must be backed up and restored.

Configuration Files

Configuration files and applications are stored in the domain home.

Database Repository Dependencies

Oracle WebLogic Server does not, by default, depend on any database repository. However, applications deployed on Oracle WebLogic Server may use databases as data sources. To back up a database, see the Oracle Database Backup and Recovery User's Guide, which is available at:

If you use Whole Server Migration, the leasing information is stored in a table in a database. If you recover Oracle WebLogic Server, you should discard the information in the leasing table. (For more information about Whole Server Migration, see "Whole Server Migration" in Oracle Fusion Middleware Using Clusters for Oracle WebLogic Server.)

This section describes the Oracle WebLogic Server JMS data that must be backed up and restored.

Configuration Files

DOMAIN_HOME/config/jms

If a JMS uses file-system accessible stores, the default file-system store is either in a user-configured location that is specified in config.xml, or in the following location:

DOMAIN_HOME/servers/server_name/data/store/default

Database Repository Dependencies

If a JMS uses a JDBC accessible store, back up the database.

Backup Recommendations

Back up the domain and the JMS file persistent store if it is not located within the domain.

Back up the schema in the database if JDBC-based persistent store is configured. Note the following:

All JMS data should be backed up using an offline backup.

Always try to keep JMS data as current as possible. This can be achieved by using the point-in-time recovery capabilities of Oracle Database (in the case of database-based persistence) or using a highly available RAID-backed storage device (for example, SAN/NAS).

It is currently not possible to take consistent backup of persistent stores for a system that uses JMS and transaction logs. This is because the transaction logs can only be file-based and the JMS can be either file-based or it can reside in the database. For highest reliability, use a highly available fault-tolerant storage (for example, SAN) for JMS and transaction log file stores.

If, for whatever reason, you need to restore JMS data to a previous point in time, there are potential implications. Restoring the system state to a previous point in time not only can cause duplicate messages, but can also cause lost messages. The lost messages are messages that were enqueued before or after the system restore point time, but never processed. If the persistent store is a custom store that is dedicated to JMS use, then you can delete the entire store.

Use the following procedure before recovery to drain messages in the JMS queue after persistent-store recovery to avoid processing duplicate messages:

Note:

Do not drain and discard messages without first being certain that the messages contain no data that must be preserved. The recovered messages may include unprocessed messages with important application data, in addition to duplicate messages that have already been processed.

Log into the Oracle WebLogic Server Administration Console.

Before recovery, configure JMS server to pause Production, Insertion, and consumption operations at boot time to ensure that no new messages are produced or inserted into the destination or consumed from the destination before you drain stale messages. To do this:

Expand Services, then Messaging, and then click JMS Servers.

On the Summary of JMS Servers page, click the JMS server you want to configure for message pausing.

On the Configuration: General page, click Advanced to define the message pausing options. Select Insertion Paused At Startup,Production Paused At Startup, and Consumption Paused At Startup.

Click Save.

Use the following procedure after recovery:

After recovering the persistent store, start the Managed Servers.

Drain the stale messages from JMS destinations, by taking the following steps:

Expand Services, then Messaging, and then JMS Modules.

Select a JMS module, then select a target.

Select Monitoring, then Show Messages.

Click Delete All.

Resume operations, by taking the following steps:

Expand Services, then Messaging, and then JMS Servers.

On the Summary of JMS Servers page, click the JMS server you want to configure for message pausing.

On the Configuration: General page, click Advanced. Deselect Insertion Paused At Startup,Production Paused At Startup, and Consumption Paused At Startup.

Click Save.

If the store is not dedicated to JMS use, use the Oracle WebLogic Server JMS message management administrative tool. This tool can perform import, export, move, and delete operations from the Administration Console, MBeans, and WLST.

For applications that use publish and subscribe in addition to queuing, you should manipulate topic subscriptions in addition to queues.

Recovery Recommendations

Recover the domain.

If the JMS persistent store is file-based, recover it from backup. If the JMS persistent store is database-based, recover the database to the most recent point in time, if needed.

This section describes the Oracle Directory Services Manager data that must be backed up and restored.

Configuration Files

Oracle Directory Services Manager, which is the graphical user interface for Oracle Internet Directory and Oracle Virtual Directory, does not have configuration files, but keeps track of host and port information of Oracle Internet Directory and Oracle Virtual Directory in serverlist.txt, which is part of the application .ear file:

This section describes the Oracle Access Manager data that must be backed up and restored.

Configuration Files

DOMAIN_HOME/config/fmwconfig/oam-config.xml

Database Repository Dependencies

The schema used by OES for the Oracle Access Manager policy store.

Backup Recommendations

Back up the Middleware home and the domain home for the Oracle Access Manager server. Back up the Oracle home and the Oracle instance for the Oracle HTTP Server that contains the Webgate, and the database containing the schema used by OES for the Oracle Access Manager policy store.

Recovery Recommendations

Recover the Middleware home and the domain home for the Oracle Access Manager server. Recover the Oracle home and the Oracle instance for the Oracle HTTP Server that contains the Webgate, as needed.

This section describes the Oracle BPEL Process Manager data that must be backed up and restored.

Configuration Files

Configuration files are stored in the database.

Database Repository Dependencies

Process definition and configuration files are stored in the MDS schema. The dehydration store is stored in the BPEL schema.

Backup Recommendations

Back up the Administration Server domain directories. Back up the database after any configuration changes, including changes to global fault policies, callback classes for workflows and resource bundles that can potentially be outside the suitcase. Also back up the database after deploying a new composite or redeploying a composite.

Recovery Recommendations

Recover the database to the most recent point in time, if needed. Point-in-time recovery ensures that the latest process definitions and in-flight instances are restored. However, this may result in reexecution of the process steps. Oracle recommends that you strive for idempotent Oracle BPEL Process Manager processes. If the system contains processes that are not idempotent, you must clean them up from the dehydration store before starting Oracle Fusion Middleware. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite for more information.

Because instances obtain the process definition and artifacts entirely from the database, there is no configuration recovery needed after the database is recovered to the most current state; instances should continue to function correctly.

For redeployed composites, a database recovery ensures consistency between the dehydrated in-flight processes and their corresponding definition since the process definition is stored in database repository where dehydrated instances are also stored.

16.5.3.7Backup and Recovery Recommendations for Oracle Business Process Management

For Oracle Business Process Management, you back up and restore the same data as Oracle BPEL Process Manager, as described in Section 16.5.3.1. This section describes data specific to Oracle Business Process Management.

Process definition and configuration files are stored in the MDS schema.

Backup Recommendations

In addition to the recommendations for Oracle BPEL Process Manager, described in Section 16.5.3.1, you must back up the Oracle homes, including all Oracle homes in a cluster. When you extend a SOA domain to Oracle Business Process Management and configure Oracle Business Process Management, the process adds files to the Oracle Business Process Management Oracle home. However, it does not copy the files to any other Oracle homes in the cluster. After you configured Oracle Business Process Management, you should have copied the files to the other Oracle homes in the cluster. As a result, you must back up all Oracle homes in the cluster.

Recovery Recommendations

In addition to the recommendations for Oracle BPEL Process Manager, described in Section 16.5.3.1, you must recover all of the Oracle homes in the cluster.

For the steps to recover Oracle Business Process Management, see Section 18.2.7.7.

16.5.4 Backup and Recovery Recommendations for Oracle WebCenter

The following sections describe backup and recovery recommendations for Oracle WebCenter:

This section describes the Oracle Web Services Manager data that must be backed up and restored.

Configuration Files

DOMAIN_HOME/config/fmwconfig/policy-accessor-config.xml

Database Repository Dependencies

If a database-based MDS Repository is used, Oracle Web Services Manager uses a partition in the MDS schema.

Backup Recommendations

Back up the Oracle Web Services Manager domain.

If Oracle Web Services Manager uses a file-based MDS Repository, back it up using a file copy mechanism. If it uses a database-based MDS Repository, back up the database using RMAN.

Recovery Recommendations

Restore the Oracle Web Services Manager Managed Server.

If Oracle Web Services Manager uses a file-based MDS Repository, restore it from the backup. If it uses a database-based MDS Repository, recover the database to the most recent point in time, if needed.

Back up the domain, the Oracle home, and database containing the OCS schema. If the Vault and WebLayout directories are not located in the domain directory, back up their directories, which are specified in:

DOMAIN_HOME/ucm/CONTEXT-ROOT/config/config.cfg

Also, back up the following directory, which is located in a shared file system:

DOMAIN_HOME/ucm/CONTEXT-ROOT/config

Recovery Recommendations

Restore the domain and the shared file system containing the Vault and WebLayout directories, depending on the severity of the failure.

Recover the database containing the OCS schema to the most recent point in time, if needed.

Because Oracle Universal Records Management depends on Oracle Universal Content Management and has no additional backup and recovery artifacts, see the backup and recovery recommendations for Oracle Universal Content Management in Section 16.5.10.3.

16.6 Assumptions and Restrictions

The following assumptions and restrictions apply to the backup and recovery procedures in this book. Also see the restrictions listed in Section 17.2.

File systems and files can only be restored as of the last good backup. There is no support for roll-forward recovery to the current point in time.

All of the files required for recovery are maintained within the Middleware home, Oracle instance home, and Oracle Inventory (for loss of host use cases) directories. Generally, if no administration changes, such as configuration changes, deployments, redeployments, or patching, have been done during the backup, it is always safe to restore the file system pertaining to a particular component to a previous point in time using the last good backup. Thus, new backups must always performed after any administration changes.

Only the user who installs the product or a user who has access privileges to the directories where Oracle Fusion Middleware has been installed should be able to execute backup and recovery operations.

If a single Managed Server and Administration Server run on different hosts and the Managed Server is not in a cluster, you must use the pack and unpack commands on the Managed Server to retrieve the correct configuration.

If multiple Managed Servers run on different hosts (not in a cluster), the domain should be configured to use an external LDAP for policy store instead of using file-based policy store.

If the Administration Server is on a different host than the Managed Servers, the domain should be configured to use an external LDAP for the policy store instead of using a file-base policy store.