For information about configuring high availability for Oracle Fusion Middleware products such as Oracle WebLogic Server, Oracle SOA Suite, Oracle Identity Management, and Oracle Business Intelligence, see the Oracle Fusion Middleware High Availability Guide.

Providing high availability for Oracle Fusion Applications involves configuring an Oracle WebLogic cluster for high availability of the middle tiers and configuring Oracle Real Applications Clusters (Oracle RAC) for high availability of the Oracle Database. It also involves scaling out Oracle Fusion Applications and Oracle RAC database instances.

Oracle Fusion Applications is built on standards-based Oracle Fusion Middleware. Therefore, it benefits from the high availability solutions for Oracle Fusion Middleware.

Characteristics of an Oracle Fusion Applications high availability configuration are:

This chapter assumes that you have a basic understanding of high availability concepts such as active-active deployments, active-passive deployments, and disaster recovery deployments. For information about these concepts, see the "Oracle Fusion Middleware High Availability Framework" chapter in the Oracle Fusion Middleware High Availability Guide.

See the following documentation resources to learn more about high availability for Oracle Fusion Middleware and Oracle Database:

Oracle Fusion Applications runs on Oracle WebLogic Server. During an Oracle Fusion Applications installation, each individual product family installs into its own WebLogic Server domain. At the time of provisioning, a WebLogic cluster (with one Oracle WebLogic Server) is created for each Java EE application that is part of the product family. If you want to scale out any of the Java EE applications, new Managed Servers can be added to the application's WebLogic cluster.

16.3 Oracle Fusion Applications High Availability

This section provides a description of the single instance architecture of Oracle Fusion Applications, and then provides information that helps you deploy Oracle Fusion Applications in a high availability configuration.

If you have not already done so, read Chapter 1 for an introduction to:

When you configure Oracle Identity Manager for Fusion Applications in a high availability environment, Oracle recommends that you use Multimaster Replication. See Setting up Multimaster Replication.

16.3.1.2 Oracle Fusion Applications Run-Time Processes

All Oracle Fusion Applications components run as JavaEE applications in WebLogic Server and do not start a process of their own. Standard WebLogic tools such as the Administration Console, Oracle WebLogic Scripting Tool (WLST), and Oracle Enterprise Manager Fusion Middleware Control start, stop, and manage the applications.

16.3.1.3 Oracle Fusion Applications Request Flow

The typical client HTTP request goes from the web browser or Web Services client to Oracle HTTP Server to JDBC/RMI/WS (HTTP). Table 16-1 shows the Oracle Fusion Applications clients, the protocols they use to connect, and their high availability configuration.

16.3.1.4 Oracle Fusion Applications Configuration Artifacts

All configuration for Oracle Fusion Applications is stored in the Oracle Fusion Applications repository database or MDS repository.

Configuration for Oracle WebLogic is stored in the domain configuration.

16.3.1.5 Oracle Fusion Applications Deployment Artifacts

The Oracle Fusion applications are deployed using the nostaged deployment model. This allows you to install a single copy of the Oracle Fusion Applications binaries on a NAS or SAN shared storage and then run Fusion Applications from multiple computers using that single copy on the shared storage. Typically, application extensions get deployed on a different Managed Server than the application itself.

16.3.2.1 Starting and Stopping the Oracle Fusion Applications Cluster

All Oracle Fusion Applications configuration changes can be done at the cluster level. You do not need to make configuration changes to individual instances in a cluster.

16.3.2.3 Oracle Fusion Applications Failures and Expected Behaviors

This section describes the different types of failures that can occur in an Oracle Fusion Applications high availability deployment, and the expected behaviors when these failures occur. This section contains the following topics:

16.3.2.3.1 Process Failure

Node Manager detects Managed Server failures and restarts the Managed Server automatically. If repeated restart attempts fail, then in case of a simple active-active deployment, surviving cluster members continue to service the request. The ORACLE_INSTANCE/config/OHS/ohsName/mod_wl_ohs.conf file for Oracle HTTP Server redirects user requests to a server which has the replica of the session state. The Node Manager RestartInterval parameter specifies the amount of time Node Manager attempts to start a failed server. The RestartMax parameter specifies the number of times Node Manager attempts to start a failed server. See the Node Manager Properties table in the Oracle Fusion Middleware Node Manager Administrator's Guide for Oracle WebLogic Server for more information about these parameters.

The server promotes the state to primary, creates a replica elsewhere in the cluster, and services the user request.

If this message appears, retry the ongoing session after logging in again (if required).

Web Services Provider Failure. If the failure occurs before the Web Services stack is invoked, the application receives an error and can retry. If failure occurs after the acknowledgement is received, then the Asynchronous Web Services infrastructure guarantees a successful response.

16.3.2.3.4 Troubleshooting Oracle Fusion Applications

If you experience issues with your Oracle Fusion Applications high availability deployment, check the WebLogic Server log. See Section 16.3.1.6 for details.

Oracle Real Application Clusters (Oracle RAC) enables you to cluster servers to provide resilient and scalable access to a database. A cluster comprises multiple interconnected computers or servers that appear as if they are one server to end users and applications. Oracle RAC simultaneously provides a highly available and scalable database for Oracle Fusion Applications.

This section provides an overview of Oracle RAC and information about setting up a highly available deployment for Oracle Fusion Applications configured with an Oracle RAC database as the persistent store for data.

Note:

This chapter discusses configuring Oracle RAC databases only. Other persistent repositories that connect with Oracle Fusion Applications, such as Oracle Hyperion, are configured implicitly and transparently.

16.4.1 Oracle Real Application Clusters

Oracle RAC and Oracle ClusterwareFoot 1 allow the Oracle Database to run any packaged or custom application across a set of clustered servers. This capability provides the highest levels of availability and the most flexible scalability. If a clustered server fails, Oracle Database continues running on the surviving servers. When more processing power is needed, you can add another server without interrupting access to data.

Oracle RAC enables multiple database instances that are linked by an interconnect to share access to an Oracle database. In an Oracle RAC environment, the Oracle Database runs on two or more systems (nodes) in a cluster while concurrently accessing a single shared database. The result is a single database system that spans multiple hardware systems and enables Oracle RAC to provide high availability and redundancy during failures in the cluster. Oracle RAC accommodates all system types, from read-only data warehouse (DSS) systems to update-intensive online transaction processing (OLTP) systems.

The clusters that are typical of Oracle RAC environments can provide continuous service for both planned and unplanned outages. Oracle RAC builds higher levels of availability on top of the standard Oracle features. All single-instance high availability features, such as the Oracle Flashback technologies and online reorganization, also apply to Oracle RAC. Applications scale in an Oracle RAC environment to meet increasing data processing demands without changing the application code. In addition, allowing maintenance operations to occur on a subset of components in the cluster while the application continues to run on the rest of the cluster can reduce planned downtime.

For information about Oracle RAC design and deployment techniques, see the "Design and Deployment Techniques" chapter of the Oracle Real Application Clusters Administration and Deployment Guide.

Providing high availability requires detailed planning to ensure there are no single points of failure throughout the infrastructure. Even though Oracle RAC makes your database highly available, if a critical application becomes unavailable, then your business can be negatively affected.

For example, because Oracle Fusion Applications uses the Lightweight Directory Access Protocol (LDAP) for authentication, the best practice is to make the LDAP server highly available. If the database is up but the LDAP server is down, users cannot connect to the applications and the entire system appears to be down.

The following sections describe deploying Oracle RAC for high availability in an Oracle Fusion Applications environment:

A data source is a Java object that application components use to obtain connections to a relational database. Specific connection information, such as the URL or user name and password, are set on a data source object as properties and do not need to be explicitly defined in an application's code. This abstraction allows applications to be built in a portable manner, because the application is not tied to a specific back-end database. The database can change without affecting the application code.

A multi data source is like a pool of data sources that provides failover processing and recovery. It also provides load balancing between nodes of a highly available database system, such as an Oracle RAC database. When an Oracle RAC instance fails, a multi data source determines which data source to use to satisfy application requests. If you are using an Oracle RAC database, you must configure multi data sources.

The data source member list for a multi data source supports dynamic updates. This allows Oracle RAC environments to add and remove database nodes and corresponding data sources without redeployment, and it provides the ability to:

Grow (scale) and shrink Oracle RAC clusters in response to throughput

Shut down Oracle RAC nodes temporarily to perform planned maintenance

See Section 16.4.3 for best practices when configuring multi data sources for Oracle RAC and Oracle Fusion Applications. For more information about using multi data sources with Oracle RAC, see the "Using Multi Data Sources with Oracle RAC" section in the Oracle Fusion Middleware High Availability Guide.

WebLogic Server periodically checks the status of data sources in a multi data source. If an Oracle RAC node or instance fails, application connection requests are managed as follows:

Existing connections

There is no failover of existing connections. In-flight transactions are typically rolled back when the database is the transaction manager. When the WebLogic Server is the Transaction Manager, in-flight transactions are failed over; they are driven to completion or rolled back based on the transaction state at the time of failure.

New connection requests

New session requests are redirected to a working Oracle RAC instance in the cluster, either by Oracle WebLogic Server or by the Oracle Thin driver.

16.4.2.4 Load Balancing Across Oracle RAC Nodes

Load balancing involves distributing requests among two or more processes. Oracle WebLogic Server provides a load balancing algorithm for multi data sources. If an application requires load balancing across Oracle RAC nodes, then WebLogic Server supports this capability through the use of JDBC multi data sources that have been configured for load balancing.

WebLogic accesses the data sources that form a multi data source using a round-robin scheme. When switching connections, WebLogic Server selects a connection from the next data source in the order listed.

See Section 16.4.3 for recommendations when configuring load balancing with WebLogic Server.

16.4.2.5 Retry Logic to Protect Read-Only Operations

Under certain circumstances, an Oracle RAC database outage generates the following message:

If this message appears, retry the ongoing session after logging in again (if required).

See Section 16.4.3 to configure retry properties with WebLogic Server.

16.4.3 Best Practices for Deploying JDBC Multi Data Sources on Servers and Clusters

When you create an Oracle Fusion Applications environment using an Oracle RAC database, the provisioning tool configures automatically a multi data source to use each Oracle RAC database instance.

The default configuration uses these best practices that provide optimal availability:

Deploys a multi data source to a cluster or server by selecting the server or cluster as a deployment target in the WebLogic Server Administration Console.

When a multi data source is deployed on a server, WebLogic Server creates an instance of the multi data source on the server. When you deploy a multi data source to a cluster, WebLogic Server creates an instance of the multi data source on each server in the cluster.

Deploys all data sources that are used by a multi data source to satisfy connection requests on the same servers and clusters as the multi data source.

Multi data sources do not route connection requests to other servers in a cluster or in a domain.

16.5 Scaling Out Oracle Fusion Applications

Oracle Fusion Applications is always deployed in a cluster. Even in a single instance deployment, the Oracle Fusion Applications instance is part of a cluster with one member.

You can use a scale out operation to move from a non-high availability deployment to a high availability deployment. You can scale up or scale out the Oracle Fusion Applications topology. When you scale up the topology, you add a new Managed Server to computers that are already running one or more Managed Servers. When you scale out the topology, you add a new Managed Server to new computers.

Before you run scale up or scale out steps, check that you meet these requirements:

There must be at least one existing computer running at least one Managed Server configured with Oracle Fusion Applications within the topology. The computer contains an APPLICATIONS_BASE home directory (/oracle in Figure 1-6) and an Oracle Fusion Applications Middleware home directory (/oracle/fusionapps in Figure 1-6) in shared storage.

The computer on which the new Managed Server is deployed can access the existing home directories for Oracle Fusion Applications. Use the existing installations in shared storage for creating a new Managed Server.

For scale up, you do not need to install Oracle Fusion Applications binaries in a new location, and you do not need to run pack and unpack to bootstrap the domain configuration.

For scale out, you must run pack and unpack to bootstrap the domain configuration in the new computer.

Note:

The steps below assume that Oracle Fusion Applications clusters such as the Sales cluster do not use WebLogic JMS or XA.

The scale up steps for Oracle Fusion Middleware components such as Oracle SOA Suite and Oracle WebCenter are in the scale up section of the component's chapter in the Oracle Fusion Middleware High Availability Guide. Scale out steps for Oracle Fusion Middleware components such as Oracle SOA and Oracle WebCenter are in the scale out section of the component's chapter in the Oracle Fusion Middleware High Availability Guide. To make Oracle HTTP Server highly available, see the Oracle HTTP Server chapter in the Oracle Fusion Middleware High Availability Guide.

In this case, you have a computer that runs a Managed Server configured with Oracle Fusion Applications components and you want to add another Managed Server to that computer. Use the steps in this section to add a second Managed Server to a computer and to add subsequent Managed Servers on that computer.

Follow these steps to scale up the topology. This example uses the Oracle Fusion Customer Relationship Management application; you can also use these steps to scale up other Oracle Fusion Applications clusters:

Using the Administration Console, clone the source Managed Server (for example, SalesServer1) into a new Managed Server. The source Managed Server to clone should be one that already exists on the computer where you want to run the new Managed Server.

Select the Managed Server that you want to clone (for example, SalesServer1).

Select Clone.

Name the new Managed Server server_nameN (for example, SalesServer2), where N is a number to identify the new Managed Server.

For the listen address, assign the host name to use for this new Managed Server.

Ensure that the port number (9001, in this example, so as not to conflict with the current port 8001 that the existing Managed Server, SalesServer1, uses) for the new Managed Server is not used on this computer.

Update the cluster address to include the new server:

Select Environment -> Cluster from the Administration Console.

Click on the Sales_Cluster server.

In the Change Center, click Lock & Edit.

Add the new server's address and port to the Cluster Address field. For example:

SALESHOST1:8001,SALESHOST1:9001

Disable host name verification for the new Managed Server. Before starting and verifying the SalesServern Managed Server, you must disable host name verification. You can re-enable it after you configure server certificates for the communication between the Oracle WebLogic Administration Server and the Node Manager in SALESHOSTn. If the source server from which the new one has been cloned had already disabled hostname verification, these steps are not required; the hostname verification setting propagates to the cloned server.

To disable host name verification:

Start the Oracle WebLogic Administration Console.

Expand the Environment node in the Domain Structure window.

Click Servers.

Select SalesServern in the Names column of the table in the Summary of Servers page.

Click the SSL tab in the Settings page for the server.

Click Advanced.

Set Hostname Verification to None.

Click Save.

Change JDBC LLR Table Name:

In the Oracle EM, expand the Environment node in the Domain Structure window

Click Servers.

Select the scaled out server.

In the General tab, click Advanced.

Change the JDBC LLR Table Name to the next higher number so that it doesn't conflict with the number of the original server.

Click Save.

Ensure that the Node Manager is running on the computer. To start the Node Manager, use the installation in shared storage from the existing computer and start Node Manager by passing the host name of the computer as a parameter as follows:

SALESHOST1> WL_HOME/server/bin/startNodeManager computer_ip

Start and test the new Managed Server from the Oracle WebLogic Server Administration Console:

Ensure that the newly created Managed Server (SalesServer2 in this example) is running.

Access the application on the newly created Managed Server (http://SALESHOST1:9001/sales/faces/index). The application should be functional.

Configure Oracle HTTP Server to route to the Administration Server that contains the new Managed Server; set the WebLogicCluster parameter to the list of computers in the cluster. Follow these steps for all Oracle HTTP Server hosts in your deployment:

16.5.1.1 Testing the Routing from Oracle HTTP Server to the New Managed Server

To test the routing from Oracle HTTP Server to the new Managed Server.

If desired, shut down the existing Managed Server (in this case, SalesServer1).

Access the application using WEBHOST1:7777/sales/faces/index through the web server, which should access the application on the new Managed Server (in this case, SalesServer2).

This step should show that Oracle HTTP Server routed the request to the new Managed Server (SalesServer2) while SalesServer1 was down, and that SalesServern serviced the request.

If you have not already done so, restart the Managed Server that you shut down.

16.5.2 Scaling Out the Topology (Adding Managed Servers to New Machines)

When you scale out the topology, you add new Managed Servers configured with Oracle Fusion Applications to new computers where no Oracle Fusion Applications Managed Servers are configured.

You scale out the topology using Enterprise Manager Cloud Control. Benefits of Cloud Control include:

Ease of use. Cloud Control automatically completes the provisioning and deployment processes. It gives you all the information you need about a Fusion Instance, Fusion Product Family, Fusion Product, and Fusion Application instances.

Increased reliability. The Cloud Control process uses a job system to automatically and regularly rediscover WebLogic domains.

Enhanced interface. Home pages enable you to monitor instances and product families at a glance. The interface makes transparent the relationship between the server and application deployment details. Cloud Control shows cluster applications from a product perspective.

Enterprise Manager Cloud Control enables you scale out a Fusion Product or a specific Fusion Application by:

Provisioning a new WebLogic Server instance within the cluster

Deploying an instance of the Fusion Application on the new WebLogic Server instance

After you provide the information required to provision the new WebLogic Server instance, Enterprise Manager invokes a deployment procedure to automatically complete the provisioning and deployment processes.

To scale out a Fusion Product Family, Product, or Fusion Cluster Application target:

From the Targets menu, choose Fusion Applications to open the Fusion Applications target home page.

Select one of the following: Fusion Product Family, Product, or Fusion Cluster Application.

Choose Provisioning > Scale Out from the Product Family, Product, or Fusion Cluster Application menu for the selected target.

Click on the Fusion Product or Fusion Application you want to scale out in the topology view in the region on the left.

Note that some Fusion Products may be associated with up to two Fusion Applications in the topology view.

The WebLogic Server cluster(s) that the selected Fusion Product or Fusion Application is currently deployed appears in the region on the right.

Click Scale Out in the row for a cluster you want to add an instance to.

Enter the credentials for the Administration Server user in the WebLogic Domain Administrator Username and Password fields.

The Administration Server is a WebLogic Server instance that manages the WebLogic Server domain the cluster belongs to.

Specify a directory to store temporary data files during the scale out operation in the Administration Host Server Working Directory field; for example, /tmp.

Enter the credentials required to access the Administration Server host.

Specify the following for the new WebLogic Server instance:

The hostname for the host that the instance will be provisioned to.

A unique identifier for the new instance.

The port that the instance will use.

Click Configure More Details.

Enter any additional information required to access the host machine that the new WebLogic Server instance will be created on. Required information includes:

The credentials required to access the host.

The directory to store temporary data files during the scale out operation; for example, /tmp

The port that the WebLogic Server instance will use.

Click OK.

Click Scale Out.

Enterprise Manager creates a Deployment Procedure job to provision the new WebLogic Service instance, add it to the cluster, and deploy the Fusion Application instance to it.

16.5.3 Scaling Out Oracle RAC Databases

This section provides guidance on how to add nodes to existing Oracle RAC environments by using Oracle cloning, and then configuring the multi data source to route database requests to the new Oracle RAC instance.

The Oracle RAC cloning procedures assume that you successfully installed and configured an Oracle RAC environment that you want to add nodes and instances to. To add nodes to an Oracle RAC environment using cloning, you must first extend the Oracle Clusterware configuration, then extend the Oracle Database software with Oracle RAC, and then add the listeners and instances by running the Oracle assistants.

16.5.3.2 Configuring the Multi Data Source to Include New Oracle RAC Nodes

After adding an Oracle RAC node to the database cluster as a part of scaling out the database, you must configure the multi data source to route database requests to the new Oracle RAC instance. You do this by adding a data source to each of the multi data sources that will route database requests to the newly added instance of the database.

On the Summary of JDBC Data Sources page, click New to create a new data source. (Choose Generic Data Source.)

In the Name field, enter the name of the computer (ApplicationDB03) and enter the JNDI Name. The Database Type is Oracle.

Click Next.

For the Database Driver, choose the driver of type Thin and RAC Server-Instance Connection.

On the next screen, add details for this data source by providing Oracle RAC instance details for the new Oracle RAC instance, plus other properties matching its peer datasource in the Multi Data Source.