Configuration Planning

This appendix contains an example of configuring and integrating Cluster Ready Services (CRS), a RAC database, and services to provide continuous application support. The basis for this example is a RAC database, called ORADB, running on a four-node cluster with one instance on each node. The instance names are RAC01, RAC02, RAC03, and RAC04. The database supports an application with five major components, ERP, CRM, SELF_SERVICE, which involve online transaction processing (OLTP) processing, and HOT_BATCH, and STD_BATCH, which are batch-oriented.

Configuration planning for high availability (HA) services involves defining which application, or parts of an application, you want to be manage with the services and which service features you want to enable.

Service Planning

To take advantage of all HA service capabilities, your plan needs to identify the service name, the primary user (client, application server, job scheduler, and so on), the preferred instances (where the service will start by default), and the available instances (where the service will run if a preferred instance becomes unavailable). You may also define the service priority (to rank importance of the services when competing for resources), and response time thresholds (to indicate when a service is not performing at its required rate).

This example includes all the options and Table A-1 summarizes the planned configuration for the ORADB database and its services, with columns Preferred Instances and Available Instances containing the HA information and columns Priority and Response Time containing the performance information.

Table A-1 Service Planning Work Sheet

Service

Usage

Preferred Instances

Available Instances

Priority

Response Time (sec) Warning / Critical

ERP

Client service

RAC01, RAC02

RAC03, RAC04

HIGH

0.5 / 0.75

CRM

Client service

RAC03, RAC04

RAC01, RAC02

STANDARD

0.5 / 1.0

SELF_SERVICE

Client service

RAC01, RAC02, RAC03, RAC04

-

STANDARD

1.0 / 1.5

HOT_BATCH

Job scheduler

RAC01

RAC02, RAC03, RAC04

HIGH

1.0 / 1.5

STD_BATCH

Job scheduler

RAC01, RAC02, RAC03, RAC04

-

LOW

3.0 / 5.0

The plan calls for the ERP service to run on instances RAC01 and RAC02 when the cluster and database starts normally. That is, ERP will become available on instances RAC01 and RAC02 as they start up. The other two instances, RAC03 and RAC04, are available for the ERP service should one of its preferred instances fail. So, for example, if RAC01 becomes unavailable, either RAC03 or RAC04 takes over running the ERP service on behalf of the failed instance while RAC02, its remaining preferred instance, continues to run the ERP service. If both RAC01 and RAC02 are disabled, the ERP service runs on RAC03 and RAC04 instead.

The plan for the CRM service is similar to that for ERP but with the instances taking on the opposite roles: RAC03 and RAC04 are instances where the CRM service should start and RAC01 and RAC02 are the instances that take over should one or both of the preferred instances fail.

Both the SELF_SERVICE and STD_BATCH services are planned to start on all four instances whereas the HOT_BATCH service starts only on RAC01. Because they are already assigned to four instances, the plan does not define any available instances for the SELF_SERVICE and STD_BATCH services. The plan allows the HOT_BATCH service to use any of the other instances should RAC01 become unavailable.

Table A-1 also lists the planned priorities and response times for the services. The plan assigns the highest priority to the ERP and HOT_BATCH services, which means they will have precedence over the other services if resources become scarce - for example, if only instances RAC03 and RAC04 are available. In such a case, the service with the lowest priority rating, STD_BATCH, may be terminated and, if necessary, the CRM or SELF_SERVICE services could be flagged for termination. The response times are thresholds for which notifications should be triggered if performance falls below the listed values.

Cluster Node and Network Interface Planning

When you have completed your logical configuration, you may want to prepare an interface worksheet to record the cluster node interface names and addresses. Most of your interfaces, particularly the public interfaces, will have equivalent host names and domain names. In cases where names are resolvable to IP addresses, you may have provided these names when using the Oracle Universal Installer (OUI) and the Database Configuration Assistant (DBCA). Similarly, you may have entered the name, if there is one, or the IP address of the private interconnects used for the cluster interconnect interface. As you complete your interface worksheet, you may want to record the names, when they exist, along with the IP addresses.

Your virtual IP addresses (VIPs) should not be fixed to any physical interface on your network and VIPs may or may not have a corresponding name. In NetCA or in tnsnames.ora, you can enter either the names or the IP addresses of the VIPs. For vendor systems that support cluster aliases, you can replace the list of names or IP addresses with the corresponding cluster alias name or IP address. The cluster alias name for this example is clusalias. To execute some steps shown in this example, you will need to know the subnet mask for all of your VIPs used and location of your CRS home directory. In this example, the subnet mask is 255.255.255.0 for all VIPs and the CRS home directory is /private/oracle/crs, neither of which is included in the following table.

Manual Configuration for High Availability

The three steps in the first part of this example show you how to build the configuration, based on the information shown in Tables (UNKNOWN STEP NUMBER) and (UNKNOWN STEP NUMBER) . See RACAD Appendix B for a complete list of SRVCTL commands and syntax.

Note:

You must be logged into the system as root on UNIX or administrator on Windows when adding the VIPs. All other SRVCTL operations are executed as the Oracle owner and DBA group.

Step 1. Add Node Applications

Most of your node applications configuration should have been completed during your Cluster Ready Services (CRS) installation. You can verify this by running crs_stat command, which should show a sequence of resource metadata and a listener resource on each active node in the cluster. If you need to add new node application manually, for example, suppose you added clusnode-5 after your initial Oracle installation, you would use the following SRVCTL command logged in as root on UNIX or administrator or Windows:

Using Services

In this section of the example, you can see how to set up your Oracle Net Services configuration files and other application-related resources to ensure your application uses the services you have configured.

Using Services with Client Applications

Applications and mid-tier connection pools select a service by using the TNS connection data. The service must match the service that has been created using add service with SRVCTL or DBCA. You can check the services that are currently running by querying the V$ACTIVE_SERVICES view.

You may use the virtual addresses for client communication - this ensures that connections and SQL statements issued against a node that is down do not result in a TCP/IP time out. If your system offers a cluster alias, you may use the cluster alias for the connection only. However, you must not use host names as addresses. The address lists in the following examples use either virtual IP addresses or cluster alias.

Listener Configuration for Services

You should cross-register your listeners using your REMOTE_LISTENERS initialization parameter so that all your listeners know about all of your services and the instances in which they run. The listeners should use server side load balancing, optionally based on session count for connection. The listeners must be listening on the VIPs and on the cluster aliases, when available. The listeners must not listen on the host name - listening on the host name will result in disconnected sessions when VIPs are relocated automatically back to their owning nodes.

Sample listener.ora Entry

Each listener on each cluster node should have dual addressing, one pointing at the node VIP name (or address) and the other pointing at the host's physical IP address (or name).

Oracle Instance Parameters

You must ensure that the LOCAL_LISTENER, REMOTE_LISTENER, and ACTIVE_INSTANCE_COUNT initialization parameter values are valid to use the VIPs for your services. The listener definition values should the same as those defined in the section "Using Services with Client Applications ". Follow these guidelines to set the correct values:

You must ensure that the ACTIVE_INSTANCE_COUNT parameter is left at its default value - this parameter must not be set.

Manual Configuration for Workload Management

The four steps in this second part of this example show you how to complete your service configuration to enable workload management, DBMS_SCHEDULER.CREATE_JOB execution time, resource consumption, and wait events. The first two steps are required to configure the service priorities and the job classes for the server side services in the Automatic Workload Repository (AWR). Steps three and four define service performance thresholds and enable the measurement of modules and actions within services.

Step 1. Add Service Priorities

Before mapping services to consumer groups, you must create the required consumer groups and their related resource plans, which can be priority based or ratio based. For this example, the site already has three consumer groups named high_priority, standard_priority, and low_priority. These consumer groups map to a database resource plan that reflects the intended resource consumption.

The following SQL*Plus commands call PL/SQL to map each service to the desired consumer group and then display the results by querying DBA_SCHEDULER_JOB_CLASSES:

Step 2. Add Job Classes

The database employs two batch queues managed by the Job Scheduler, called HOT_BATCH and STD_BATCH. These queues correspond to job classes with services of the same name. The following PL/SQL code creates the job classes with assigned services:

Step 4. Enable Service, Module, and Action Monitoring

You can enable performance data and tracing for important modules and actions within each service. The performance statistics are available in the V$SERV_MOD_ACT_STATS view. The following commands, executed in a SQL*Plus session, perform these actions:

Enable monitoring for the exceptions pay action in the module, payroll, under the ERP service

Enable monitoring for the all actions in the module, payroll, under the ERP service

Enable monitoring for the all actions in the module, posting, under the HOT_BATCH service

Using Services with Job Scheduler

Use the DBMS_SCHEDULER.CREATE_JOB procedure to define jobs to execute under the job classes. In this example, the MY_NAME.MY_PROC procedure will run in the HOT_BATCH service because of the job class assignment defined earlier, in Step 2:

Using Callouts for Fast Application Notification

Custom-written application callouts are programs or shell script wrappers that can be used to start and stop on- or off-cluster applications, or connection pools managed by middleware. They are immediately executed by RAC when a service or any part of the service starts, stops or fails to automatically restart. Other actions that can be encoded as callouts (besides restarting applications) include: logging fault tickets, e-mailing or paging administrators, and invoking third-party event systems or clusterware components.Callouts are not a requirement to deploy RAC-HA on CRS, but Oracle strongly advises customers to build notification mechanisms using callouts. Unless your CRS home directory is shared across the network, you must deploy each new callout under /private/oracle/crs/racg/usrco directory on each RAC node.The following example, a Bourne shell script, contains a number of callout options that are invoked whenever an HA event occurs. The callouts perform the following two actions: write an uptime status record to the log, and log a fault ticket (with the IT trouble ticket application) for all DOWN conditions.

Configuring JDBC Fast Application Notification

To use Fast Application Notification, the application must use the JDBC Implicit Connection Cache. JDBC connection pools are integrated with the callout mechanism, providing the following benefits:

Balancing connections across RAC when a service first starts up - rather than directing the minimum sessions defined for the connection pool to the first RAC instance that supports the service. Consuming additional RAC instances immediately a service is registered UP at additional instances. Cleaning up terminated connections immediately a service is declared DOWN at any instance, and immediately that nodes are declared DOWN Reporting an error to clients immediately the NOT RESTARTING status is detected, instead of making the client wait while the service repeatedly tries to restart.Distributing client work requests at runtime across the RAC instances supporting a service.

Configuring the JDBC Client Side

Refer to JDBC User's Guide and Reference for configuring the JDBC Implicit Connection Cache and Oracle Notification Service (ONS). If ONS is not configured correctly, the implicit connection cache creation fails and an appropriate exception occurs upon the first getConnection()request.Set the ConnectionFailoverEnabled property before making the first getConnection() request to a DataSource. When Fast Connection Failover is enabled, the failover applies to every connection in the connection cache. If your application explicitly creates a connection cache using the Connection Cache Manager, you must first set ConnectionFailoverEnabled.

Configuring the RAC High Availability Server Side

The RAC event system needs to be configured to forward the HA events to every ONS, so that ONS clients at the mid-tier can receive and respond to the state changes.

Step 1 - Configure the ONS Daemon

The ONS daemon must be configured to broadcast the HA events from RAC to Oracle Application Server 10g clients. The ONS configuration file is located in --$ORACLE_HOME/opmn/-conf/ons.config and file should be built by OUI during installation Here is a sample RAC ons.config file:

where nodes is a list of all ONS daemons that talk to each other on RAC and Oracle Application Server. These values are given as a list of either host name or IP address plus port combinations in a comma separated list. Note that the port value that is given is the remote port that each ONS instance is listening on.

Note that the port value that is given is the remote port that each ONS instance is listening on.

Step 2 - Check that the ONS Daemon is Running

The ONS daemon is running as a node application. To check node applications use the command: srvctl status nodeapps. Your results should be similar to the following:

NAME=ora.clusnode-4.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE

Use onsctl ping to check that the ONS daemon is active.

Using a Shared Oracle Home

ONS requires that the $ORACLE_HOME/opmn/log directory is private for each ONS daemon. If using a cluster file system for $ORACLE_HOME, each node should define $ORACLE_HOME/opmn/log as a link to a node specific directory, for example, $ORACLE_HOME/opmn/clusnode1/log