5 Configuring High Availability for Oracle SOA Suite

Oracle SOA Suite provides a complete set of service infrastructure components for designing, deploying, and managing composite applications. Oracle SOA Suite enables you to create, manage, and orchestrate services into composite applications and business processes. Composites enable you to assemble multiple technology components into one SOA composite application. Oracle SOA Suite plugs in to heterogeneous IT infrastructures and enables enterprises to incrementally adopt SOA.

This chapter describes Oracle SOA Suite components from a high availability perspective. This chapter also includes sections that outline the single-instance concepts for SOA components that are important for designing a high availability deployment.

5.1 Introduction to Oracle SOA Suite

As shown in Figure 5-1, Oracle SOA Suite provides a comprehensive suite of products for developing, securing, and monitoring service-oriented architecture (SOA). Oracle SOA Suite 11g provides a unified runtime engine based on the SCA standard. The runtime engine consists of service engines (BPEL Process Manager, Human Workflow, Mediator, Rules) and binding components (JCA Adapters, B2B) that a common service infrastructure manages and inter-connects. The service infrastructure also provides common services such as lifecycle management and deployment.

A SOA composite application is the basic unit of deployment to the SOA runtime engine. Service components (BPEL process, business rule, human task, and Mediator routing rule) are the building blocks of a SOA composite. Service components target service engines during deployment, while services and references are enabled using the binding components. At runtime, the binding component receives messages, which the service infrastructure then routes to the appropriate service engines.

The SOA runtime engine operates within the context of an application server, such as the Oracle WebLogic Application Server. It leverages the underlying application server capabilities for load balancing and high availability.

This guide provides high availability information for the following Oracle SOA Suite components:

Oracle SOA Service Infrastructure

Oracle BPEL Process Manager (Oracle BPEL PM)

Oracle BPM Suite

Oracle Mediator

Oracle Human Workflow

Oracle JCA Adapters

Oracle B2B

Oracle Web Services Manager

Oracle User Messaging Service

Oracle Business Activity Monitoring

5.2 Oracle SOA Service Infrastructure High Availability

This section describes the issues and considerations necessary for designing a SOA Service Infrastructure high availability cluster.

Oracle SOA Service Infrastructure is a Java EE application that provides the foundation services for running Oracle SOA Suite. This Java EE application is a runtime engine that deploys automatically when you install Oracle SOA Suite. You deploy composites, the basic artifacts in a Software Component Architecture, to the Oracle SOA Infrastructure and it provides the required services for the composites to run. Oracle SOA Infrastructure provides deployment, wiring, and thread management services for the composites. These services sustain the composite's lifecycle and runtime operations.

Oracle SOA service infrastructure provides the required services for running composites. A composite is a basic unit of deployment for Service Component Architectures (SCA). The SCA Assembly Model consists of a series of artifacts, which are defined by elements contained in XML files.

Composites are software packages made up of components, wires, services, and references. For example, an Oracle BPEL process is a component; an inbound adapter is a service; and an outbound adapter is a reference. Wires provide connections between service engines.

Each component targets a specific service engine such as Oracle BPEL PM. Oracle SOA Service Infrastructure connects to a SOA database to maintain composite state and to the SOA Oracle Metadata Services (MDS) repository to maintain composite metadata, such as deployments and version tracking. These two databases may be the same physical database but the schemas used for each purpose are different. SOA infrastructure provides a servlet for remote deployment of composites. The metadata and artifacts for remote deployments are also stored in the MDS repository. For more information on the MDS repository, see Section 4.1.3.1, "Configuring Multi Data Sources for MDS Repositories."

The Oracle SOA service infrastructure application is also responsible for targeting the individual components to their specific engine and for instantiating these composites when requests reach the SOA system. After targeting and instantiation, the Oracle SOA service infrastructure controls the thread and resource assignment. This happens in the JVM, where the composite runs.

As shown in Figure 5-3, the Oracle SOA service infrastructure integrates SOA composite applications with UDDI registries. UDDI registries provide a standards-based foundation for locating published services, invoking services, and managing service metadata. The Oracle SOA service infrastructure is also the central hub used by the service engines to deliver messages through Oracle User Messaging Service infrastructure to communication channels, such as email and voice.

5.2.1.1 Oracle SOA Service Infrastructure Application Characteristics

The soa-infra-wls.ear file contains Oracle SOA service infrastructure services. None of the services provided by the Oracle SOA service infrastructure system are singletons; therefore, the Oracle SOA service infrastructure can run in full active-active mode. The SOA Service Infrastructure Java EE application contains a web module that provides browsing of the deployed composites and links to the test pages for these composites. This web module uses /soa-infra as the associated URL context. This web module is stateless and does not have any specific session replication requirements.

Other modules in the Oracle SOA Service Infrastructure application provide task control for process instantiation and process tracking, and client services for accessing Oracle User Messaging System (UMS).

A task service controls instantiating and tracking processes asynchronously. In addition, there are multiple EJBs used by the Oracle SOA service infrastructure system. However, all of the EJBs are stateless, and there are no requirements for stateful session bean replication in an Oracle SOA cluster. The processing of transactions by these EJBs relies on Oracle WebLogic Server transaction control service. Configure the appropriate transaction stores as recommended in the basic Oracle WebLogic Server guidelines to guarantee recovery across failures in an Oracle WebLogic Server container.

These components are assembled together into an Oracle SOA composite application. This application is a single unit of deployment that simplifies the management and lifecycle of Oracle SOA applications.

When the Oracle SOA Service Infrastructure application starts, it initializes the different service engines and loads composites from the MDS repository. It targets the individual components to their specific engines. After the composite loads, the system is available to receive requests. At runtime, the Oracle SOA Service Infrastructure manages all communication across service components. Calls between service engines are in-process calls. Figure 5-4 shows the sequence for the Oracle SOA Service Infrastructure startup and processing of work.

5.2.1.3 Oracle SOA Service Infrastructure External Dependencies

Composite metadata is stored in the MDS database schema, which acts as a repository.

In a clustered environment, the deployment coordinator service depends on the underlying Coherence cluster for signal propagation.

These components must be available for the Oracle SOA Service Infrastructure to start and run properly.

5.2.1.4 Oracle SOA Service Infrastructure Startup and Shut Down of Processes

The Oracle SOA Service Infrastructure application starts, by default, whenever any Oracle WebLogic Managed Server to which the Oracle SOA Service Infrastructure was deployed is started. Typically, you do not need to stop the Oracle SOA Service Infrastructure or any of its components by themselves. Some operations may require Oracle WebLogic Managed Server where the SOA Service Infrastructure runs to be rebooted. Only some patching scenarios require stopping the application.

You can use the Administration Console to verify status and to start and stop Oracle WebLogic Server. You can also use the WebLogic Server WLST command line to control the application. Oracle Enterprise Manager Fusion Middleware Control also enables multiple operations and configuration of the Oracle SOA Service Infrastructure application and monitoring its status. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite for information on monitoring and controlling the Oracle SOA Service Infrastructure application.

After JNDI names are read from this file, the databases used by the system are determined by the data sources that matched those JNDI names in the WebLogic Server JDBS resources configuration.

The server and callback server URL

A callback server URL is the address that asynchronous services specify to be notified of a response to the service they invoked.

Note:

If a request to an external or internal asynchronous service originates from Oracle SOA Suite, the callback server URL is determined using the following in decreasing order of preference:

Use callbackServerURL specified as a binding property for the specific reference. (You can set this when modeling the composite or at runtime using the MBeans). This enables different service calls to have different callback server URLs. At runtime, you set this property with System MBean Browser, through the corresponding binding Mbean. To add a specific URL, add a callbackServerURL property to its Properties attribute, then invoke the save operation.

Use the callback server URL as specified in the SOA Database. In this case, only one address can be specified and you must use an address that works well for all possible services.

Use the callback server URL as the frontend host specified in WebLogic Server for the SOA_Cluster. In this case as well, only one address can be specified, and the recommendation is the same as for the SOA Database configuration option.

Use the local hostname as provided by WebLogic Server MBean APIs. This is not recommended in high availability environments.

The audit level of information to be collected by the message tracking infrastructure.

Other configuration options at the container level, such as data sources, JTA configuration, and persistent stores, are maintained as part of the WebLogic Server Domain configuration, and are synchronized across a cluster of Oracle WebLogic Servers by the Oracle WebLogic Server core infrastructure.

5.2.1.6 Oracle SOA Service Infrastructure Log File Locations

Oracle WebLogic Managed Server logs operations that Oracle SOA Service Infrastructure and its components perform where the SOA Service Infrastructure is running. You can find these logs at the following location:

DOMAIN_HOME/servers/WLS_ServerName/logs/WLS_ServerName.log

The log files for the different Oracle WebLogic Server managed servers are also available from Administration Console. To verify the logs, access Administration Console using the following URL: admin_server_host:port/console. Click Diagnostics-Log Files.

It is also important to verify the output of the Oracle WebLogic Managed Server where the Oracle SOA Service Infrastructure is running. This information is stored at the following location:

DOMAIN_HOME/servers/WLS_ServerName/logs/WLS_ServerName.out

Additionally, a diagnostic log is produced in the log directory for the managed server. This log's granularity and logging properties can be changed through the domain_dir/config/fmwconfig/servers/WLS_ServerName/logging.xml file. The properties in this file can also be modified from Oracle Enterprise Manager Fusion Middleware Control by selecting Farm, SOA, SOA Server. Right-click, and select Logs, and then Log Configuration.

Oracle Enterprise Manager Fusion Middleware Control enables performing selective searches in all the logs in the SOA domain. To do a selective search, access Oracle Fusion Middleware Control and click on Farm-Logs and enter the search criteria that pertain to soa-infra or deployed composites. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite for more details on the logs and information reported for a SOA System in Oracle Enterprise Manager Fusion Middleware Control.

Figure 5-5 shows a two-node Oracle SOA Service Infrastructure cluster running on two Oracle WebLogic Servers. The Oracle WebLogic Servers are frontended by two Oracle HTTP Server instances on web tier hosts, which receive requests from a load balancer in front of them.

Figure 5-5 shows the following main characteristics of this high availability configuration:

The SOA Service Infrastructure runs in Oracle WebLogic Server managed servers that are part of an Oracle WebLogic Server cluster. Oracle WebLogic Server cluster synchronizes the configuration for common artifacts of an Oracle WebLogic Server cluster that Oracle SOA Service Infrastructure uses such as JTS configuration, data sources, and persistent store definitions.

The Oracle SOA Service Infrastructure uses the front-end host and port information configured for the Oracle WebLogic Server cluster as the server and callback server URL. You define these settings with the Administration Console. Select Clusters, SOA_Cluster_Name, HTTP/HTTPS frontend host, and then Port. If there is no address for the Oracle WebLogic Server cluster where the Oracle SOA Service Infrastructure is running, the system uses the physical hostname as the server and callback server URL.

For SOA high availability installations frontended by Oracle HTTP Server, monitoring is done on the Oracle HTTP Server listen ports. This is the case when a deployment uses all components deployed to the SOA managed server. A simple HTTP monitor that pings the HTTP/HTTPS port and expects a predetermined response will suffice. If only a specific SOA component is being used, such as B2B, then a monitor that does a deeper level check all the way to the damaged server can be considered to validate the health of the component in use. Check with your load balancer vendor on how to set up the HTTP monitors with your load balancer.

For more information about the server and callback server URLs, see the Oracle Fusion Middleware Administrator's Guide. Changing the HTTP front-end address for a cluster requires a restart of the managed servers that are part of the cluster.

The deployment coordinator is configured and used to deploy and update composites. The deployment coordinator sends notifications to deployment coordinator cluster members to retrieve new artifacts from the MDS repository, when they are updated by the group leader. A leader node performs singleton operations for the cluster, such as updating the MDS after deployments or changes are made to the composites.

The Oracle SOA Service Infrastructure system uses the Oracle WebLogic Server cluster name as its key to confirm a deployment coordinator group. If all nodes in a Oracle WebLogic Server cluster can communicate (over multicast or unicast), the deployment coordinator cluster is the same as the WebLogic Server cluster in which the Oracle SOA Service Infrastructure runs.

The Administration Server runs in active-passive mode. Whenever a failure occurs in SOAHOST1, you can restart the Administration Server in SOAHOST2; it uses a virtual IP or virtual hostname as a listener address.

Although SOA Service Infrastructure can run in full active-active mode, the architecture uses the Oracle WebLogic Server Migration feature to protect some SOA components against failures.

As shown in Figure 5-5, WLS_SOA1 listens on VIP1, and WLS_SOA2 listens on VIP2. Each managed server uses the other node as a failover resource; the system is configured in a cross manner. WLS_SOA1 fails over to SOAHOST2, and WLS_SOA2 fails over to SOAHOST1. The appropriate capacity planning must be done to anticipate the scenario where the two SOA managed servers are running on the same node. For more information on Server Migration features, see Chapter 3, "High Availability for WebLogic Server" in this guide.

To resume transactions after a server migration, configure the transaction and JMS stores in a shared storage. In case of failure in one of the server infrastructure instances, other instances can resume transactions and JMS operations by reading the persistent stores from that shared storage.

The metadata store is configured in an Oracle RAC database to protect it from database failures. Similarly, the SOA process state information is also stored in an Oracle RAC database. In this example, both Oracle RAC databases are the same.

About Oracle SOA Service Infrastructure Components

These high availability characteristics apply to most of Oracle SOA Suite components contained in the composite applications deployed across the cluster. For specific two-node high availability characteristics of the individual components, see the specific component sections that follow in this chapter.

5.2.2.1.1 WebLogic Server Failure

If a WLS_SOAx server fails, Node Manager restarts it locally. If repeated restarts fail, the WebLogic Server infrastructure performs a server migration of the WLS_SOAx server to the other node in the cluster. While the failover takes place, the other SOA Service Infrastructure instance becomes the leader for deployments and composite updates and provides the basic services required by the service engines in the system.

Ongoing requests from the Oracle HTTP Server time out and new requests are directed to the other WLS_SOAx server. After the server's restart completes on the other node, Oracle HTTP Server resumes routing any incoming requests to the server. The migrated server reads the MDS repository for any updates that took place during restart, and joins the deployment coordinator cluster to listen for new updates. The migrated server also resumes any pending transactions from the transaction logs in shared storage.

In the server migration scenario, the service engines, such as Oracle BPEL PM and Oracle Mediator, are failed over together with the SOA Service Infrastructure. They do not re-issue any requests to the other SOA Service Infrastructure instances by themselves. They resume operations together with the SOA Service Infrastructure after failover is complete.

The Oracle SOA Service Infrastructure application may be unavailable due to failure accessing resources, errors caused by the deployment coordinator infrastructure, or other issues unrelated to whether the managed server is running. Therefore, Oracle recommends administrators monitoring the soa-infra application for errors caused by the application in the managed server logs. For information about log file locations, see Section 5.2.1.6, "Oracle SOA Service Infrastructure Log File Locations".

5.2.2.1.2 Node Failure

If node failure occurs, server migration is triggered after the available server verifies the time stamp in the database leasing system. If the failed server was the deployment coordinator cluster master, the available server becomes the new master and the SOA Service Infrastructure remains available for deployment and for composite lifecycle. After the time stamp for leasing is verified, Node Manager in the node that still remains available attempts to migrate the VIP address the failed managed server uses, and restarts it locally. The result is that the SOA Service Infrastructure application has two instances running in the same node. For more information on the failover process, see Section 3.9, "Whole Server Migration".

Service engines are deployed to the container as a part of the Service Infrastructure application. These service engines contain all of the ear files and library calls.

5.2.2.1.3 Database Failure

GridLink data sources and multi data sources protect the SOA Service Infrastructure system against database failures. You configure GridLink and multi data sources during the system setup and guarantee that when an Oracle RAC database instance fails, the connections reestablish with available database instances.

Note:

You must disable the fast connection failover flag for SOA or a Null Pointer Exception occurs.

5.2.2.2 Oracle SOA Service Infrastructure Cluster-Wide Deployment

As previous sections explain, SOA Service Infrastructure centrally stores composite deployments in the MDS repository. Each time the SOA Service Infrastructure starts, it synchronizes itself with the MDS repository and SOA store to get the deployment and process state. The deployment coordinator infrastructure creates notifications for composite deployments and updates. When a new deployment or update takes place, deployment coordinator notifies all members in the cluster. When all members in the cluster confirm that the deployment succeeded, the master sends a notification to start the composite. If a deployment fails on a node, it rolls back to the rest of the cluster. An error message in the deployment coordinator master (WebLogic Server managed server) shows the node that the deployment fails on. Figure 5-6 shows this process.

5.2.2.3 Online Redeployment of Oracle SOA Service Infrastructure Composites in a Cluster

When the Oracle SOA Service Infrastructure performs an update or redeployment of a composite, it can overwrite an existing version (x) or create a new version (x+1). All composites are uniquely identified based on the composite name and revision. By default, clients accessing a composite use the version that is identified in the MDS repository as the default version (also visible from Oracle Enterprise Manager Fusion Middleware Control). You can manage the lifecycle of each version separately and perform online redeployments of composites, even with one single instance of SOA Service Infrastructure. To perform this operation:

Deploy version x.

Mark version x as the default.

Deploy a new version of the composite (x+1) and mark version x as the default. This step stops users from accessing the new version in default access.

Test the new version by accessing from a test client by specifying the version of the composite (x+1)

After verifications are complete, mark x+1 as the default version. New clients are routed to version x+1, while old clients complete their work in version x.

It is possible to leave the previous version deployed, or undeploy it. If you undeploy a composite that has in-flight instances, they become stale and do not complete. Undeploying a composite removes the composite from the MDS repository.

The standard Java EE artifacts that the Oracle SOA Service Infrastructure uses are configured as part of the Oracle WebLogic domain in which SOA is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, persistent stores, and JMS modules, across WebLogic Server domain. At the same time, the WebLogic Server cluster controls synchronization of the deployments and libraries used by the SOA Service Infrastructure.

As explained in Section 5.2.1, "Oracle SOA Service Infrastructure Single-Instance Characteristics," SOA Service Infrastructure-specific configuration is stored in the SOA database. Changes are applied once per SOA server, but the changes affect all SOA servers in the same SOA domain. For example, in the high availability topology described in this chapter, if you change the callback server URL or audit level for server WLS_SOA1, the change also applies to server WLS_SOA2.

Note:

Changes to properties of a composite from the Oracle Enterprise Manager Fusion Middleware Control using the MBean browser are also reflected in all the nodes of a SOA cluster. For example, if you change the Mbean by selecting oracle.soa.config node, server:WLS_SOA1(clustered node), SCAComposite, My Composite, this change propagates to other servers in the cluster.

5.3 Oracle BPEL Process Manager and High Availability Concepts

This section guides you through the issues and considerations that you must know before configuring Oracle BPEL PM for high availability.

5.3.1 Oracle BPEL Process Manager Single-Instance Characteristics

Service engines are containers that host the business logic of service components in a SOA composite application. Each service component, such as Oracle BPEL PM, runs in its own service engine. (Decision Service runs in the business rules service engine.) A service engine plugs into the Oracle SOA Service Infrastructure. Oracle BPEL Process engine is the service engine running in the Oracle SOA Service Infrastructure that enables the execution of BPEL processes.

A BPEL process provides the standard for assembling a set of discrete services into an end-to-end process flow, and developing synchronous and asynchronous services into end-to-end BPEL process flows. It provides process orchestration and storage of long running, asynchronous processes.

Figure 5-7 shows that the BPEL engine is a stateless part of the Oracle SOA Service Infrastructure. It starts with the Oracle SOA Service Infrastructure application and contains modules that run BPEL processes. If a deployed composite contains a BPEL process, SOA Service Infrastructure invokes BPEL Process Manager to get the component from the MetaDataStore.

The BPEL Process Manager engine saves the processes' execution state in the SOA database through a persistence module based on Oracle TopLink. The audit framework continuously audits the work being processed by storing process execution information in the SOA database.

5.3.1.1 BPEL Process Manager Component Characteristics

The BPEL Service Engine runs inside the SOA Service Infrastructure Java EE application (soa-infra.ear). The BPEL Service Engine does not have any singleton services. All states associated with a BPEL process are stored in a database (dehydration store), and there is no in-memory state replication required.

The processing of work by SOA Service Infrastructure Java EE EJBs is transactional and relies on Oracle WebLogic Service transaction control service. You configure the appropriate transaction stores as recommended in the basic WebLogic Server guidelines to guarantee recovery across failures in the WebLogic Server container. The BPEL engine system doesn't contain any web modules; therefore, session replication is not required in the servlet layer when running Oracle BPEL PM in active-active mode.

In this release of Oracle Fusion Middleware, BPEL Service Container does not rely on JMS for asynchronous message dispatching. Therefore, there is no dependency on a distributed JMS infrastructure.

External Dependencies

The BPEL engine system depends on the following components:

SOA Service Infrastructure database for BPEL process state persistence

MDS repository for BPEL process metadata store

Both components must be available for the BPEL engine system to start and run properly.

5.3.1.2 Oracle BPEL Process Manager Startup and Shutdown Lifecycle

As Figure 5-8 shows, when the Oracle SOA Service Infrastructure application starts, it initializes the BPEL engine and loads the composites from the MDS repository. If the composite contains any BPEL processes, it targets those individual components to the BPEL engine. After the process loads, the system can receive requests. At runtime, the BPEL engine waits for requests from different channels, such as JMS and the database.

Composites load from the MDS repository by SOA Service Infrastructure.

BPEL PM components are dispatched to the BPEL engine to be loaded.

Composite-binding components are activated.

The BPEL engine services requests.

The shutdown signal is received by SOA Service Infrastructure.

SOA Service Infrastructure starts undeploying loaded composites.

Composite-binding components are disabled.

BPEL components are dispatched to the BPEL engine to be unloaded.

The BPEL engine shuts down.

5.3.1.3 Oracle BPEL Process Manager Request Flow and Recovery

Recoverable activities are activities that failed and can be recovered. For example, if you are using the file adapter to initiate an asynchronous BPEL process and your system fails while the instance is processing, you can manually perform a recovery when the server restarts to ensure that all message records are recovered.

There are two types of BPEL Processes based on the invocation interface:

One-way: Most commonly asynchronous fire-and-forget pattern

Two-way: Synchronous request-response pattern

The recovery semantics after a server failure are based on whether the process is invoked synchronously and asynchronously. The following describes the behavior based on the invocation and process type:

Synchronous invocation (request): For synchronous requests, an error is thrown back to the client if a server fails. It is the client's responsibility to handle the error message and take the appropriate action, such as retrying the request. This is true for both transient and durable processes.

Note:

For durable processes, the message is persisted to the dehydration store at certain points. It is possible to recover the message from the dehydration store and replay the process using Enterprise Manager, however, the client cannot be notified of the response. Therefore, Oracle does not recommend this option; it is preferable to handle all recovery from the client.

Asynchronous invocation (post): There are two types of asynchronous invocations: one that starts a new process and one that is a callback to an existing process. For a callback, the engine, after recognizing that it is a continue operation, attempts to resolve the subscribing process, first by conversation ID, and then by a correlation set if defined. Messages associated with asynchronous requests are persisted to the dehydration store as part of the client call. If there is a failure before the persistence, the client receives an error message and handles the error in the same way that it handles synchronous invocation errors. If the persistence is successful, the client call returns and further processing is done outside the context of the client call. In the case of a server failure, one-way processes that were invoked asynchronously can be restarted using Oracle Enterprise Manager.

5.3.1.4 Oracle BPEL Process Manager Configuration Artifacts

To configure these parameters, go to the Oracle Enterprise Manager Fusion Middleware Control, and in the navigation tree, select SOA, soa-infra, (server_name). Right-click SOA Administration, and select BPEL Properties, or use the MBean Browser to go to the appropriate property.

These properties are specific to each WebLogic domain directory that contains WebLogic Servers running Oracle SOA Suite. Other configuration options at the container level, such as data sources and persistent stores, are maintained as part of the WebLogic Server Domain configuration, and are synchronized across a cluster of WebLogic Servers by the WebLogic Server core infrastructure. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite for more details on BPEL engine configuration.

Figure 5-5 describes an Oracle SOA Service Infrastructure two-node cluster running on two WebLogic Servers. Oracle BPEL PM deploys as part of the Oracle SOA Service Infrastructure.

Note:

If you make BPEL Process Manager performance tuning changes, note the following:

If you set OneWayDeliveryPolicy to async.cache, invoke messages do not persist to the database; they are in memory. If the SOA node that holds these messages fails, they are lost. Use the async.cache option only when performance is critical and when you have an external means to track messages and redeliver them.

If you set auditStorePolicy to async, audit messages write to the database in a separate thread and transaction, which increases transaction speed. However, if a node crash occurs, audit trace functionality may be lost. Oracle recommends setting auditStorePolicy to async only if enhanced performance is critical.

The WebLogic Server infrastructure protects the BPEL engine system from all process failures. The following process failure considerations apply to Oracle BPEL PM:

If managed servers fail, Node Manager tries to restart them locally. If the whole server migration is configured and repeated restarts fail, the WebLogic Server infrastructure performs server migration to the other node in the cluster, if it is configured. After the server on the other node restarts, Oracle HTTP Server resumes routing any incoming requests to it. The migrated server reads the SOA database, resumes any pending processing, and resumes transactions from the transaction logs in shared storage.

The BPEL PM Service Engine or the entire SOA Service Infrastructure may be unavailable because of errors related to JDBC data sources, or Oracle Coherence configuration, even after a successful managed server startup. Therefore, you must monitor SOA Service Infrastructure logs for errors. For the location of server logs, see Section 5.2.1.6, "Oracle SOA Service Infrastructure Log File Locations".

In addition, for the handling of failures in the BPEL Engine itself with the WebLogic Server infrastructure, you can define and perform fault recovery actions on BPEL process faults identified as recoverable in Oracle Enterprise Manager Fusion Middleware Control. The recovery actions you perform on faults are based on actions you defined in your fault recovery policy files for BPEL process service components.

Oracle Enterprise Manager Fusion Middleware Control can show these types of faults:

Business: Application-specific faults that are generated when there is a problem with the information being processed (for example, a social security number is not found in the database).

System: Network and other types of errors, such as a database server being completely unavailable or a web service being unreachable.

The inbound payload for one-way invocation requests is stored in the dehydration store and committed with the current global transaction context. If the caller has already started a global transaction, the invocation message is saved when the caller commits the transaction; if there is no incoming transaction, a new transaction is started to commit the invocation message.

If you cannot find a BPEL PM instance in Oracle Enterprise Manager, the invocation message may be in the manual recovery console, which lists all incoming messages that have not been delivered yet.

If the invocation message is not in the recovery console, check the status of the transaction from the caller side.

Transactional Issues with Endpoints

BPEL asynchronously saves the audit trail for instances that cannot be dehydrated within the global transaction in which the request was started. If any transactional services that are referenced from a BPEL component roll back the transaction, the BPEL instance audit trail should still appear in Enterprise Manager. If there is a problem with the dehydration store, the entire global transaction rolls back and no audit trail is saved. In this case, a message or activity can be recovered to pick up processing from the last known dehydrated point.

If the dehydration point is an Oracle RAC database with multiple nodes and the Java EE server on which the BPEL engine is hosted loses its JDBC connection to the Oracle RAC node, the BPEL engine attempts to retry the transaction (thereby rolling back whatever changes were made in the current transaction). If a new connection cannot be established to a new Oracle RAC node, recover the BPEL message or activity with the recovery console.

Logging

BPEL engine loggers are set to the INFO level by default. To find a message or instance, set the loggers to TRACE level to enable more output to the WebLogic Server logs. For dehydration log messages, enable oracle.soa.bpel.engine.data. For thread or dispatcher log messages, enable oracle.soa.bpel.engine.dispatcher. To capture all engine log messages, enable oracle.soa.bpel.engine.

5.3.2.1.1 Recovering Failed BPEL and Mediator Instances

If a server fails, in-flight one-way processes that are invoked asynchronously may require manual recovery. These processes are marked as Recoverable; you can restart them using Oracle Enterprise Manager as Figure 5-11 shows.

The standard Java EE artifacts that BPEL engine uses are configured as part of Oracle WebLogic Domain in which SOA is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, persistent stores, and JMS modules across the WebLogic Server domain.

As Section 5.3.1, "Oracle BPEL Process Manager Single-Instance Characteristics" explains, BPEL engine-specific configuration is stored in the SOA database. Changes are applied once per SOA server but affect all BPEL instances in the same SOA domain. For example, if WLS_SOA1 and WLS_SOA2 are part of the same SOA cluster and the number of dispatcher threads for BPEL in WLS_SOA1 is changed to 20, WLS_SOA2's dispatcher threads are set to 20.

5.4 Oracle BPM Suite and High Availability Concepts

Oracle BPM Suite provides a complete set of components for designing, deploying, and managing business process management (BPM) projects.

5.4.1 Oracle BPM Suite Single Instance Concepts

As Figure 5-12 shows, Oracle BPM Suite provides a comprehensive suite of products for developing, managing, and monitoring BPM projects. Based on Oracle SOA Suite 11g, the complete set of technologies and standards such as SCA can be leveraged by the BPM Suite 11g. The BPM runtime consists of service engines (BPMN Service Engine, BPEL Process Manager, Human Workflow, Rules, Mediator) and binding components (JCA Adapters, B2B) that a common service infrastructure manages and interconnects. The service infrastructure also provides common services for lifecycle management and deployment of BPM projects.

A BPM project is the basic unit of deployment to the BPM runtime engine. A BPM project comprises service components (such as BPMN Process, Business Rules, Human Workflow) and references organized as a SOA Composite, organizational data (such as Roles and Business Calendars), business indicator metadata and dashboard data.

Components are targeted to service engines during deployment while services and references are enabled using the binding components. The metadata for organizational data, business indicators, and dashboards is persisted and evaluated by appropriate components at runtime. At runtime, the binding component or the BPM Workspace receives messages, which the Service Infrastructure routes to the appropriate service engines.

The BPM runtime engine performs within the context of an application server such as the Oracle WebLogic Server. It leverages the underlying application server capabilities for load balancing and high availability.

5.4.1.1 Oracle BPM Suite Component Characteristics

Figure 5-14 shows the components in BPM Suite, their relationship, and the technologies they use to communicate with each other.

BPMN is an abbreviation for Business Process Modeling Notation. The BPMN Service Engine is an extension of the existing BPEL Service Engine and it leverages the core infrastructure of the BPEL. The BPMN Service Engine leverages JPA/EclipseLink to store or recover the state of a process instance in the SOA Infrastructure dehydration store maintained by a database and to persist audit records that are created in the course of running a process. MDS APIs retrieve metadata information about the BPMN process model and other BPM project artifacts such as the Business Catalog.

BPM Composer

The BPM Composer is a standard J2EE web application. End users access it from a browser. The BPM Composer leverages the BPM infrastructure libraries. It uses the Project Metadata Layer to create and retrieve artifacts of a BPM project (such as a business rule or BPMN process) from MDS.

BPM Workspace

The BPM Workspace is a standard J2EE web application. End users access it from a browser. The BPM Workspace uses Oracle Business Process Interface (OBPI) APIs to retrieve information about BPMN process instances and user tasks for display in the worklist. Also, it leverages the cube persistence APIs for standard and custom dashboards.

BPM Infrastructure Libraries

The most important BPM Infrastructure Libraries are:

OBPI:

The Oracle Business Process Interface provides a way to access information about process instances, user tasks, and so on. It deploys as a library in WebLogic Server and exposes an EJB interface as well as Spring Beans for dependency injection. The OBPI leverages existing SOA Services such as Human Workflow Service, Identity Service, and Facade API. OBPI is the primary interface for clients that want to access the BPM Infrastructure.

PML:

The Project Metadata Layer is an internal API for managing BPM projects. It leverages MDS for storage, retrieval and labeling of BPM projects and is used by BPM Composer and the BPMN Service Engine

Analytics:

The Analytics library provides a way to manage Process Cubes. The library exposes a EJB3 (stateless) interface and uses JPA/EclipseLink for persistence. This library is used by BPM Workspace.

Note:

Because Oracle BPMN Service Engine is the core piece of BPM Suite, the following sections describe it in detail.

5.4.1.2 Oracle BPM Suite Component Interaction

Table 5-1 shows how the BPM Suite components interact and the tasks that can be performed by each component during the execution of a BPMN process.

After the heading row in Table 5-1, each row in the table is for a particular BPM Suite component. After the first column in Table 5-1, each column heading represents a point in time during the execution of the BPMN process; T1 is the first point in time, T2 is the next point in time, and so on. T8 is the last point in time for the process execution. The tasks that each component can perform are shown in the row for that component at the point in time that the task is performed during the BPMN process.

The following list provides additional details about the eight points in time (T1 through T8) during the BPMN process execution shown in Table 5-1:

T1. The example in Table 5-1 assumes that a BPM project is available either in BPM Studio or BPM Composer and is ready for deployment. As part of deploying a BPM project from BPM Studio or BPM Composer, an (SCA) archive is created and transferred to the standard SOA Composite Deployer Servlet running in the WebLogic Server. The composite deployer servlet stores the content of the BPM Project in MDS and notifies the appropriate service engines that a new BPM Project has been deployed. After that, the BPMN Service Engine is ready to receive requests for processing.

T2. There are two ways to initiate a BPMN process:

Use an Initiate Task:

If the user has modeled a Initiate Task in the BPMN process diagram to start the process instance, then after deployment the task would become visible in BPM Workspace and the process instance can be started from here.

Invoke the Composite Service using any Web Service (WS client) Using an Initiate Task. If the BPMN Process exposes a service interface, that service is exposed as a composite service and you can use any WS client to instantiate the process instance.

T3. The BPMN Service Engine starts running the process and continues to run it until the first User Task runs. For a User Task, the BPMN Service Engine creates a Human Task using OBPI. As a result, a new task is created in the SOA Infrastructure dehydration store.

T4. BPM Workspace uses OBPI to query new tasks for display in the Worklist. The user logged into BPM Workspace is supposed to work on the task. Eventually the task is approved or rejected and work on it has completed. BPM Workspace uses OBPI to complete the task.

T5. When the task is completed, that also completes the User Task Activity in the BPMN Process, and the process is able to continue.

T6. At any time during the execution of the process instance, a user might use BPM Workspace to monitor the task performance and workload. For this, BPM Workspace leverages the Cube Persistence API to query task performance data from the Process Cubes stored in the database.

T7. Eventually the BPMN process is completed.

T8. Typically, this last step return a result by means of a Message End Event to the client that invoked the process. When the process completes, a user may use BPM Workspace to monitor the process performance.

This displays the BPMN Service Engine Properties page shown in Figure 5-16. Additional parameters are available from the System MBean browser. Click More BPMN Configuration Properties... on the BPMN Service Engine Properties page to launch the BPMN MBean browser.

5.4.2 Oracle BPMN Service Engine High Availability

This section provides single instance and high availability information for Oracle BPMN service engine.

5.4.2.1 Oracle BPMN Service Engine Single Instance Characteristics

Service engines are containers that host the business logic of service components in a BPM project. Each service component, such as Oracle BPMN Service Engine, runs in its own service engine. A service engine plugs into the Oracle SOA Service Infrastructure.

Oracle BPMN service engine is the service engine running in SOA Service Infrastructure that enables the execution of BPMN processes.

A BPMN process provides the standard for assembling a business process using standardized activities, gateways and events in a well-defined process flow. The BPMN service engine provides functionality for the execution of potentially long-running BPMN process models. It leverages the core-infrastructure features of the Oracle BPEL Process Manager such as dehydration, dispatching, and service orchestration.

The state of process execution is saved in the SOA database through a persistence module based on Java Persistence Architecture (JPA). The auditing infrastructure of the BPMN service engine continuously audits the work being processed by the engine and stores audit records in the BPM database. Those audit records are used as a source for measurements and integration with Oracle Business Activity Monitoring (BAM).

5.4.2.1.2 Oracle BPMN Service Engine External Dependencies

SOA Service Infrastructure database for BPMN process state persistence

BPM database for persistence of analytics data

By default, the BPM database is collocated with the SOA Service Infrastructure database and requires no additional setup.

MDS repository for BPMN process metadata store

The components in Figure 5-17 must be available for the BPMN service engine to start and run properly.

Depending on the BPM project, the BPMN service engine might depend on the following additional components:

BAM adapter

User directory. For example, Oracle Internet Directory or another LDAP server that is configured to work with BPM.

5.4.2.1.3 Oracle BPMN Service Engine Startup and Shutdown Lifecycle

As shown in Figure 5-18, when the Oracle SOA Service Infrastructure application starts, it initializes the BPMN service engine and loads the composites from the MDS repository. If the composite contains any BPMN processes, it targets those individual components to the BPMN service engine. Once the process is loaded and its BPM specific metadata persisted in the database, the system is available to receive requests.

BPM project composites are loaded from the MDS repository by the SOA Service Infrastructure.

BPMN components are dispatched to the BPMN service engine to be loaded.

The BPM project metadata, such as organization data and audit/measurement metadata, is persisted in the infrastructure database.

Composite binding components are activated.

The BPMN engine services requests.

The shutdown signal is received by the SOA Service Infrastructure.

The SOA Service Infrastructure starts unloading composites.

Composite binding components are disabled.

BPMN components are dispatched to the BPMN engine to be unloaded.

The BPMN service engine shuts down.

5.4.2.1.4 Oracle BPMN Service Engine Log Files

BPMN service engine loggers are set to INFO by default. When trying to track down a message or instance, you may want to set the loggers to TRACE level to enable more output to the WebLogic Server log files.

The home page of the BPMN service engine comprises several sub pages. The Dashboard page is the default page. To recover BPMN service instances, select the Recovery tab, which opens the BPMN Service Engine Recovery page; see Figure 5-20.

The standard Java EE artifacts that Oracle BPMN components use are configured as part of Oracle WebLogic's domain in which SOA is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, persistent stores, and JMS modules across the WebLogic Server domain. At the same time, the WebLogic Server cluster is in charge of synchronizing the deployments and libraries used by the Oracle BPMN components. As explained in Section 5.4.1.4, "Oracle BPM Suite Configuration Artifacts," the Oracle BPMN components' configuration is stored in the SOA MDS database. Changes are applied once (per SOA server), but affect all Oracle BPMN instances in the same SOA domain.

5.4.3 Oracle Business Process Web Applications High Availability

This section provides single instance and high availability information for Oracle Business Process web applications.

Both BPM Workspace and BPM Composer are deployed separately from the BPMN Service Engine, but depend on it completely. If the BPMN Service Engine is stopped (soa-infra system), neither BPM Workspace nor BPM Composer will be able to access information and metadata about deployed projects. Given the dependency of soa-infra on the SOA MDS database, by extension the Business Process web applications depend on the MDS database to be available.

BPM Workspace and BPM Composer are standard J2EE applications. They are started when the WebLogic Server where BPM has been deployed is started. They can be controlled from the Administration Console and can be stopped with a forced shutdown or a graceful shutdown.

Both BPM Workspace and BPM Composer are stateless web applications. When the WebLogic Servers they reside on are deployed behind a load balancer or an HTTP Server, the front end devices route requests indistinctly to either node on which the applications are running. When a node failure occurs, requests are redirected to the other available WebLogic Server and work on the user interface can continue without interruptions.

5.4.3.2.2 Configuring Oracle Business Process Web Applications for High Availability

There are no specific configuration files for the web applications that reside locally on the file system for the servers. Some property files, like the Workspace property file, workspace.properties, are part of the OracleBPMWorkspace.ear file but do not include any instance-specific settings. They are deployed and available wherever Oracle BPM Workspace is deployed.

5.4.4 Oracle Business Process Analytics High Availability

This section provides single instance and high availability information for Oracle Business Process Analytics.

The Oracle BPM Suite has built-in analytical capabilities for support of business-friendly process dashboards and real-time monitoring of business processes using Oracle Business Activity Monitoring (BAM).

From a technical perspective, the Oracle BPM Suite Analytical Infrastructure is used for the support of process analytics as follows:

Audit Persistence

The BPMN service engine is continuously generating audit events comprised of activity runtime data. The data of those audit events is persisted in audit tables of the service engine dehydration store. The audit data is the source of all analytical data.

JMS Topic

To de-couple process execution from the preparation and publishing of analytical data, a JMS topic configured as part of the SOA Service Infrastructure is used.

Cube Action MDB

A message driven bean (MDB) is used to trigger aggregation and persistence of the analytical data to the BPM cube schema stored in the SOA Service Infrastructure database.

BAM Action MDB

A message driven bean (MDB) is used to publish analytical data towards the BAM adapter installed as part of the SOA Service Infrastructure.

Process Persistence

For the persistence of audit events and analytical data to the SOA Infrastructure database, the Oracle BPM Suite leverages the Java Persistence API (JPA) Infrastructure.

The persistence unit is configured to use the JTA data source jdbc/SOADataSource and the provider org.eclipse.persistence.jpa.PersistenceProvider.

5.4.4.1.2 Oracle Business Process Analytics External Dependencies

The Oracle Business Process Analytics libraries and components run as part of the BPMN Service Engine infrastructure. They rely heavily on the BPMN Service Engine artifacts (queues, stores, and JDBC resources) to function properly. As is the case for the service infrastructure, the SOA database must be available for the Analytics framework to work. Additionally, when feeding data to BAM, the required BAM system must be up and running to process the analytical information.

Because Oracle Business Process Analytics is a subsystem in charge of feeding information to other systems (mainly read-oriented information), no special considerations for failover are required. A number of JMS queues used by Oracle Business Process Analytics are configured (as part of the SOA system high availability setup) as uniform distributed destinations for optimum availability and load balancing. All BPM Cube data persisted uses the SOA JDBC services, which use multi datasources configured for Oracle RAC databases.

5.4.4.2.2 Configuring Oracle Business Process Analytics for High Availability

5.5.1 Oracle Mediator Single-Instance Characteristics

Oracle Mediator is a service engine within the Oracle SOA Service Infrastructure. Oracle Mediator provides the framework to mediate between various providers and consumers of services and events. The Oracle Mediator service engine runs in-place with the SOA Service Infrastructure Java EE application. The runtime state for execution started by asynchronous interactions or involving parallel routing rules is maintained in the SOA runtime database. For details about administering Oracle Mediator, see the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite.

Synchronous and Asynchronous Interactions

Oracle Mediator provides support for synchronous and asynchronous request-response interactions. In a synchronous interaction, the client making the request remains blocked, awaiting the response. In an asynchronous interaction, the client invokes the service but does not wait for the response. You can specify a time out period for an asynchronous interaction.

Sequential and Parallel Message Routing

Oracle Mediator can route messages to their destinations either sequentially, or in parallel.

In a sequential routing, processing of data takes place in one single transaction.

In a parallel routing scenario, one transaction is used for en-queueing information, and another one for de-queuing it.

5.5.1.1 Oracle Mediator Component Characteristics

If a composite contains an Oracle Mediator component, SOA Service Infrastructure targets the component to the Oracle Mediator engine for deployment. None of the services provided by the Oracle Mediator engine system are singletons, therefore, Oracle Mediator engines can run in full active-active mode. The processing of messages by the worker threads in Oracle Mediator is transactional and relies on Oracle WebLogic Server transaction control service. Configure the appropriate transaction stores as recommended by WebLogic Server guidelines to guarantee recovery across failures in the WebLogic Server container. Additionally, Oracle Mediator's engine does not contain any stateful web modules or stateful session beans, therefore you are not required to configure any sort of session replication when running Oracle Mediator in active-active mode. The state of work and work-to-be processed is maintained by Oracle Mediator in the database. Therefore, it is critical that Oracle Mediator's database be highly available. This requires configuring multi data sources for the SOA data source as described in Section 5.13.4, "Running Oracle Fusion Middleware Configuration Wizard on SOAHOST1 to Create the SOA Domain". For information about multi data source configuration with Oracle RAC and the MDS repository, see Section 4.1.3, "Using Multi Data Sources with Oracle RAC."

External Dependencies

Oracle Mediator depends on the following components:

SOA database for Mediator message and message state persistence

MDS repository for composite metadata store

Both components must be available for Oracle Mediator to start or run properly.

5.5.1.2 Oracle Mediator Startup and Shutdown Lifecycle

When the Oracle SOA Service Infrastructure application starts, it initializes the Oracle Mediator engine and loads the composites from the MDS repository. If the composite contains any Oracle Mediator components, it targets them to the Oracle Mediator engine. At runtime, Oracle Mediator routing rules can be invoked through an inbound binding component or by another service engine. Graceful shutdown of the Oracle Mediator engine is initiated by the SOA Service Infrastructure and involves sending signals to in-flight instances and unloading of loaded components.

5.5.1.3 Oracle Mediator Request Flow

The recovery semantics of Oracle Mediator after a server failure are based on client interaction type and routing rule type.

Oracle Mediator provides support for synchronous and asynchronous request-response interactions. The following describes the behavior based on interaction type:

Synchronous Interaction: For synchronous interactions, an error is thrown back to the client in the case of a server failure. It is the client's responsibility to handle the error message and take appropriate action such as retrying the request.

Asynchronous Interaction: There are two types of asynchronous invocations - one that starts a new routing rule execution and one that is a callback to an existing rule execution. In the case of a callback, the engine attempts to resolve the subscribing instance through a correlation id. If there is a failure in handling the callback, the client receives an error message and has to handle the error appropriately. The client invocation must be transactional in order to guarantee reliable handling of a callback in the case of a server failure.

Oracle Mediator can route messages either sequentially or in parallel. Only messages processed in parallel need manual recovery in the case of a server failure. The following describes the behavior based on routing rule type:

Sequential Routing Rule: For sequential routing rule, complete rule runs in a single transaction. In the case of server failure, Oracle Mediator relies on the underlying transaction manager for recovery.

Parallel Routing Rule: For parallel routing rule, Oracle Mediator uses a store-and-forward paradigm that involves two separate transactions - one transaction for persisting the message and a second one for running the routing rule. If there is a failure prior to the persistence, the client receives an error message and has to handle the error in the same way as in a sequential routing rule. If the persistence is successful, the client call returns and further processing is done outside the context of the client call. In the case of a server failure, Oracle Mediator will initiate recovery upon server restart after a configurable time interval specified by ContainerIdLeaseRefresh

To configure these parameters, go to the Oracle Enterprise Manager Fusion Middleware Control, and in the navigation tree, select SOA, soa-infra, (server_name). Right-click SOA Administration, and select Mediator Properties, or use the MBean Browser to go to the appropriate property.

Oracle Mediator is an embedded service engine that leverages the same high availability architecture as the Oracle SOA Service Infrastructure. Figure 5-5 describes a two-node Oracle SOA Service Infrastructure cluster running on two WebLogic Servers. Oracle Mediator is deployed as part of the Oracle SOA Service infrastructure composite application.

In a clustered configuration, Oracle Mediator can run in an active-active mode as there are no singleton services and all state is stored in the SOA runtime database.

Note:

In a single-node environment, Oracle Mediator performance is slow if there a high latency between the middle tier and database, for example, if the middle tier and database are in different subnets.

During the execution of a Parallel Routing Rule, Oracle Mediator obtains a logical lock on a batch of messages in the SOA runtime database. This lock contains a reference to the container ID that uniquely identifies the Oracle Mediator engine. The container ID is assigned at startup time. The user can configure the batch size using DeferredMaxRowsRetrieved. A smaller batch size ensures that there are a lower number of locked rows requiring recovery (as explained above) in the case of a server failure.

In case of unplanned outages, you must wait as much time as specified in the ContainerIdLeaseRefresh interval after restarting the server. This enables the server to complete the instances still in the running state.

In a multi-node cluster environment, if Oracle Mediator is used for an asynchronous message exchange pattern, there could be a possibility that the callback will not be handled by Oracle Mediator if it arrives before the request has completed. This could happen in a scenario where the request initiated by Oracle Mediator to the target service takes longer to complete, and callback from the target service arrives before the request completes.

Process Failure

Oracle Mediator is protected from all process failures by the WebLogic Server infrastructure. The following process failure considerations apply to Oracle Mediator:

If the managed servers fail, Node Manager attempts to restart them locally. If whole server migration is configured and repeated restarts fail, the WebLogic Server infrastructure attempts to perform server migration of the managed server to the other node in the cluster, if it is configured. Once the server on the other node is restarted, Oracle HTTP Server resumes routing any incoming requests to it. The migrated server reads the SOA database, resumes any pending processing, and resumes transactions from the transaction logs in shared storage.

The SOA Service Infrastructure application where the Oracle Mediator engine runs may be down due to failure in accessing resources, errors caused by the coherence infrastructure, or other issues unrelated to the status of the managed server where it is located. Therefore, monitor the Oracle SOA Service Infrastructure application to watch for errors caused by the application in the managed server logs. For the location of server logs, see Section 5.2.1.6, "Oracle SOA Service Infrastructure Log File Locations").

During recovery after a server failure, Oracle Mediator attempts to redeliver messages that were partially processed. If the automatic retry fails, Oracle Mediator enqueues the message to the error hospital for manual recovery.

Node Failure

In case of a node failure, server migration is triggered after the available server verifies the time stamp in the database leasing system. If the crashed server was the coherence cluster master, the available server become the new master, and the Oracle Mediator engine remains available for processing messages from binding components. After the time stamp for leasing is verified, the Node Manager in the remaining node attempts to migrate the VIP used by the failed managed server, and restarts it locally. This effectively results in the Oracle Mediator engine having two instances running in the same node.

Cluster-Wide Configuration Changes

The standard Java EE artifacts that Oracle Mediator engine uses are configured as part of Oracle WebLogic's domain in which SOA is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, persistent stores, and JMS modules across the WebLogic Server domain. At the same time, the WebLogic Server cluster is in charge of synchronizing the deployments and libraries used by the Oracle Mediator engine.

As explained in Section 5.5.1, "Oracle Mediator Single-Instance Characteristics," Oracle Mediator engine-specific configuration is stored in the SOA MDS database. Changes are applied once (per SOA server), but affect all Oracle Mediator instances in the same SOA domain. For example, if WLS_SOA1 and WLS_SOA2 are part of the same SOA cluster and the heartbeat period for Oracle Mediator in WLS_SOA1 is changed to 20, WLS_SOA2's heartbeat period is also set to 20.

5.5.2.1.1 Recovering Failed Mediator Instances

During recovery from a server failure, under certain circumstances, messages with in-flight routing rules may be placed in the error hospital for manual recovery. These messages can be recovered using Enterprise Manager as shown in Figure 5-24.

5.5.2.1.2 Oracle Mediator's Resequencing in a Cluster

Oracle Mediator's resequencer enables rearranging a stream of related (but not necessarily ordered) messages back into order. It sequences the incoming messages that arrive in a random order, and then sends them to the target services in an orderly manner.

The resequencer works with two central concepts: groups and sequence IDs. The sequence ID is an identifier part of the message, based on which messages are rearranged. Messages arriving for resequencing are split into groups, and the messages within a group are sequenced according to the sequence ID. The sequencing is based on the sequencing strategy selected (the Standard, FIFO and Best Effort strategies are available in the Oracle JDeveloper Composite Designer).

Message processing is based on assigning a group id to batches of messages in Oracle Mediator's queue system. The resequencer dequeue system uses an instance identifier (ID) to lock a group, then processes the group messages. The heartbeat infrastructure creates and maintains the instance ID in Oracle Mediator. There is one instance ID for every Oracle Mediator instance. The heartbeat infrastructure inserts the instance ID and current time into the table MEDIATOR_CONTAINERID_LEASE. The heartbeat infrastructure has one heartbeat thread. This thread periodically updates the time associated with the instance ID to announce its presence to other Oracle Mediator instances. The Oracle Mediator configuration parameter ContainerIdLeaseRefresh, specified in minutes (default: 1), is used for this purpose. The heartbeat thread also looks for instance IDs that have not been updated within a configurable time period. The Oracle Mediator configuration parameter ContainerIdLeaseTimeout, specified in minutes (default: 5), is used for this purpose. The thread releases the locks held by these instance IDs. After the groups are unlocked, the dequeue system of other Oracle Mediator instances can pick up the groups for processing.

The possible states for a group stored in the MEDIATOR_GROUP_STATUS table are:

Group Ready for processing - 0

Group Locked - 1

Group Processing Completed - 2

Error in processing group - 3

You can configure both the ContainerIdLeaseTimeout and ContainerIdLeaseRefresh parameters from the Oracle Enterprise Manager Fusion Middleware Control configuration screens for Oracle Mediator. These properties determine the behavior of a cluster of Oracle Mediator Instances configured for re-sequencing. Each group is processed by one single server until its lease expires (the instance ID is not updated). If a managed server goes down while it was processing a group, the other instances in the cluster will proceed to its ordering once the lease has expired (that is, once the unique ID assigned to the group has not been updated during a lease timeout period).

5.5.2.2 Troubleshooting Oracle Mediator High Availability

To debug Oracle Mediator failures, check the database tables to determine which container failed. To identify requests that were in progress when Oracle Mediator failed, find the rows that are still locked, and unlock them. You may also view the payload, as it is stored as a blob.

The poll interval for Oracle Mediator Instance Manager should be the same across all the nodes. The time stamp used is the database time stamp.

The possible states of messages in the database are:

Ready

Locked

Completed

Error

5.6 Oracle Human Workflow and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for configuring Oracle Human Workflow for high availability.

5.6.1 Oracle Human Workflow Single-Instance Characteristics

Oracle Human Workflow is a service engine running in the Oracle SOA Service Infrastructure that enables the execution of interactive human driven processes. A human workflow provides the human interaction support such as approve, reject, and reassign actions within a process or outside of any process. The Human Workflow service consists of a number of services that handle various aspects of human interaction with a business process.

All human task metadata is stored and managed in the Metadata Service (MDS) repository. The Human Workflow engine consists of a Service Engine running within the SOA Service Infrastructure and additional Java EE applications for DefaultToDoTaskFlow and Worklist applications. A human workflow leverages an internal JMS queue for notifications related to a human task.

5.6.1.1 Oracle Human Workflow Startup and Shutdown Lifecycle

The Human Workflow startup consists of two phases: loading of the Java EE applications, and initialization of the service engine. The Java EE applications are loaded as part of the application server startup. The service engine initialization and shutdown is controlled by SOA Service Infrastructure. Post initialization, composites that contain Human Workflow components are targeted to the Human Workflow service engine by SOA Service Infrastructure.

5.6.1.2 Oracle Human Workflow Request Processing

A human workflow can be initiated by an invocation from another SOA engine (such as Oracle BPEL PM). The message is routed to the engine by SOA Service Infrastructure and is persisted by the Workflow engine within its runtime schema. The message becomes available for human actions through the browser based UI after the client transaction that is associated with the invocation is committed. Each update on the message or the runtime state through a user action is a separate transaction.

As soon as Workflow commits its transaction, the control passes back to BPEL which almost instantaneously commits its transaction. Between this window, if the Oracle RAC instance goes down, on failover, the message is retried and can cause duplicate tasks.

To configure these parameters, go to the Oracle Enterprise Manager Fusion Middleware Control, and in the navigation tree, select SOA, soa-infra, (server_name). Right-click SOA Administration, Workflow Notification/Task Service Properties, or use the MBean Browser to go to the appropriate property.

5.6.1.3.1 Managing the URI of the Human Task Service Component Task Details Application

You can add or remove the URI of the task details application used in Oracle Human Workflow.

To manage the URI of the human task service component task details application:

Access this page through one of the following options:

From the SOA Infrastructure Menu...

From the SOA Folder in the Navigator...

Select Home.

Select the Deployed Composites tab.

In the Composite section, select a specific SOA composite application.

Under soa-infra, select a specific SOA composite application.

Select the human task service component in the Component Metrics table.

Click Administration.

The Administration page shows the URI for the task details application.

Figure 5-5 describes a two-node Oracle SOA Service Infrastructure cluster running on two WebLogic Servers. Oracle Human Workflow is deployed as part of the Oracle SOA Service infrastructure composite application.

Oracle Human Workflow's engine uses transactional EJBs for persistence and JMS queues for user notification. All state is stored in the database and the JMS queue, and there is no in-memory session state to be replicated for recovery. Therefore, Oracle Human Workflow's service engine and the associated Java EE applications are run in an active-active topology on a WebLogic cluster. In the case of a server or hardware failure, whole server migration must be configured for recovering pending transactions and JMS messages stored on the local queue. Notifications are not sent out until the server is restarted.

The FabricInstanceManager.persistCompositeInstanceBean API (or other InstanceManager APIs) does not have built-in retry logic. When there are transient failures caused by Oracle RAC failover, the WS invocation is rejected directly. This manual recovery of these rejected messages is done using Oracle Enterprise Manager Fusion Middleware Control.

To manually recover the rejected messages, in the Oracle Enterprise Manager Fusion Middleware Control, select SOA, soa-infra(soa_server1), <composite_name>, Faults and Rejected Messages, select the message, and choose Recover With Options. This causes the workflow request to be retried.

5.6.3 Troubleshooting Oracle Human Workflow High Availability

Human Workflow works like a standard Java EE application with a browser based client. Once a message is persisted to its runtime schema, it can be processed using the workflow UIs. If a server fails in the middle of the transaction, you must recover the server for successful recovery or rollback. The following logs are relevant for debugging errors associated with Oracle Human Workflow:

oracle.soa.services.common

oracle.soa.services.notification

oracle.soa.services.workflow

oracle.soa.services.workflow.common

oracle.soa.services.workflow.evidence

oracle.soa.services.workflow.metadata

oracle.soa.services.workflow.persistency

oracle.soa.services.workflow.query

oracle.soa.services.workflow.runtimeconfig

oracle.soa.services.workflow.soa

oracle.soa.services.workflow.task

oracle.soa.services.workflow.task.dispatch

oracle.soa.services.workflow.task.routing

5.7 Oracle B2B and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for configuring Oracle B2B for high availability.

Oracle B2B is a binding component within SOA Service Infrastructure. B2B metadata consists of trading partner details along with their supported documents and delivery channels which is stored within the MDS repository. The SOA composite has a reference to this metadata.

Oracle B2B receives messages from different channels. A listener thread logs the messages in a table in the SOA database and sends the corresponding events to a JMS queue. Event handlers listen for events to process the requests in the JMS queues.

Oracle B2B supports the following transport protocols for communication with business entities:

HTTP(S)

Oracle Advanced Queue

Email (SMTP 1.0, IMAP 1.0, POP3)

File

FTP and SFTP (SSH FTP)

TCP/IP

JMS

For HTTP protocol, Oracle B2B uses the system's HTTP listener. For FTP and Email, Oracle B2B uses the external FTP and email server configured in the system. Even in high availability mode, only one B2B instance polls for incoming message for FTP, File, and email protocols.

Note:

MLLP & TCP/IP protocols are not supported in a clustered environment.

5.7.1.1 Oracle B2B Component Characteristics

Oracle B2B runs in-place with SOA Service Infrastructure Java EE application. The Oracle B2B user interface is a web application that is deployed as a standalone war file on the same managed server as the SOA Service Infrastructure. Oracle B2B UI application is stateful and stores information in the HTTP Session.

Oracle B2B stores state information within JMS queues and the SOA run-time database. Whole server migration is required for automatic JMS message and transaction recovery after a server failure.

The SOA database and the MDS repository must be available for Oracle B2B to start and run properly.

5.7.1.2 Oracle B2B Startup and Shutdown Lifecycle

When the Oracle SOA Service Infrastructure application starts, it initializes Oracle B2B's engine. Oracle B2B metadata deployment should occur before composites are deployed to the Oracle SOA Service Infrastructure. Oracle B2B end-points are defined as channels which are started as part of the engine initialization. Based on the protocol, each configured channel has external dependencies, such as:

If FTP is used, the appropriate FTP server must be started.

If email is used, the appropriate email server must be started.

If HTTP/HTTPS is used, the HTTP server front ending the B2B system must be started

5.7.1.3 Oracle B2B Request Flow

In an Oracle B2B system all processing is done asynchronously. When a request arrives, the corresponding message is inserted in the SOA database. A notification is inserted in a JMS queue as a placeholder for the pending work. The event handler threads listen for events in these queues and process the work sequentially.

5.7.1.4 Oracle B2B Configuration Artifacts

You can enable and disable metrics for Oracle B2B's Server using Oracle Enterprise Manager Fusion Middleware Control. This is the only property directly exposed by Oracle Enterprise Manager Fusion Middleware Control. The configuration for Oracle B2B's Server is maintained in the SOA database and most of it is exposed by the Oracle B2B user interface application. For details about Oracle B2B configuration, see the Oracle Fusion Middleware User's Guide for Oracle B2B.

Figure 5-5 describes a two-node Oracle SOA Service Infrastructure cluster running on two WebLogic Servers. Oracle B2B is deployed as part of the Oracle SOA Service infrastructure composite application. The following high availability characteristics are specific to an Oracle B2B high availability deployment:

The Oracle B2B user interface application runs inside each one of Oracle WebLogic Server servers, and as part of the same cluster as the Oracle SOA Service Infrastructure application.

Oracle B2B's server maintains the partners, documents, and channels definitions in the SOA database using an Oracle RAC database.

5.7.2.1 Oracle B2B Protection from Failures and Expected Behavior

This section describes how an Oracle B2B high availability cluster deployment protects components from failure and the expected behavior if component failure occurs.

Oracle B2B UI Failure

The Oracle B2B user interface application maintains some navigation information in memory. When a failure occurs in one of the managed servers running the Oracle B2B user interface, users' requests are redirected to another active WebLogic Server running the application. Ongoing requests from Oracle HTTP Server time out according the time out setting in Oracle HTTP Server's configuration. Failover requires those users accessing the failed instance to log in again, since the application is not enabled for serializing the session information.

The Oracle B2B user interface in-memory data is non-transactional, and some steps may need to be revisited in the case of failover.

If a node failure occurs, or if the local Oracle WebLogic Server Node Manager reaches the maximum restart tries on the failed managed server, whole server migration is triggered after the available server verifies the time stamp in the database leasing system. The other Oracle B2B engine remains available for processing new messages from the different channels. At the same time, the remaining Oracle B2B server should resume the pending operations for singleton channels, such as FTP, email, or file, after the default timeout period is reached in the time stamp the failed node sets in the database (two minutes by default). For Oracle B2B's User Interface application, ongoing requests from Oracle HTTP Server time out according to Oracle HTTP Server configuration.

5.7.2.2 Oracle B2B Cluster-Wide Configuration Changes

The standard Java EE artifacts that the Oracle B2B engine uses are configured as part of the Oracle WebLogic domain in which the Oracle SOA Service Infrastructure is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, persistent stores, and JMS modules across the WebLogic Server domain. At the same time, the WebLogic Server cluster controls synchronization of deployments and libraries used by Oracle B2B's engine.

As explained in Section 5.7.1, "Oracle B2B Single-Instance Characteristics," all Oracle B2B server-specific configuration is maintained in the database, and configuration changes are applied to all the SOA Servers running in a WebLogic Server domain. Therefore, configuration properties, such as Payload Size, and Outbound Dispatcher Counts are applied cluster-wide, meaning they are used by all instances in the Oracle SOA cluster.

To set up File, FTP, or Email transports in a high availability environment, specify a unique name for each instance by using b2b.HAInstanceNameunique_instance_name. If you use #ServerName# for the value, Oracle B2B retrieves the WebLogic Server name as the HAInstanceName.

5.7.2.3 Oracle B2B Deployments in a Cluster

Using the command line utility for deploying, purging, or importing metadata in B2B may cause inconsistencies and errors in the B2B system. For B2B, deploy agreements and purge or import metadata ONLY from the GUI available in the B2B console, instead of using the command line utility.

5.7.2.4 Troubleshooting Oracle B2B Active-Active Configuration

5.7.2.4.1 Purge, Import, or Deployment of B2B Metadata

When performing purge, import, or deployment of B2B metadata in a cluster, error timing and load balancing may cause exceptions that are unlikely to happen if a retry of the operation is performed.

There is no clean up nor other additional steps required. If errors, such as [java] MDS-02202: Content of the metadata object appear for deployment or "postTransfer: MDS-00521: error while reading document... appear for purge or import, retry the operation.

5.7.2.4.2 Error While Retrieving Oracle B2B Document Definitions

Problem: Error happens when trying to retrieve a document definition XSD from Oracle B2B. B2B resides in a cluster and is accessed through a load balancer. B2B console report the following:

An error occured while loading the document definitions.
java.lang.IllegalArgumentException: Cluster address must be set when clustering
is enabled.

Solution: This occurs if you do not set the front end HTTP host and port for the Oracle WebLogic cluster where Oracle B2B resides. To eliminate this error, set the front end address for the SOA Cluster:

In the Administration Console, in the Change Center section, click Lock & Edit.

In the left pane, choose the Environment in the Domain Structure window and then choose Clusters. The Summary of Clusters page appears.

Select the WLS_SOA cluster.

Select HTTP.

Set the values for the following:

Frontend Host:soa.mycompany.com

Frontend HTTPS Port:443

Frontend HTTP Port:80

Click Save.

To activate the changes, click Activate Changes in the Change Center section of the Administration Console.

Restart the servers to make the Frontend Host directive in the cluster effective.

5.8 Oracle Web Services Manager and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for configuring Oracle WSM for high availability.

5.8.1 Oracle WSM Single-Instance Characteristics

Oracle Web Services Manager (Oracle WSM) provides a policy framework to manage and secure web services consistently across your organization. It provides capabilities to build, enforce, run and monitor web service policies including security, WSRM, MTOM and addressing policies. It typically gets deployed along with SOA Service Infrastructure.

Oracle WSM consists of the following components:

Policy Manager reads and writes security and management policies including predefined and custom policies from the MDS repository. It is typically deployed on the Oracle SOA Service Infrastructure managed servers. However, you can deploy Policy Manager on separate managed servers.

Policy Manager is a stateless Java EE application. It exposes its capabilities through stateless session beans. Policy Manager does not cache any data, (all accesses go to the MDS repository).

Agent is responsible for policy enforcement, execution and gathering of runtime statistics. The Oracle WSM Agent is available on all Oracle Fusion Middleware managed servers. It is configured on the same server as the application which it protects.

The Oracle WSM Agent is made up of a set of jar files, which are a part of underlying web service stack. It does not have any session state. The Agent maintains an in-memory policy cache, which is populated at the Agent startup time. It does not use any JTA or JMS.

The Oracle WSM Agent is made up of the following two pieces:

Policy Access Point (PAP) communicates with Policy Manager. The Agent communicates with the Policy Manager through EJB invocations over T3 (or T3s if SSL is enabled) protocol.

Policy Interceptor is generated when a web service is deployed and activated, or when a policy is attached to a web service using Enterprise Manager. If new web services are protected using Oracle WSM, an additional instance of the interceptor is generated for each new web service. Interceptor is responsible for policy enforcement.

Metadata Store Policies are stored in the MDS repository. It is typically backed by an Oracle database. For high availability purposes, Oracle recommends using an Oracle RAC database as the back end for MDS repository.

Enterprise Manager is used to configure Oracle WSM. It also displays different web services metrics gathered by Oracle WSM.

5.8.1.1 Oracle WSM Component Characteristics

The Oracle WSM Agent is a set of .jar files available on every Oracle Fusion Middleware managed server in a web services stack.

Policy Manager is contained in the wsm-pm.ear file. None of the services provided by Oracle WSM are singletons, therefore, it can run in full active-active mode. Oracle WSM services can be validated by http://SOAHOSTx:port/wsm-pm/validator. This validator displays Oracle WSM policies.

The Oracle WSM Agent and Oracle Enterprise Manager interact with Policy Manager using the EJB interfaces. The EJBs used in Oracle WSM are stateless and can be deployed in a clustered environment. Therefore, there is no requirement to enable state replication in the cluster.

The Oracle WSM Agent and Policy Manager need not be co-located. However, the Agent expects Policy Manager to be deployed on at least one node of the domain. The Oracle WSM Agent has capabilities to auto-discover Policy Managers deployed in the domain.

Neither the Oracle WSM Agent nor Policy Manager use JTA or JMS messaging. The MDS-based policy store also does not support JTA.

External Dependencies

Oracle WSM Policy Manager depends on the following components:

MDS repository for storing the policies

Oracle WSM Agent depends only on Oracle WSM Policy Manager.

Both components must be available for Oracle WSM to start and run properly.

5.8.1.2 Oracle WSM Startup and Shutdown Lifecycle

The following key Oracle WSM components are involved in the startup lifecycle for Oracle WSM:

Oracle WSM Policy Manager

Oracle WSM Agent

Policy Manager is a stateless application which does not perform any caching. There is no special application level startup sequence performed when the managed server where Policy Manager is deployed starts up. Policy Manager communicates with the MDS repository to retrieve policies. The MDS repository can be stored in an Oracle RAC database to provide MDS high availability.

When a managed server on which an Agent is configured comes up, the Agent connects to Policy Manager to get latest revision of policies. If it succeeds, the changes to the policies are downloaded and cached. Once the Agent is up and running, it periodically attempts a cache refresh at a configurable interval. The default time is every one minute.

Oracle WSM Agent communicates with the Policy Manager through EJB invocations over T3 (or T3s if SSL is enabled) protocol. If Policy Manager is deployed on different nodes and some of them have SSL enabled and others don't, Agent communicates only with the nodes with SSL connections.

If the Policy Manager to which the Oracle WSM Agent is connected becomes unavailable, the underlying infrastructure automatically connects to another Policy Manager instance running elsewhere in the cluster. This is achieved through Oracle WebLogic EJB clustering.

For high availability scenarios, if an Oracle WSM application is targeted to multiple nodes, it should be targeted to a cluster rather than to individual managed servers.

If a managed server has web services deployed which are protected by Oracle WSM, and the Oracle WSM Agent is not able to communicate with any of the Policy Managers at startup time, web service invocation fails.

5.8.1.3 Oracle WSM Request Flow

When a protected web service is accessed by a client application, the Oracle WSM Agent queries the policy cache and enforces the applicable policies. Based on the policies, the request is authenticated, encrypted, decrypted, authorized or logged. It does not connect to Policy Manager for any of these operations. Runtime availability of Policy Manager does not affect the functioning of the Oracle WSM Agent, unless there is a configuration change, such as new web services, which are protected by Oracle WSM, being deployed, or new policies attached to existing web services. If there is such a configuration change, then the Oracle WSM Agent must connect to Policy Manager to get the applicable policies. If it cannot connect after initial startup, it continues to operate based on the cached policies.

5.8.1.4 Oracle WSM Configuration Artifacts

Oracle WSM Agent configuration resides in policy-accessor-config.xml. This file is located in the DOMAIN_HOME/config/fmwconfig directory. With this configuration file you can specify:

Policy Manager URL (if configured)

Cache Refresh Interval

Clock skew, to allow for differences in system clock of the client and servers

These options are also available from Oracle Enterprise Manager Fusion Middleware Control and are specific to each Oracle WSM Agent installation.

Other configuration options at the container level, such as data sources for MDS repository location, and application targeting, are maintained as part of Oracle WebLogic Server Domain configuration, and are synchronized across a cluster of Oracle WebLogic Servers by Oracle WebLogic Server core infrastructure.

Figure 5-5 describes an Oracle SOA Service Infrastructure two-node cluster running on two WebLogic Servers. Oracle WSM is deployed as part of the Oracle SOA Service infrastructure composite application. The following high availability characteristics are specific to an Oracle WSM high availability deployment:

Oracle WSM Policy Manager and Oracle WSM Agents are deployed on WLS_SOA_INFRA. Oracle WSM Agents are available on any custom WLS cluster in case there is a need to protect any custom web services deployed on them.

Oracle WSM Policy Manager and Oracle WSM Agents run in WebLogic Server Infrastructure managed servers that are part of a WebLogic Server cluster. The WebLogic Server cluster synchronizes configuration for common artifacts of WebLogic Server used by Oracle SOA Service Infrastructure (JDBC).

All Oracle WSM Policy Manager instances in the cluster point to the same MDS repository.

The MDS repository where Oracle WSM policies are stored is configured in an Oracle RAC database to protect from database failure.

5.8.2.1 Oracle WSM Protection from Failures and Expected Behavior

Since the Oracle WSM Agent is deployed on the same managed server as the application is deployed, it will be available again as soon as the application becomes available due to server restart/migration. The following two sections describe the failover for Policy Manager.

Process Failure

Oracle WSM components are protected from process failures by the WebLogic Server infrastructure:

If the managed servers fail, Node Manager attempts to restart them locally. If whole server migration is configured and repeated restarts fail, the WebLogic Server infrastructure attempts to perform server migration of the managed server to another node in the cluster. Once the server on the other node is restarted, Oracle HTTP Server resumes routing any incoming requests to it. At startup time, Oracle WSM Agent and Policy Manager go through the startup lifecycle as described in Section 5.8.1.2, "Oracle WSM Startup and Shutdown Lifecycle".

If the managed server running Policy Manager is restarted or migrated, the failover is transparent to the agents connected to it. Policy Manager leverages underlying EJB clustering and the failover infrastructure of Oracle WebLogic Server.

Node Failure

If node failure occurs and whole server migration is not configured, the Oracle WSM Agent fails over transparently to the other Policy Manager instance.

If whole server migration is configured, server migration is triggered after the available server verifies the time stamp in the database leasing system. After the time stamp for leasing is verified, the Node Manager in the node that still remains available attempts to migrate the VIP used by the failed managed server, and restarts it locally. This effectively results in the Oracle WSM Policy Manager application having two instances running in the same node. Refer to the Section 3.9, "Whole Server Migration" for more details on the failover process. In this situation, the Oracle WSM Agents load balance against both Policy Manager instances running on the same node.

5.8.2.2 Oracle WSM Cluster-Wide Configuration Changes

The standard Java EE artifacts that Policy Manager uses are configured as part of the Oracle WebLogic domain in which the Oracle SOA Service Infrastructure is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts, such as data sources, across the WebLogic Server domain. At the same time, the WebLogic Server cluster controls synchronizing the deployments and libraries used by different Oracle WSM components.

As explained in the single-instance section, Oracle WSM instance specific aspects are configured individually per WebLogic Server domain directory (typically one per node) through the policy-accessor-config.xml file. This file is included in the.jar file that is created when using the Oracle WebLogic Server Pack utility, therefore it is propagated to other nodes when you run pack/unpack to set up high availability for the Oracle SOA Service Infrastructure. However, ongoing changes to policy-accessor-config.xml are not replicated automatically to the other nodes. This implies that for updates, you must modify each node individually, for example, if you updated the Policy Manager URL or Cache Refresh Interval. These modifications must be manually performed in all the domain directories across a high availability topology. You can edit the policy-accessor -config.xml file directly, or use Oracle Enterprise Manager Fusion Middleware Control to make the configuration change on each managed server.

Oracle WSM Agents are capable of automatically discovering the deployed Oracle WSM Policy Manager deployments. If you want to override the behavior and some point to a specific Policy Manager instance, you can do so by editing the policy-accessor-config.xml file.

5.8.2.3 Configuring the Java Object Cache for Oracle WSM

The Java Object Cache (JOC) should be configured among all the servers running Oracle WSM. This local cache is provided to increase the performance of Oracle WSM.

5.9.1 Oracle User Messaging Service Single-Instance Characteristics

Oracle User Messaging Service (Oracle UMS) enables two way communication between users and deployed applications. It has support for a variety of channels, such as email, IM, SMS, and text-to-voice messages. Oracle UMS is integrated with Oracle Fusion Middleware components such as Oracle BPEL PM and Oracle WebCenter Portal. It is typically deployed along with the Oracle SOA Service Infrastructure.

Oracle UMS comprises the following components:

UMS Server is a Java EE application that runs on Oracle WebLogic Server. The UMS Server orchestrates message flows between applications and users. The server routes outbound messages from a client application to the appropriate driver, and routes inbound messages to the correct client application. The server also maintains a repository of previously sent messages in a persistent store, and correlates delivery status information with previously sent messages.

UMS Drivers connect UMS to the messaging gateways, adapting content to the various protocols supported by UMS. Drivers can be deployed or undeployed independently of one another depending on the messaging channels available in a given installation.

UMS Drivers adapt sent and received content to and from external protocols, such as email, XMPP, and SMPP. UMS Drivers are also Java EE applications deployed to an Oracle WebLogic Server.

UMS Client applications implement the business logic of sending and receiving messages. Examples of a UMS client application include an Oracle SOA application that sends messages as one step of an Oracle BPEL PM workflow, or an Oracle WebCenter Portal: Spaces application that can send messages from a web interface.

UMS client applications have either a UMS-specific EJB module embedded in them, or interact as standard web service clients.

Figure 5-27 shows the services and dependencies of an Oracle UMS single-instance architecture.

JCA resource adapter (embedded within the EAR) in order to interface with external protocol gateways

Two MDBs, one for inbound, and one for outbound, that interface between the resource adapter and the JMS queues

Some UMS drivers that implement an HTTP-based protocol, such as VoiceXML, have a servlet module as well.

UMS client applications that use the EJB interface have a stateless session bean that provides the API used by client business logic, and an MDB to asynchronously receive inbound messages.

UMS depends heavily on JMS. JMS queues are also used to buffer content between clients and servers, and between servers and drivers.

Each typical messaging operation (sending an outbound message or receiving an inbound message) involves two JMS queues, one between the client and server, and one between the server and driver.

UMS messaging state is stored in the database and in persistent JMS queues.

External Dependencies

UMS depends on an external database repository to maintain message and configuration state. It shares this state between clustered instances. The UMS Server accesses the database using a Java EE JDBC data source provisioned at installation time. This data source is a non-XA data source. Therefore, UMS does not depend on the JTA capabilities of the underlying infrastructure.

UMS uses JMS to deliver messages among messaging applications. By default it is configured to use a file-based persistent JMS store, therefore it depends on the storage device where those files are located.

5.9.1.2 Oracle User Messaging Service Startup and Shutdown Lifecycle

As Java EE applications, all UMS components are started by Oracle WebLogic Server container.

UMS Server Startup

At server startup time, UMS Server initializes a TopLink session that creates a connection to the database repository. UMS Server then begins listening on web service endpoints. EJBs also become available for invocation of functionality such as sending messages or retrieving delivery status.

UMS Driver Startup

When a UMS driver is deployed to a cluster, it sends a registration message to the local UMS server. The UMS server records the registration information in the database, making it available to all UMS servers in the cluster. Once this occurs, any UMS server can route messages to any UMS driver in the cluster. This happens when new drivers are deployed, or when existing drivers are restarted following a configuration change.

At the time of server startup, UMS drivers typically establish a connection to the external gateway for which they are configured. Some of these are persistent socket-level connections, such as SMPP and XMPP. Some connections are established and torn down for each request, such as SMTP and IMAP connections. Drivers then make a remote EJB call to the UMS server to register themselves.

UMS Client Application Startup

When the managed server where the clients are deployed starts up, EJB clients typically make a remote EJB call to register themselves with the UMS server. Web service clients do not have explicit registration mechanisms.

5.9.1.3 Oracle User Messaging Service Request Flow

UMS clients make a series of short-lived requests to the UMS server. Web service clients make requests using SOAP or HTTP. EJB clients make remote EJB calls to the server for initialization and configuration management, in addition to interacting using JMS.

After initial startup and registration, subsequent UMS request flows can be categorized as follows:

Outbound messaging: For an outbound message, a client application wishes to send a message to a user. The client application either makes a web service request to the UMS Server, or enqueues a message in the UMS Server's sending JMS queue. The UMS Server receives the request, records the messaging state, and selects an appropriate driver. The UMS Server enqueues the message in a different outbound JMS queue. The corresponding driver, which is listening to the outbound queue, dequeues the message, converts it to the appropriate format for the given protocol, and delivers it to an external gateway (such as an SMTP server or IM gateway.)

Inbound messaging: For an inbound message, the end user wishes to send a message back to the client application. The user sends a message from a device, such as an IM client or phone. The message passes from the external gateway to the driver. The driver adapts the message to the UMS format, and enqueues it in a JMS queue. The server dequeues the message from the JMS queue, records the message state, and enqueues the message in the appropriate application's inbound JMS queue. The client application then dequeues the message and processes it as needed.

5.9.1.4 Oracle User Messaging Service Configuration Artifacts

The UMS Server and drivers each have an XML configuration file that is the artifact of a configuration Mbean, implemented using Oracle Fusion Middleware JMX Framework. You can perform configuration changes using Oracle Enterprise Manager Fusion Middleware Control.

All UMS components use standard Java EE deployment descriptors and Oracle WebLogic proprietary descriptors for configuring Java EE components. You can configure these descriptors using standard Oracle WebLogic tools, such as the Administration Console or WLST. For more information, see UMS administration topics in Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite.

If you are using Oracle SOA Suite in a clustered environment, any configuration property changes you make in Oracle Enterprise Manager on one node must be made on all nodes. Configuration properties are set in Oracle Enterprise Manager through the following options of the SOA Infrastructure menu:

All UMS components are deployed to Oracle WebLogic Service Infrastructure managed servers that are part of an Oracle WebLogic Server cluster. An Oracle WebLogic Server cluster synchronizes configuration for common artifacts of Oracle WebLogic Server used by UMS, such as JDBC data sources.

UMS relies on JMS distributed destinations for load balancing and availability across cluster nodes. It also relies on the JMS connection factory's capability to failover to a different JMS server if a failure occurs.

The user messaging preferences user interface does not require session stickiness. It remains available through the use of a basic load balancing. There are no sticky session routing requirements, as all session state is persisted in the database and shared across the clustered instances.

Oracle UMS is typically deployed on the same managed server as the Oracle SOA Service Infrastructure. The Oracle SOA Service Infrastructure is protected from process and node failures using Oracle WebLogic Server whole server migration. Whole server migration also provides failover capabilities for JMS usage.

This section describes specific considerations for process failure in an Oracle User Messaging Service high availability configuration:

Node Manager attempts to restart the managed servers locally if a failure occurs.

If whole server migration is configured for the Oracle WebLogic Server managed server to which Oracle UMS components are deployed, and the restart count threshold is exceeded, Oracle WebLogic Server infrastructure attempts to perform a server migration of the managed server to another node in the cluster. After the server migration completes successfully, at the startup time, UMS Server and Drivers go through the startup cycle as previously described, including driver registration. Driver registration is an independent operation and does not have any affect on other available instances. See Chapter 3, "High Availability for WebLogic Server" for more information about server migration.

At restart (on the same or different node), the UMS JMS server in the managed server starts producing and consuming messages from its JMS store.

If a managed server running UMS Server and Drivers is restarted or migrated, the failover is transparent to the connected UMS Clients. The failover is transparent because UMS components are stateless. Once the server's restart is finished, the web server starts routing requests to it for web service clients. Similarly, EJB clients become aware of the server availability and start routing requests to it. This is made possible by Oracle WebLogic clustering infrastructure.

Before attempting a message delivery, UMS first persists the message to the database. If an external Messaging Gateway becomes unavailable, the corresponding UMS driver periodically attempts to reconnect to the gateway and deliver any undelivered messages persisted in the database. Alternatively, if the messages are not delivered, administrators can manually resend the messages using the UMS server's Message Status Page in Oracle Fusion Middleware Enterprise Manager.

UMS configuration is file-based, standard Java EE deployment descriptors and JMX configuration Mbeans, using a standard JMX framework. Changes are propagated using standard Oracle WebLogic Server Mbean server mechanisms. There are no cluster-wide configuration capabilities. As a result, configuration changes must be repeated on every member of a cluster

UMS uses standard Java EE artifacts, configured as part of Oracle WebLogic's domain in which Oracle UMS is installed. Oracle WebLogic Server clusters provide automatic configuration synchronization for artifacts, such as data sources, across Oracle WebLogic Server domain. At the same time, Oracle WebLogic Server cluster controls synchronization of deployments and libraries used by different Oracle UMS components.

5.10 Oracle JCA Adapters and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for configuring Oracle JCA Adapters for high availability.

5.10.1 Oracle JCA Adapters Single-Instance Characteristics

Oracle JCA Adapters are JCA binding components that enable the Service Infrastructure to communicate to endpoints using different protocols. Oracle JCA Adapters are deployed as a JCA resource (RAR) and are not part of the Oracle SOA Service Infrastructure.

The run-time component of Oracle JCA Adapters is the J2CA 1.5 resource adapter for the specific back-end application. Oracle JCA Adapters are deployed in J2CA container of Oracle WebLogic Server. Oracle Fusion Middleware integrates with these J2CA 1.5 adapters through the JCA Binding Component, which converts web service messages into J2CA interactions and back. Adapters can, therefore, fully leverage the scalability and high availability of the underlying Oracle WebLogic Server platform.

5.10.1.1 Oracle JCA Adapters Component Lifecycle

The lifecycle of an adapter is controlled by the Oracle SOA Service Infrastructure. There are three high-level steps in the initialization of an Oracle JCA Adapter:

Deployment: Oracle JCA Adapters are deployed during installation as J2CA 1.5 resource adapters (RAR files) within the same Oracle WebLogic Server container as Oracle Fusion Middleware. The physical deployment of adapters involves using the RAR file and registering the adapters as connectors with the underlying Oracle WebLogic Server, or the middle tier platform. The RAR file contains the ra.xml file, which contains declarative information about the contract between Oracle WebLogic Server and the resource adapter. In addition, the RAR file contains weblogic-ra.xml which is the deployment descriptor file containing deployment-specific information about the resource adapter.

Loading: Loading refers to the process of creating in-memory resources and creating connections to the configured end-point. Although Oracle JCA Adapters are physically deployed as J2CA 1.5 resource adapters, their logical deployment involves creating the Connection Factory entries for the J2CA 1.5 resource adapter by editing the weblogic-ra.xml file. This file is Oracle WebLogic Server-specific deployment descriptor for a resource adapter. It contains configurations for deploying resource adapters to Oracle WebLogic Server, which includes the back-end application connection information as specified in the deployment descriptor of the resource adapter, Java Naming and Directory Interface (JNDI) name to be used, connection pooling parameters, resource principal mapping mechanism, and configurations.

Activation: Activation refers to initiation of a JCA binding component (Service and Reference) within a Composite. Listeners are started for the endpoint referenced by the Adapter configuration within the Composite.

Property Changes During Oracle JCA Adapters Runtime

Oracle JCA Adapters expose the underlying back-end operation-specific properties as header properties and enable the manipulation of these elements within a business process.The underlying properties are as follows:

interactionspec or activationspec Properties - Require the adapter endpoint to be recycled (re-activated).

Endpoint Properties - Changes to these properties are notified to the adapter without requiring the endpoint to restart.

For a synchronous process, the global transaction initiated by the adapter spans message delivery and composite execution.

For an asynchronous service entry point, a transactional adapter initiates a global JTA transaction before sending an inbound message to a composite. When control returns to the adapter, it commits the JTA transaction, running the following set of actions as an atomic unit of work:

Commit the removal of the message from the inbound adapter endpoint, for example, table and queue.

Commit the execution of the composite instance.

If anything fails during this process, both of these actions are rolled back based on XA guarantees.

Outbound Transactions

For transactional adapters, outbound JCA interactions (the invoke activities) are scoped with the global JTA transaction of the Composite instance. This means that all composite activities, including Oracle JCA adapter invocations, are part of a global transaction, and as such all activities are either committed or rolled back if an error occurs. Therefore, one can guarantee exactly-once message delivery when both inbound and outbound adapters are transactional and the connection factories have been configured to support XA global transactions.

Nontransactional

The Oracle File Adapter picks up a file from an inbound directory, processes the file, and sends the processed file to an output directory. However, during this process if a failover occurs in the Oracle RAC backend or in an SOA managed server, then the file is processed twice because of the nontransactional nature of Oracle File Adapter. As a result, there can be duplicate files in the output directory.

5.10.1.3 Oracle JCA Adapters - Rejected Message Handling

The messages that error out before being posted to the Oracle SOA Service Infrastructure mesh are referred to as rejected messages. For example, Oracle File Adapter picks a file having data in CSV format and tries to translate it to the XML format using NXSD. Now, if there is any errors in the translation, the message is rejected and is not posted to the target composite.

All rejected messages are stored in the database with payload. There are two destinations for rejected messages:

Adapter Rejection Table - Messages are stored in the Adapter Rejection table when there is a binding error related to a corrupt message.

In case the database hosting the Rejected Message Table is offline, rejected messages are temporarily stored in the local file system (as a backup) and eventually loaded back into the database when it is detected as being online again.

Composite Instance Failure Table - This table is part of instance tracking, populated by each component detecting a failure.

In the case of errors being thrown by the SOA Service Infrastructure layer, the Adapter framework behavior depends on whether or not the error is marked as retriable. If the error is not retriable, the message is treated as a rejected message and persisted to the Rejected Message Table.

Configuring Rejection Handlers

Rejection handlers are defined and configured by using fault policies as described in the following steps:

Define a fault policy for the rejected messages in the fault-policies.xml file, which is stored along with composite.xml in the Oracle JDeveloper project directory.

Associate the fault policy with a service endpoint of the composite in the fault-bindings.xml file.

Retry frequency and retry interval are both configurable. Service engines can mark an invocation as retriable. In a clustered environment, if rollback occurs, it may be picked up by another instance. If another node picks up the message, it generates a unique ID. The number of retries are configured within the composite, under the binding.jca element. For more information, see the Oracle Fusion Middleware User's Guide for Technology Adapters.

System-Defined Rejected Message Handlers

The following system-defined error handlers can be configured in fault policies:

Web Service Handler - A predefined WSDL interface must be implemented by the target service

Custom Java Handler - A predefined Java interface must be implemented by the target class.

JMS Queue - The rejected message is enqueued to a queue as a JMS message with the appropriate context and payload.

File - The rejected message is stored in the file system with the proper context and payload.

Payload Persistence

For resubmitting rejected messages, payload persistence is imperative. Payloads are stored in the database and be viewed through Oracle Enterprise Manager Fusion Middleware Control. Error handlers invoked during automatic error handling also receive the payload during their invocation. Error payload persistence in the database is enabled by default.

This section describes Oracle JCA Adapters high availability considerations for configuring Oracle JCA Adapters to run on an Oracle SOA cluster.

5.10.2.1 Oracle JCA Adapters High Availability Error Handling

This section describes Oracle JCA High Availability error handling.

Connection Errors

Oracle JCA Adapters and Oracle Adapters for Oracle Applications handle connection errors for the following interactions:

Outbound Interaction: Oracle JCA Adapters and Oracle Adapter for Oracle Applications raise the oracle.tip.adapter.api.exception.PCRetriableResourceException exception for transient connection errors, which are recoverable connection errors. For example, a database listener may not have started, resulting in connection errors. You can define the maximum number of attempts to reconnect made using the fault-policy.xml file. The parameters for attempts to reconnect can be specified using this file. After the configured number of retries is reached, the fabricInvocationException exception is thrown.

Fault handling and retry properties are defined as part of the binding level configuration. When the binding level retry has expired, the fault is handled based on policies specified within the fault-policy.xml. For more information on binding properties, see the Oracle Fusion Middleware User's Guide for Technology Adapters.

Inbound Interaction: Oracle JCA Adapters, Oracle Adapter for Oracle Applications, and legacy adapters support a poll model for connecting to the back-end application for receiving events. In case of unrecoverable connection failures, the adapters recycle old connections, and send out alerts or notifications to the Oracle SOA Service Infrastructure. The inbound interaction connection errors are written to a log, and can be viewed through the Administration Console.

5.10.2.2 Oracle File and FTP Adapters High Availability

The Oracle File and FTP Adapters can be configured for high availability using an active-active topology. A cluster-based high availability configuration is supported for both inbound and outbound operations. File and FTP Adapters do not support XA and hence can only guarantee at-least-once delivery of messages. Thus, it is possible to have duplicate messages after recovery from a server failure.

Prerequisites for High Availability

The following list describes prerequisites for Oracle File and FTP Adapters high availability:

In a clustered configuration, inbound adapters across managed servers must point to the same physical directory, a directory on a shared drive.

Connection-factories must specify the same shared folder as the control directory, and their names must match. For example, if the deployment descriptor for one connection-factory has /shared/control_dir as the value for controlDir, the other deployment descriptor must also have the same value.

The Oracle File and FTP Adapters must ensure that only one node processes a particular file in a distributed topology. You can use the database table as a coordinator to ensure that Oracle File and FTP Adapters are highly available for inbound operations. See the Oracle Fusion Middleware User's Guide for Technology Adapters for details on configuring a database table as a coordinator.

The Oracle File and FTP Adapters must ensure that if multiple references write to the same directory, then these do not overwrite each other. The following locking capabilities can be used to make Oracle File and FTP Adapters highly available for outbound operations:

A cluster-based high availability configuration is supported for both inbound and outbound operations. However, consider the following:

Inbound Operations - Oracle File and FTP Adapters must ensure that only one node processes a particular file in a distributed topology.

Outbound Operations - Oracle File and FTP Adapters must ensure that if multiple references write to the same directory, these do not overwrite each other.

Database-based mutexes are used as coordinators to ensure these behaviors in a clustered topology.

Configuring a Database Mutex

Configure a database table as a coordinator by modifying Oracle File and FTP Adapter deployment descriptor for the connection-instance corresponding to eis/HAFileAdapter from the Administration Console:

Click FileAdapter under Summary of Deployments on the Administration Console.

Click the Outbound Connection Pools tab, and expand javax.resource.cci.ConnectionFactory to see the configured connection factories.

Click eis/HAFileAdapter.

Update the connection factory properties.

Update the Adapter configuration to use the connection factory as shown in the following example:

The location attribute is set to eis/HAFileAdapter for the connection factory.

The new parameters in connection factory for Oracle File and FTP Adapters are as follows:

controlDir - Set this parameter to the directory structure where you want the control files to be stored. Set it to a shared location if multiple Oracle WebLogic Server instances are running in a cluster.

inboundDataSource - Set this parameter to jdbc/SOADataSource. This is the data source where the schemas corresponding to high availability are pre-created. The pre-created schemas are located under MW_HOME/ASreleaseSOA/rcu/integration/soainfra/sql/createschema_soainfra_oracle.sql. To create the schemas elsewhere, use this script. Set the inboundDataSource property accordingly if you choose a different schema.

outboundDataSource - Set this parameter to jdbc/SOADataSource. This is the data source where the schemas corresponding to high availability are pre-created. See inboundDataSource for schema location.

outboundLockTypeForWrite - Set this parameter to oracle if you are using Oracle database. By default Oracle File and FTP Adapters use an in-memory mutex to lock outbound write operations. Choose one of the following values for synchronizing write operations:

memory - Oracle File and FTP Adapters use an in-memory mutex to synchronize access to the file system.

oracle - The adapter uses Oracle database sequence.

db - The adapter uses a pre-created database table (FILEADAPTER_MUTEX) as the locking mechanism. Use this option only if you are using a schema other than the Oracle database schema.

user-defined - The adapter uses a user-defined mutex. In order to configure the user-defined mutex, implement the mutex interface: oracle.tip.adapter.file.Mutex and then configure a new binding-property with the name oracle.tip.adapter.file.mutex and value as the fully qualified class name for the mutex for the outbound reference.

Note:

For large payloads, increase transaction timeout for the SOA DataSource by adding the following:

<xa-set-transaction-timeout>true</xa-set-transaction-timeout>

<xa-transaction-timeout>1000</xa-transaction-timeout>

Increase global transaction timeouts if a database is used as the coordinator.

5.10.2.3 Oracle Database Adapters High Availability

The Oracle Database Adapter supports high availability in an active-active setup. In an active-active setup, distributed polling techniques can be used for inbound Database Adapters to ensure that the same data is not retrieved more than once. For more information on best practices for distributed polling, see the "Oracle JCA Adapter for Database" chapter in Oracle Fusion Middleware User's Guide for Technology Adapters. Similar to other adapters, an Oracle Database Adapter can also be configured for singleton behavior within an active-passive setup. This enables a high performance multi-threaded inbound Oracle Database Adapter instance running in an active-passive setup, to follow a fan out pattern and invoke multiple composite instances across a cluster. The Oracle Database Adapter also supports the high availability feature when there is a database failure or restart. The DB adapter picks up again without any message loss.

The Oracle Database Adapter can survive a database down scenario and pick up again without any message loss. For avoiding data loss, the data source needs to be XA enabled and configured for Oracle RAC (multi data source). For information on configuring a datasource for high availability, see Appendix B, "Recommended Multi Data Sources."

Transaction control is very relevant for consistency and reliability purposes when using the Oracle JMS Adapter. The Oracle JMS Adapter supports global transactions based on the JCA 1.5 XA contracts that leverage the underlying application server transaction manager. In context of the JMS Adapter, a transaction enables an application to coordinate a group of messages for production and consumption, treating messages sent or received as a single unit. When an application commits a transaction, all messages it received within the transaction are removed by the JMS provider. The messages it sent within the transaction are delivered as one unit to all JMS consumers. If the application rolls back the transaction, then the messages it received within the transaction are returned to the messaging system and the messages it sent are discarded.

When using the WLS JMS provider, the Adapter supports XA transactions by using an XA-enabled connection factory (weblogic.jms.XAConnectionFactory, which is the default). For this provider, the high availability behavior is tied to the configuration of the queues and topics that the adapter uses and to the JMS Infrastructure that Oracle Weblogic Server provides. There are two main aspects that affect the behavior of the JMS Adapter when using Oracle WLS JMS as the provider:

Message redelivery

Use of WebLogic Server distributed destinations

The next two sections describe these topics. For information on configuring high availability for JMS Adapter, see Enabling High Availability for Oracle JMS Adapter in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

5.10.2.4.1 Message Redelivery

Message redelivery plays an important role in a highly available configuration. In WLS JMS, you can delay the redelivery of messages when a temporary, external condition prevents an application from properly handling a message. This enables an application to temporarily inhibit the receipt of "poison" messages that it cannot currently handle. When a message is rolled back or recovered, the redelivery delay is the amount of time a message is put aside before an attempt is made to redeliver the message. If JMS immediately redelivers the message, the error condition may not be resolved and the application may still not be able to handle the message. However, if an application is configured for a redelivery delay, then when it rolls back or recovers a message, the message is set aside until the redelivery delay has passed, at which point the messages are made available for redelivery. All messages consumed and subsequently rolled back or recovered by a session receive the redelivery delay for that session at the time of rollback or recovery. Messages consumed by multiple sessions as part of a single user transaction will receive different redelivery delays as a function of the session that consumed the individual messages. Messages that are left unacknowledged or uncommitted by a client, either intentionally or as a result of a failure, are not assigned a redelivery delay.

A session in JMS context inherits the redelivery delay from its connection factory when the session is created. The RedeliveryDelay attribute of a connection factory is configured using the Administration Console (Configuration->Default Delivery screen). WebLogic JMS defines two default connection factories, which you can look up using the following JNDI names:

weblogic.jms.ConnectionFactory

weblogic.jms.XAConnectionFactory

The JMS Adapter uses weblogic.jms.XAConnectionFactory by default when configured for the WLS JMS Provider, therefore using the same redelivery settings as the XAConnectionFactory. For details on configuring connection factories for different adapters, refer to "Oracle JMS Adapters Use Cases" in Oracle Fusion Middleware User's Guide for Technology Adapters.

5.10.2.4.2 Use of WebLogic Server Distributed Destinations

Previous releases of Oracle JMS Adapter could not guarantee exactly-once message delivery when using some types of Oracle WebLogic Uniform Distributed Destinations. Oracle Fusion Middleware 11gR1 includes several enhancements in the WLS JMS layer that have been consumed by the JMS Adapter to better support Uniform Distributed Destinations. Shared ClientID, Partitioned Distributed Topics, and Shared Subscriptions are the main enhancements. These enhancements facilitate scalability, high availability, and consistent message delivery in different scenarios:

Shared ClientID: Enables durable subscribers to consume from distributed destinations using the same ClientID. The JMS standard requires that topic applications that consume from durable subscriptions must specify a ClientID and that the same ClientID should not be used by more than one running connection at any given time. This feature enables "overriding" this limitation.

Partitioned Distributed Topics: A partitioned distributed topic is a blend of uniform distributed queue and uniform distributed topic behavior. It enables messages to be load balanced among distributed topic members as in a distributed queue, and then allows for multiple receivers of the message (subscribers) on the single distributed topic member. One message published to a partitioned distributed topic land on only one of its members, not all of them as in previous implementations.

Shared Subscription: Multiple subscribers can optionally share the same subscription, durable or non-durable, by setting the connection's SubscriptionSharingPolicy to SHARABLE. When you use this feature with a partitioned topic and an UNRESTRICTED connection client-id policy, it guarantees that, among all subscribers who share the same distributed subscription, only one subscriber receives a particular message sent to the topic. All subscribers that share the same distributed subscription will together process each of the messages that are published to the topic.

Also, the WLS JMS infrastructure provides notification helpers in 11g that allow for notifications of member runtime availabilities of specific distributed destinations.From the JMS Adapter perspective, these JMS features have the following impact when using distributed queues:

Inbound - Distributed Queues

The JMS Adapter automatically registers a notification listener with WLS JMS on endpoint activation. If an "unavailable" notification is received, the corresponding poller threads stop and the necessary cleanup is done. Similarly, if an "available" notification is received, the poller thread starts and the adapter begins consumption from the new member.

Note:

Oracle BAM's Enterprise Message Source (EMS) cannot be configured with distributed queues fed by producers load balancing to the UDD. UDD's behavior pins each consumer to one single physical destination, and this could cause missed messages on the EMS side. As an alternative, configure EMS with the standard destination cluster wide JNDI name, but force producers to pin to one single physical JMS queue (by using that particular server host rather than using a cluster-wide JNDI name).

Outbound - Distributed Queues

There is no change in the JMS Adapter behavior when producing a message to a distributed queue. Available/Unavailable/Failure notification has no impact on the working of outbound adapter reference since messages are produced by creating a MessageProducer for the distributed destination and not for a specific member.

These JMS features have the following impact when using distributed topics:

Inbound - Distributed Topics

The JMS Adapter automatically registers a notification listener with WLS JMS on endpoint activation. If an "unavailable" notification is received, the corresponding poller threads are stopped and the necessary cleanup is done. Similarly, if an "available" notification is received, the poller thread is started and the adapter begins consumption from the new member. The durable subscription is maintained in a similar fashion as in a non-distributed topic scenario. The adapter will unregister the notification listener on endpoint deactivation. Any message arriving at a distributed topic will be processed based on the various settings used and the type of distributed destination in use, as detailed below:

One copy of message per application: Each message is processed exactly once (no duplicate processing). The use of partitioned distributed topics and configuring the JMS adapter to use unrestricted clientid and shared subscription policy (these are the default settings) is recommended. In this scenario, the client id and subscription name are the same for everyone, and each adapter instance creates subscriptions on every member. The name is unique and immutable across server restarts. The specific clientid configuration in the connection instance (weblogic-ra.xml for a local cluster) for the adapter is required:

When using replicated distributed topics, it is also possible to force one copy of message per application. For this, it will be required to configure JMS adapter to use unrestricted clientid and shared subscription policy (the default settings), but in addition, it will require specifying the following message selector "(NOT JMS_WL_DDForwarded)" when defining an activation spec. Note that this type of configuration is highly inefficient and not recommended, because it distributes messages to all consumers but then does the rejection on the consumer end.

One message per adapter endpoint instance in a cluster: In this scenario either the clientid or subscription name are unique for each adapter instance. Also, the unique part of the name should be immutable across server restarts. This type of configuration is possible using both Replicated and Partitioned Topics.

For Partitioned Distributed Topics, configure the JMS adapter to use unrestricted clientid and a shared subscription policy (that is, set "SubscriptionSharingPolicy" property to SHARABLE), which are the default settings. Also, to ensure subscription name uniqueness, set the TopicMessageDistributionAll property to true (default value is false). This property can be defined by setting the "FactoryProperties" property of the connection instance in weblogic-ra.xml. An example usage (snippet of connection instance from weblogic-ra.xml is:

For Replicated Distributed Topics, configure the JMS adapter to use unrestricted clientid and shared subscription policy as well (that is, set "SubscriptionSharingPolicy" property to SHARABLE), which are the default settings. As with the Partitioned Distributed Topic case, to achieve subscription name uniqueness, the adapter requires that the property TopicMessageDistributionAll be set to true (the default value is false). The property can be defined by setting the "FactoryProperties" property of the connection instance in weblogic-ra.xml. An example usage (snippet of connection instance from weblogic-ra.xml for a local cluster) is shown above.

For Replicated Distributed Topics to guarantee one copy per adapter endpoint, the message selector "NOT JMS_WL_DDForwarded" needs to be used when defining an activation spec, as shown in the following.jca file example:

Note, however, that this configuration is highly inefficient and not recommended; it distributes messages to all consumers but does the rejection on the consumer end. For better performance, use Partitioned Distributed Topics.

Outbound - Distributed Topics

There is no change to adapter behavior when producing message to a distributed topic. Available/Unavailable/Failure notification does not affect the working of outbound adapter reference. The message is produced by creating a MessageProducer for the Distributed Destination and not for a specific member.

Note:

Oracle BAM does not support durable, distributed topics but it does support non-durable (transient) distributed topics.

5.10.2.5 Oracle JCA Adapters Log File Locations

You can view the logs for Oracle JCA Adapters as follows:

Oracle JCA Adapters and Oracle Adapter for Oracle Applications: For both outbound and inbound interactions, the log files are redirected to the soa-diagnostic.log file.The log files for Oracle SOA Suite that is deployed to the server-soa managed server are located in:

Packaged-application adapters: These adapters do not implement the LogManager interface because it is not part of the J2CA 1.5 standard. Therefore, for OPMN-managed components the log outputs are redirected to

ORACLE_INSTANCE\diagnostics\logs\component_type\component_name

For outbound interactions, the logs are directed to the same location. On the other hand, for inbound interactions, logs are redirected to soa-diagnostic.log.

Legacy adapters: In addition to the J2CA resource adapter, legacy adapters consists of Oracle Connect, which consists of native adapters for communicating with the mainframe application and data stores. Oracle Connect logs can be viewed using Oracle Studio, which is the mainframe adapter design-time tool and Oracle Connect management tool. Oracle Connect generates various types of logs, such as the daemon log, workspace log, and server process log.

5.11 Oracle Business Activity Monitoring and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for configuring Oracle BAM for high availability.

Oracle BAM provides the tools for monitoring business services and processes in the enterprise. It enables correlating of market indicators to the actual business process and to changing business processes quickly or taking corrective actions if the business environment changes. Oracle Business Activity Monitoring (Oracle BAM) provides the necessary tools and runtime services for creating dashboards that display real-time data inflow and define rules to send alerts under specified conditions.

Figure 5-30 shows the main services and dependencies that characterize an Oracle BAM instance.

Oracle BAM Server is a set of runtime components that handle incoming data from different data sources. Oracle BAM components are also used to evaluate conditions for sending alerts to users and triggering the required actions configured for these alerts. The following are the main components in an Oracle BAM Server:

Active Data Cache is designed and optimized to handle large amounts of data in a real-time solution. To make data readily accessible and deliverable, it maintains real-time views of the data. The data feed to the Active Data Cache is a combination of business data sources, from data warehouse information to transactional feeds and other enterprise sources. The various data streaming technologies integrated with Oracle BAM send this information to the Active Data Cache in a continuous stream as data changes occur.

The Active Data Cache hosts and runs the data objects, the view sets and the active view sets. It receives transactions (insert, update, and delete) to its data objects, and these data objects notify other data objects which are linked to them through lookups. Active view sets which are monitoring these data objects are notified of the changes and produce active data.

Event Engine: Event Engine monitors complex data conditions and implements specified rules. Rules can include a series of conditions and actions attached to an event. The Event Engine continuously monitors the information in the Active Data Cache for certain conditions and runs the related actions defined in associated rules.

The Event Engine is responsible for tracking events based on date, time or data changes. The design of the Event Engine uses a satellite concept, in which there are four different systems (satellites), with which event clauses can be registered and in which they can be tracked.

Report Cache off loads the burden of maintaining the view set snapshot in memory from the Active Data Cache. The Report Cache opens view sets and active view sets in the Active Data Cache for the Report Server in Oracle BAM Web Applications set of components. It then caches the snapshot (in chunks) and the active data before sending it to the Report Server. This allows for random access into the snapshot and recovery from losing internet connectivity. The Report Cache also enables the Report Server to be stateless, and with the Active Data Cache it supports view set sharing.

Real Time Data Streaming Clients: In an Oracle BAM system clients can feed data into Oracle BAM server by using different types of protocols and APIs. The following are the available mechanisms to send data to the Oracle BAM server:

Oracle BAM Adapter is a JCA-compliant Adapter, which Java EE applications can use to send data to the Oracle BAM Server.

Enterprise message sources are used by applications to provide direct Java Message Service (JMS) connectivity to the Oracle BAM server by mapping messages directly to Oracle BAM data objects. The Oracle BAM server can read data directly from any JMS based message queue or topic. This option offers guaranteed messaging.

In an Oracle BAM high availability environment, involving Enterprise Message Sources, the Enterprise Message Source property Start when BAM Server starts should be set to Yes.

Durable Subscriber Name must be set in the Enterprise Message Source to ensure that no messages are lost during the failover.

Oracle Data Integrator is the ETL tool that is used with Oracle BAM to perform rigorous data transformations. The Oracle BAM Server has been implemented as an ODI Technology (for example, DB2, SQL Server are ODI Technologies) and Oracle BAM has ODI Knowledge Modules which let ODI perform all of the operations on the Oracle BAM Server to facilitate reading and writing data in various ways, including Change Data Capture.

Oracle BAM Web Applications are the web user interfaces (Java EE web applications) for viewing Oracle BAM data. Oracle BAM Web Applications also allow for building data models and creating dashboards and alerts. Oracle BAM Web Applications are the browser based interfaces provided for building reports and administrating the users that have access to the BAM User Interface system. As part of the web applications, a Report Server applies the report definitions to the data sets retrieved from the Active Data Cache for presentation in a browser.

ICommand is a command line interface for administrating the data in the Active Data Cache. It provides a set of commands to export, import, rename, clear, and delete items from Active Data Cache.

The BAM application runs separately from the rest of Oracle SOA Suite components. Oracle BAM Web Applications are deployed together with the BAM Server. Oracle BAM Server uses EJBs and DAOs for implementing its services and persisting information to the database. All of the EJBs are stateless. However, the Oracle BAM Server is a singleton: only one active Oracle BAM server is used for maintaining fed data and for pushing it into the Web Application Reports. Oracle WebLogic Server Migration feature is used to protect Oracle BAM server from failures. Oracle BAM server uses JMS intensively. However, all Oracle BAM JMS queues are internal and do not require to be configured in a high availability configuration. When failover occurs, the queues are recreated on the new active Oracle BAM instance.

Oracle BAM Web Application Modules can run in full active-active mode connecting to the available Oracle BAM server. The information is retrieved using RMI protocol to access the EJBs that provide the information to remote consumers. Oracle BAM Web Applications are stateless, except for the Reports Server, which maintains a session ID for reports or users accessing the systems. This requires enabling session replication for the WebLogic Managed Server where Oracle BAM Web Applications run.

The processing of work by the EJBs and threads in Oracle BAM is non transactional and does not require Oracle WebLogic Server special transaction logs configuration. At the same time, Oracle BAM uses the database intensively, therefore it is important that Oracle BAM's database access is prepared for failure in the database. This requires configuring GridLink data sources or multi data sources for Oracle BAM data source as Section 5.15, "Configuring High Availability for Oracle BAM" describes.

External Dependencies

Oracle BAM engine system depends only on the Oracle BAM database (which contains Oracle BAM schemas) for Oracle BAM data and report metadata. The database must be available for Oracle BAM system to start and run properly.

As shown in Figure 5-31, when the WebLogic Server is started, the Oracle BAM Server application is initialized. During startup, the Active Data Cache loads all the data from the repository. The Reports Cache is initialized with the data required for the available systems. The Event Engine starts analyzing the data and preparing notifications. Oracle BAM Web Applications are configured at boot time with an Oracle BAM Server's address. When Oracle BAM Web Applications are started, they connect to the Active Data Cache in Oracle BAM server and retrieve the corresponding viewsets through the Reports Server. Once initialization is done, the system is ready to receive data from clients and raise events and update reports dynamically.

5.11.1.3 Oracle Business Activity Monitoring Startup and Shutdown of Processes

Oracle BAM Server is an application, oracle-bam, that operates independently from the rest of the SOA Service Infrastructure. It runs in the managed server where Oracle BAM is installed. It gets started by default with Oracle WebLogic Managed Server to which Oracle BAM is deployed. Use the Administration Console to verify Oracle BAM Server's status and to start and stop it. You can also use WebLogic Server WLST command line to control the application.

Figure 5-32 Startup and Shutdown of Oracle Business Activity Monitoring using WebLogic Server

The BAM Web Applications run collocated by default in the same managed server as BAM Server.For example, the BAM server and BAM Web Applications are part of the same deployment in Administration Console. They can't be stopped and managed separately from BAM Server by default. However, you can manipulate WebLogic Server application targeting to separate both components. The Oracle WebLogic Administration Console enables starting and stopping Oracle BAM.

Although Enterprise Manager offers separate start and stop for Oracle BAM Server and Oracle BAM Web Applications, this is in reality a stop operations that affects both components, meaning if Oracle BAM Server is stopped from Enterprise Manager, the corresponding Oracle BAM Web Applications are also stopped, and the reverse is also true. This also applies for start operations.

5.11.1.4 Oracle Business Activity Monitoring Configuration Artifacts

Oracle Enterprise Manager Fusion Middleware Control exposes some of the configuration options for Oracle BAM Server.

Configuration options at the container level, such as data sources and persistent stores, are maintained as part of Oracle WebLogic Server Domain configuration and are synchronized across a cluster of Oracle WebLogic Servers by Oracle WebLogic Server core infrastructure. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite for details on configuring Oracle BAM.

Figure 5-37 describes a two-node Oracle BAM cluster running on two Oracle WebLogic Servers. The Oracle WebLogic Servers are frontended by Oracle HTTP Servers which load balance incoming requests to them. The following are the main characteristics of this configuration:

Oracle BAM Web Applications run on two clustered WebLogic Server managed servers. The WebLogic Server Cluster synchronizes configuration for common artifacts of WebLogic Server used by Oracle BAM Web Applications, such as data sources, persistent store, and definitions. Oracle BAM Server is targeted by any of the servers where BAM Web applications are running. Only one WebLogic server runs Oracle BAM Server.

The architecture uses Oracle WebLogic Server Migration feature to protect Oracle WebLogic Server that runs both Oracle BAM Web Applications and Oracle BAM Server (in BAMHOST1) against failures. This provides protection for Oracle BAM Server which is a singleton. The WebLogic Managed Server in which Oracle BAM Server runs is listening on a Virtual IP that gets migrated to another node when the failover occurs. This is the address that Oracle BAM Web Applications in BAMHOST2 use to connect to an Oracle BAM Server. Plan appropriately to account for the scenario where Oracle BAM Server and two instances of Oracle BAM Web Applications are running on BAMHOST2. For more information on Server Migration features, see Chapter 3, "High Availability for WebLogic Server."

Oracle BAM Server and Oracle BAM Web applications are protected from all process failures by the WebLogic Server infrastructure. This section also describes expected behavior if component failure occurs.

Process Failure

Oracle BAM Server and BAM Web Applications are protected from all process failures by the WLS infrastructure. This section describes process failure considerations for Oracle BAM.

If the WLS_BAMx server fails, Node Manager attempts to restart it locally. It attempts to restart the servers according to the configured restart count threshold.

For failures related to the WebLogic Server on which the BAM Server is running, if repeated restarts fail, the WebLogic Server infrastructure attempts to perform a server migration to the other node in the cluster. Ongoing requests from the clients time out during failover. Once the server's restart completes on the other node, the clients should be restarted to continue routing to it. For opened reports running in the web browser, if a request is delivered though the BAM WebApps running in the same managed server as BAM Server, a Reconnecting message appears in the BAM WebApps report. If the request is being delivered through the BAM WebApps running in the other node, no message appears. In this situation, and during failover, you may be viewing stale data (while a failover takes place or while connection from webapps to servers is reestablished). The BAMWebApps Servlet remains available, and reconnects as soon as BAM Server comes up. For new reports or requests, BAM Web Applications inBAMHOST2 reconnect once the VIP is migrated and the managed server is restarted. BAM Web Applications listening on VIP1 become functional once the server migration completes. At this point, the HTTP Server restarts routing HTTP requests to the Managed Server.

Failures affecting the WebLogic Server where only the BAM Web Applications are running, do not affect the BAM system. Other BAM Web Applications (running in BAMHOST1) remain available and maintain session information by using the WebLogic Server Session replication cluster. Failover should be transparent.

The oracle-bam application where BAM Server and BAM Web Applications run may be down due to failure in accessing resources, errors in reading the database, or other issues unrelated to the status of the managed server where it is located. Therefore, you should monitor the Oracle SOA Service Infrastructure application and watch for errors caused by the application in the managed server logs, for log file locations, see Section 5.2.1.6, "Oracle SOA Service Infrastructure Log File Locations".

Failover may not succeed for an open report if when the report is opened, only one BAM managed server in the cluster was up, and after starting an additional managed server in the cluster, no other operations were performed. One trigger for session state replication is opening a report. Session state replication is not triggered by active data updates.

In an Oracle BAM high availability environment, when the Oracle BAM Active Viewer is running on an Oracle BAM server that is shut down and then restarted, a "viewset not found" error will be written to the server's log. This error does not affect the functionality of the Oracle BAM Active Viewer and can be ignored.

Node Failure

For node failures in SOAHOST2, the behavior in case of a node failure is equivalent to the process failure scenario: Oracle BAM Web Applications in the other node remain available and can serve requests. Sessions are preserved by the session replication framework provided by WebLogic Server and failover to the other node should be transparent.

For node failures in SOAHOST1, server migration is triggered after the available server verifies the time stamp in the database leasing system. While failover occurs, clients are unable to feed data into the system and retry appropriately. Oracle BAM Web Applications connecting to the server attempt to reconnect until the VIP is migrated and the server is restarted.

The standard Java EE artifacts that Oracle BAM Server and Oracle BAM Web Application use are configured as part of Oracle WebLogic Domain in which Oracle BAM is installed. Oracle WebLogic Clusters provide automatic configuration synchronization for artifacts such as data sources, persistent stores, and JMS modules, across the WebLogic Server domain. At the same time, the WebLogic Server cluster is in charge of synchronizing the deployments, and libraries used by Oracle BAM Web Applications and Oracle BAM Server.

As explained in the single instance section Oracle BAM Server's and Oracle BAM Web Applications configuration are maintained in the DOMAIN_HOME/servers/BAM_Server_Name/tmp/_WL_user/oracle-bam_11.1.1/yhryfp/APP-INF/classes/config directory (notice that yhryfp is the random directory generated at installation time for deploying BAM applications). The properties in these files can be modified by using the Mbeans exposed in Oracle Enterprise Manager Fusion Middleware Control. The properties exposed through MBeans are specific to each server. The properties exposed through Enterprise Manager-specific screens are cluster-wide and are only modified on one server. All properties, whether applied in Enterprise Manager or in an MBean browser require a restart of Oracle WebLogic Servers where Oracle BAM runs. For details on the configuration options for BAM Server and BAM Web Applications see the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite and the Oracle Fusion Middleware User's Guide for Oracle Business Activity Monitoring.

One of the configuration options related to high availability environments is the Application URL which is used to determine the front end host used by the system in a cluster configuration. This option is used to produce the copy shortcut URL for reports and alerts. The other relevant parameter for Oracle BAM high availability configuration is the Server Name in OracleBAM Web configuration screen. This parameter is used by Oracle Web Application to determine Oracle BAM server to which it connects for accessing the Active Data Cache.

For SOA high availability installations frontended by Oracle HTTP Server, Oracle recommends that you monitor the Oracle HTTP Server ports of the real backend servers. This is the case when a deployment is using all the components deployed to the SOA Managed Server. A simple HTTP monitor that pings the HTTP/HTTPS port and expects a pre-determined response is sufficient. If you use a specific SOA component only, such as B2B, you may want to consider a monitor that does a deeper-level check all the way to the Managed server to validate the health of the component in use. Please check with your load balancer vendor on setting up the HTTP monitors with your load balancer.

5.11.2.3 Considerations for BAM Client Retries

Oracle BAM clients communicating with the BAM Server will retry "retryable" methods if the failure of the method execution was due to a "retryable" exception. A typical "retryable" exception is one that indicates a failure to communicate with the BAM Server (like a network connection exception). Some of the BAM Server's method are automatically retried by the BAM client when a failure occurs. Additionally the list of exceptions to be retried can be extended. This can be done by adding a comma-separated list of exception class names to be treated as retryable inside the following configuration fields in the clients' configuration files:

<BamEjbRetryableExceptions></BamEjbRetryableExceptions>

Other configuration parameters for retryable exceptions are:

<BamEjbRetryCount>180</BamEjbRetryCount>: This parameter configures the number of retries to be attempted if getSession or retryable method invocations fail due to a retryable exception. The default is 180.

<BamEjbRetryInterval>10000</BamEjbRetryInterval>: This parameter configures the number of milliseconds to pause between retry attempts. The default is 10000 milliseconds or 10 seconds.

These parameters can be added in the configuration files for the following clients accessing the BAM Server:

5.12 Oracle Service Bus and High Availability Concepts

Oracle Service Bus is a proven Enterprise Service Bus (ESB) built from the ground up for SOA lifecycle management that provides foundation capabilities for service discovery and intermediation, rapid service provisioning and deployment, and governance.

Figure 5-38 shows the high level functional features of Oracle Service Bus.

The Java EJB protocol proxy service. This service, also known as JEJB, is used to expose proxy services as EJBs.

Service Bus transport. Also known as SB, this protocol is used for communication between two Oracle Service Bus services.

The test console for testing web services.

JMS: Is used for inbound message processing for the FTP, SFTP, Email, and File transports. It is also used for the Oracle Service Bus reporting framework and for Web Services Reliable Messaging (WS-RM).

JDBC Data Source: Oracle Service Bus defines a data source for the database tables associated with the JMS Reporting provider.

JCA Adapters: Oracle Service Bus supports the DB, Oracle Applications, AQ, File, BAM, FTP, SAP, Siebel, PeopleSoft, JD Edwards adapters using JCA transport. The adapter implementation for these adapters is same as the one used in SOA Suite.

Oracle Service Bus also uses the following non-Java EE components:

Configuration Framework: The configuration framework component provides CRUD (Create, Read, Update, Delete) capabilities for Oracle Service Bus resources along with resource and file integrity protection, session management, caching and indexing, referential integrity, and configuration propagation. The Oracle WebLogic Server Administration Server has the master configuration in files and these are propagated to the managed servers using Weblogic Deployment Service.

Monitoring and Alert Framework: The monitoring framework defines statistics based on counters (message, error counts), intervals (min/max/avg. response or execution time) and status (end point up/down). The monitoring framework has the following sub-components:

Collector: collects statistics at one minute intervals called checkpoints, stores collected statistics to Weblogic Persistent Store, and sends the collected statistics to the Aggregator. Collector runs on each managed server in the cluster.

Aggregator: aggregates statistics received from each of the managed servers in the cluster. Aggregator is a singleton service that runs on only one managed server in the cluster.

Retriever: retrieves the statistics that are stored in the memory. This is only in the managed server, which contains the Aggregator.

Alert Manager: The Alert Manager fires alerts based on the aggregated statistics. This is only in the managed server that contains the Aggregator. The Aggregator pushes aggregated statistics to the Alert Manager when there is an aggregation interval (defined for a service or an alert rule) for which rules need evaluation. The Alert Manager is triggered in a separate thread than the Aggregator.

Transport SDK: Oracle Service Bus processes messages independently of how they flow into or out of the system. The Transport SDK provides a layer of abstraction between Oracle Service Bus and components that deal with the flow of data in and out of Oracle Service Bus. This layer of abstraction enables users to develop new transport providers to handle unique transport protocols. A transport provider implements the interfaces of the Transport SDK and provides a bridge between Oracle Service Bus and mechanisms by which messages are sent or received. A transport provider manages the life cycle and runtime behavior of transport endpoints. An endpoint is a resource where messages originate or are targeted.

5.12.1.1 Oracle Service Bus Session State

Oracle Service Bus is optimized for short-lived message processing/mediation use cases. Thus, all message context state is kept in memory. For synchronous transports such as HTTP, EJB, and SB, the client must retry if the Oracle Service Bus server fails in the middle of processing the synchronous request. Oracle Service Bus supports request/response pattern for asynchronous transports such as JMS and WS (WS-RM). In these cases correlation information, which maps the request to its corresponding response destination, is kept in memory. If the Oracle Service Bus server fails before the response is sent, the correlation information is lost. Therefore, Oracle Service Bus will fail to send the response after server restart. For more details about JMS transports and WS transports, see the "JMS Transport" chapter and the "WS Transport" chapter in the Oracle Fusion Middleware Developer's Guide for Oracle Service Bus.

5.12.1.2 Oracle Service Bus External Dependencies

Oracle Service Bus has the following external dependencies:

The JMS Reporting Provider uses a database to store the report data.

UDDI (Oracle Service Registry) is used to store Oracle Service Bus proxy services and to browse services to create Oracle Service Bus business services.

Oracle Enterprise Repository is used to browse and consume services. The result is generation of Oracle Service Bus business services. Oracle Service Bus proxy services are harvested into Oracle Enterprise Repository.

Oracle Service Bus implements an Oracle WSM agent that enforces Oracle WSM security policies attached to Oracle Service Bus proxy and business services.

Oracle Service Bus depends on JCA Adapter implementations (JCA Adapters for HTTP, FTP, and so on) and the JCA framework from SOA Suite.

Oracle Service Bus depends on the Coherence infrastructure for Service Result Caching (caching of results from services invoked by Oracle Service Bus).

Oracle Service Bus message flows are short lived in general. Message flow state is always in memory. Oracle Service Bus provides various out of box transports. For example, the HTTP transport supports HTTP 1.1 persistent connections.

5.12.1.3 Oracle Service Bus Configuration Artifacts

DOMAIN_HOME/config/config.xml: has all applications and libraries, JMS system resources, JDBC system resources, work manager, startup/shutdown classes, SAF agent, security configuration.

DOMAIN_HOME/config/osb/coherence/osb-coherence-override.xml: This Oracle Coherence override file specifies Oracle Coherence unicast/multicast listener information. This file is propagated by the Administration Server from the Administration Server domain directory to the domain directory for other managed servers.

DOMAIN_HOME/config/osb/coherence/osb-coherence-cache-config.xml: This Coherence cache configuration file defines the cache used by the Oracle Service Bus service result caching feature.

DOMAIN_HOME/config/osb/transports/sftp/known_hosts: This file is used by the SFTP transport. It contains information about remote SFTP servers. See Oracle Fusion Middleware Developer's Guide for Oracle Service Bus for information about creating the known_hosts file. Note that virtual hostnames can be used in the known_hosts file.

5.12.1.4 Oracle Service Bus Deployment Artifacts

You can use Eclipse IDE or the Oracle Service Bus web console to export your Oracle Service Bus artifacts (resources), such as proxy service, business service, wsdl, xsd, and xslt resources, to a jar file with a name that you choose. The jar file contains all the Oracle Service Bus resources that were exported. You have two ways to import the configuration jar file (and the Oracle Service Bus resources inside it) into an Oracle Service Bus server:

WLST/ANT scripts using public Oracle Service Bus APIs

Oracle Service Bus web console: Using the Import from Configuration Jar option

Oracle Service Bus resources are managed by the configuration framework component. The configuration framework stores all its data in DOMAIN_HOME/osb/config (CONFIG_HOME) on each managed server (and admin server). Each resource is stored in its own file in CONFIG_HOME/core. The Oracle Service Bus configuration framework automatically assigns unique file names to the file used for each resource to ensure that no name collisions occur among these files. The CONFIG_HOME/core directory is the master view of all the resources and configuration data. The runtime works off of this data. It changes only when changes made in a session are activated. Each managed server has a copy of the resources in CONFIG_HOME/core. Depending on the location of the managed server, this directory could be mounted on a separate system.

5.12.1.5 Oracle Service Bus Startup and Shutdown

You can use Node Manager or the startWebLogic/stopWebLogic scripts to start and stop the Oracle Server Bus server. Oracle Enterprise Manager Grid Control supports management and monitoring of Oracle Service Bus services.

Oracle Service Bus is initialized through application listeners. The ALSB Framework Starter Application does the main framework initialization through an application listener. This application is deployed to the Administration Server and the cluster. The prestart for this application listener initializes Logging, ALSBConfigService, Security service, cluster timer service, and ALSBStatisticsManager. XBus Kernel is the next important application deployed that performs the rest of the initialization and start procedures for the Oracle Service Bus components. This application is targeted to the Administration Server and cluster, and it initializes upgraders, resources, split-join, Coherence, and so on.

Death detection capabilities are provided by Oracle WebLogic Server.

5.12.1.6 Oracle Service Bus Log File Locations

Oracle Service Bus operations are logged to the Oracle WebLogic Managed Server where the application is running:

DOMAIN_HOME/servers/WLS_ServerName/logs/WLS_ServerName.log

Oracle Service Bus also supports Debug logging, which can be enabled or disabled by updating DOMAIN_HOME/alsbdebug.xml and DOMAIN_HOME/configfwkdebug.xml.

5.12.2 Oracle Service Bus High Availability Architecture and Failover Considerations

Figure 5-40 shows a two-node Oracle Service Bus cluster running on two Oracle WebLogic Servers. The Oracle WebLogic Servers are frontended by two Oracle HTTP Servers on web tier hosts, which receive requests from a load balancer in front of them.

The main characteristics of the high availability configuration in Figure 5-40 are:

The Oracle WebLogic Server server migration feature is used to protect some of the components against failure. This implies that each of the WebLogic managed servers in which Oracle Service Bus runs is listening on a virtual IP that is migrated to another box upon failover. Enterprise Manager is deployed to the Administration Server in Figure 5-40.

As shown in Figure 5-40, the WLS_OSB1 managed server listens on VIP1, and the WLS_OSB2 managed server listens on VIP2. Each managed server uses the other node as a failover resource; the system is configured in a cross manner. WLS_OSB1 fails over to OSBHOST2, and WLS_OSB2 fails over to OSBHOST1. The appropriate capacity planning must be done to anticipate the scenario where the two Oracle Service Bus managed servers are running on the same node. For more information on server migration features, see Section 3.9, "Whole Server Migration."

To resume transactions after a server migration, configure the transaction and JMS stores in a shared storage. In case of failure in one of the server infrastructure instances, other instances can resume transactions and JMS operation by reading the persistent stores from that shared storage.

The Administration Server runs in Active-Passive mode. Whenever a failure occurs in OSBHOST1, the Administration Server can be restarted in OSBHOST2. Therefore it uses a virtual IP or virtual hostname as listen address. For information about configuring virtual IPs for the Administration Server and configuring the Administration Server for high availability, see Chapter 12, "Active-Passive Topologies for Oracle Fusion Middleware High Availability."

An Oracle RAC database is recommended to protect from database failures. In this Oracle Service Bus high availability configuration, an Oracle RAC database is used for the JMS Reporting Provider and Oracle WSM policies.

Note:

To keep the Oracle Service Bus high availability configuration figure above readable, the Oracle SOA Service Infrastructure high availability configuration is not shown. See Figure 5-5 to view the Oracle SOA Service Infrastructure high availability figure.

The ALSB Cluster Singleton Marker Application determines the managed server in the cluster that runs the singleton monitoring Aggregator and Alert Manager. Similarly the Message Reporting Purger MDB, which helps purging reporting data from the database (for the JMS Reporting Provider), is deployed on only one managed server in the cluster. The remaining Oracle Service Bus components are all either targeted to the cluster, or to the cluster and Administration Server. Only the ServiceBus_Console, which provides web console support for Oracle Service Bus, and the UDDI Manager, which registers resource types for UDDI import with the configuration framework are targeted to the Administration Server only. These components are not defined as Weblogic Singleton services. Their high availability is assured by Weblogic whole server migration.

All Oracle Service Bus services and resources are deployed uniformly to the cluster (to all managed servers in the cluster). The only exceptions to this are:

Inbound (Proxy Service) endpoints for the poller transports: Email, File, FTP, and SFTP. The poller polls for messages on only one managed server in the cluster. The user configuration determines the managed server that polls.

Inbound (Proxy Service) endpoints for JMS transport where the JMS transport is listening to a Weblogic JMS Topic destination (using compatibility Topic Message Distribution Mode) and is configured to process the message from the Topic only once (in the cluster) on the managed server configured by the user.

The basic highly available topology is a Weblogic Server homogenous cluster (except for the Oracle Service Bus singletons services mentioned above) with one Administration Server and two managed servers running on different systems.

Only one cluster of Oracle Service Bus is supported in a Weblogic domain. Local data should be stored on shared storage, such as a SAN storage system, multiported disk, or NAS storage.

The local data (accessed as local files, private to a managed server or Administration Server) includes:

System files like Weblogic configuration files and server logs. Optionally, you can store WLS JMS data in an Oracle RAC database. JMS is used both internally in Service Bus and as transports to external and proxy services.

Oracle Service Bus data such as configuration files and logs

User-defined Oracle Service Bus configuration data

User files, such as data files, that are read or written by a proxy service with a File/FTP transport

The Oracle RAC database is used for high availability of the reporting provider and also for the leasing datasource used for server migration.

Optionally, a web server farm may be used as a front end to an Oracle Service Bus cluster. A hardware load balancer can load balance web servers or application servers directly. If web servers are used, WebLogic plugins should be configured to load balance HTTP traffic among the Oracle Service Bus managed servers.

Email, FTP, or NFS servers can exist in the network, and third party JMS servers or server clusters can also exist in the network.

5.12.2.1 Oracle Service Bus Protection from Failures and Expected Behavior

This section describes how an Oracle Service Bus high availability cluster deployment protects components from failure. This section also describes expected behavior if component failure occurs.

Oracle Service Bus is protected from all process failures by the WebLogic Server infrastructure.

5.12.2.1.1 WebLogic Server Failure

Oracle Service Bus does not maintain any state, nor does it support the concept of user sessions. Therefore, Oracle Service Bus does not implement session state replication and failover. For synchronous inbound transports such as HTTP, SB, and EJB, if the managed server processing the request goes down in the middle of request processing, the client receives a connection exception and needs to retry.

There is no automatic failover for Oracle Service Bus singleton components such as the Aggregator, Alert Manager, and Reporting Message Purger. Use WebLogic server migration to perform the failover of singleton components.

Note:

Although the Aggregator, Alert Manager, and Reporting Message Purger services are singletons, there are no service outages if they fail. For poller transports (file, FTP, SFTP, and e-mail), only the proxy services that use the poller transports fail if the server fails.

5.12.2.1.2 Node Failure

The external load balancer can be used to directly load balance HTTP requests to Oracle Service Bus HTTP Proxy Services on the managed servers in the cluster or to a Web Server/Oracle HTTP Server (OHS) cluster front-ending the Oracle Service Bus cluster. There are no sticky session routing requirements.

Node Manager can be configured to failover failed Oracle Service Bus nodes.

If an Oracle Service Bus managed server fails in the middle of processing HTTP requests, clients need to submit those requests again. For JEJB and SB transports, a client gets a connection error when the server goes down. For all poller transports, the integrity of message processing is handled by transactional semantics as the message is dequeued using XA connection factory. Poller transports provide "at least once" semantics. However, after whole server migration (of the failed node), the poller needs to have access to the resources it is polling. JTA transactions will also preserve transactional integrity when using JMS transport. JTA TLogs need to be recovered to recover in-flight JTA transactions.

When a managed server starts up, any in-flight global transactions are recovered. For example, recovery of global transactions may be required when the proxy service is bridging between two JMS providers using XA connection factories, and QoS of exactly once. Another example is the JMS reporting provider. With the default JMS Reporting Provider, the report message is first written to the JMS queue. The MDB dequeues the report message from the JMS queue and then writes it to the database. The (database) data source is configured with transaction semantics of LLR (Last Logging Resource optimization). In this case, the database must be running during recovery, otherwise the server will not start. The Tlog is used even when LLR is in effect. However, this happens only on a per transaction basis. Therefore the Tlog must still be made highly available even when all transactions are LLR transactions. Transaction Manager persists checkpoint TLog records that are unrelated to specific transactions, but that are still required to provide full transactional safety.

There are two cases where failure of the managed server causes certain state to be lost, which prevents correct processing of messages:

JMS request/response business service: Maintains in-memory table mapping correlation information. This is lost when the managed server goes down. Thus, the response from the JMS service cannot be sent back to the original client of the proxy service, which routed to the JMS request/response business service.

WS (Reliable Messaging) business service: Similar to the JMS request/response business service, the WS business service keeps an in-memory table mapping correlation information. This is lost when the managed server goes down, and the response cannot be handled.

5.12.2.1.3 Database Failure

Beyond the implications for database leasing (used for server migration) database failures are relevant for Oracle Service Bus only in the context of the JMS reporting provider functionality. When there is report action in the proxy service, the report data is enqueued into the reporting JMS queue. The reporting MDB dequeues the message from the JMS queue and inserts the data into the database using the data source configured with the Logging Last Resource (LLR) global transaction protocol. The reporting JMS queue is configured with a redelivery limit of 2 and an error queue. When a database failure occurs, report messages are moved to the error queue after the redelivery limit is reached. When the database is running again, you can move these report messages back to the JMS reporting queue so that they are inserted into the database. If the database fails in the middle of LLR transaction processing, transaction recovery is performed as described in the chapter on Logging Last Resource Transaction Optimization in Oracle Fusion Middleware Programming JTA for Oracle WebLogic Server.

5.12.2.2 Oracle Service Bus Cluster-Wide Deployment

Oracle Service Bus resources are updated/deployed onto the WebLogic Administration Server. The Administration Server uses the WebLogic deployment service to propagate the Oracle Service Bus resources to all the managed servers in the cluster.

5.12.2.3 Online Redeployment of Oracle Service Bus in a Cluster

Oracle Service Bus does not keep multiple records (versions) or historical copies of resources. When a resource is updated, it is as if a new version is deployed and the older version is overwritten. The configuration framework makes changes to resources in a session, which is a copy of the core (that all managed servers are currently using) when the session is created. When the session is committed, the core is updated with the changes. The changes are then propagated in a transactional manner to the managed servers in the cluster. In-flight requests (requests already running the Proxy Service Message flow) continue to use the old copy of the resources when the update is in progress. Requests after the changes are added to the core use the updated resources.

5.12.2.4 Oracle Service Bus Cluster-Wide Configuration Changes

Oracle Service Bus resources are updated at the cluster level. The WebLogic Administration Server propagates these changes to all the managed servers in the cluster. You can make configuration changes using the Oracle Service Bus web console, the Administration Console, and public MBeans.

The following files are excluded from the automatic propagation to the managed servers in the cluster:

Files in the DOMAIN_HOME/config/osb/coherence/ directory

DOMAIN_HOME/config/osb/transports/sftp/known_hosts

Each managed server must have access to the DOMAIN_HOME to access files in these directories.

5.13 Configuring High Availability for Oracle SOA Service Infrastructure and Component Service Engines

The procedures described in this section include setting up the Service engines contained in the Oracle SOA Service Infrastructure system, such as Oracle BPEL/Oracle PM, Oracle Mediator, Oracle Human Workflow and Oracle Decision Services, as well as Oracle B2B and Oracle User Messaging Service.

Oracle strongly recommends reading the release notes for any additional installation and deployment considerations prior to starting the setup process.

Note:

The architectures and deployment procedures defined in this guide enable simple clustered deployments. The procedures described in these chapters can be used as a building block to enable this and other similar high availability topologies for these Fusion Middleware components. It is also expected that production deployments will use other required procedures, such as associating security policies with a centralized LDAP server. For complete details of secured, multi-tiered architectures, and deployment procedures, please refer to the Enterprise Deployment Guide for the component you are configuring.

Figure 5-41 represents the example architecture that the configuration steps in this section create.

Figure 5-41 describes a two-node SOA cluster running on two Oracle WebLogic Servers. The Oracle WebLogic Servers are frontended by Oracle HTTP Servers, which load balance incoming requests. A load balancer front ends the system and distributes incoming requests from clients to the two Oracle HTTP Servers. A separate Oracle WebLogic Server (not shown in the figure) is typically used for custom logic and application deployment. This configuration uses an Oracle RAC database for storing metadata and SOA schemas, and shared storage for transaction and JMS stores. Virtual IP addresses (VIPs) provide manual failover for the Administration Server and for Oracle SOA Servers (for Server Migration). For more details about the components contained in this architecture, see the individual component sections in this chapter.

SQL>select version from sys.product_component_version where product like 'Oracle%';

5.13.1.2 VIP and IP Prerequisites

As shown in Table 5-2, you configure the Administration Server and the SOA managed servers to listen on different virtual IPs. This requires the provisioning of the corresponding VIP in the node and related hostnames in the DNS system in your network. Ensure that the different VIPS are available and are reachable from SOAHOST1, SOAHOST2, and the client systems before running the installation.

Table 5-2 Virtual Hosts

Virtual IP

VIP Maps to...

Description

VIP0

SOAHOST1VHN0

SOAHOST1VHN0 is the virtual hostname that is the listen address for the Administration Server and fails over with manual failover of the Administration Server. It is enabled on the node where the Administration Server process is running (SOAHOST1 by default).

VIP1

SOAHOST1VHN1

SOAHOST1VHN1 is the virtual hostname that maps to the listen address for WLS_SOA1 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA1 process is running (SOAHOST1 by default).

VIP2

SOAHOST2VHN1

SOAHOST2VHN1 is the virtual hostname that maps to the listen address for WLS_SOA2 and fails over with server migration of this managed server. It is enabled on the node where WLS_SOA2 process is running (SOAHOST2 by default).

5.13.1.3 Shared Storage Prerequisites

For proper recovery in case of failure, store both JMS and transaction logs in a location that is accessible to all the nodes that can resume the operations after a failure in a managed server. This requires a shared storage location that can be referenced by multiple nodes. Table 5-3 lists the contents of shared storage.

The shared storage can be a NAS or SAN device. Specifically for NFS mounted systems, different issues related to file locking and abrupt node failures have been detected. Check the Oracle Fusion Middleware Release Notes and with your storage vendor for the main recommended parameters to be used as mount options. The following is an example command based on a NAS device. Your options may be different from the ones specified in this section:

ORACLE_HOME: This environment variable and related directory path refers to the location where Oracle FMW SOA Suite is installed.

ORACLE_COMMON_HOME: This environment variable and related directory path refers to the Oracle home that contains the binary and library files required for the Oracle Enterprise Manager Fusion Middleware Control and Java Required Files (JRF).

There can be only one Oracle Common home within each Middleware home.

DOMAIN Directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored.

ORACLE_INSTANCE: An Oracle instance contains one or more system components, such as Oracle Web Cache, Oracle HTTP Server, or Oracle Internet Directory. An Oracle instance directory contains updateable files, such as configuration files, log files, and temporary files.

The values used and recommended for consistency for this directories are:

Ensure that the following initialization parameter is set to the required value. It is checked by Repository Creation Assistant.

Table 5-4 Required Initialization Parameters

Parameter

Required Value

Parameter Class

PROCESSES

300 or greater

Static

To check the value of the initialization parameter using SQL*Plus, you can use the SHOW PARAMETER command.

As the SYS user, issue the SHOW PARAMETER command as follows:

SQL> SHOW PARAMETER processes

Set the initialization parameter using the following command:

SQL> ALTER SYSTEM SET processes=300 SCOPE=SPFILE

Restart the database.

Note:

The method that you use to change a parameter's value depends on whether the parameter is static or dynamic, and on whether your database uses a parameter file or a server parameter file. See the Oracle Database Administrator's Guide for details on parameter files, server parameter files, and how to change parameter values.

Host Name: Enter the name of the node that is running the database. For an Oracle RAC database, specify the VIP name, or one of the node names as the hostname: SOADBHOST1VIRTUAL.

Port: The port number for the database: 1521

Service Name: Enter the service name of the database: soaha.mycompany.com

Username: SYS

Password: Enter the password for the SYS user.

Role: SYSDBA

Click Next.

If you receive the following warning message:

The database you are connecting is with non-UTF8 charset, if you are going to use this DB for multilingual support, you may have data loss. If you are not using for multilingual support you can continue, otherwise we strongly recommend using UTF-8 database.

Click Ignore or Stop.

In the Select Components screen, select Create a New Prefix, and enter a prefix to use for the database schemas, for example, SOAHA

Write down the schema names so they are available in later procedures.

You need the appropriate database privileges to enable the Oracle WebLogic Server transaction manager to query for transaction state information and issue the appropriate commands, such as commit and rollback, during recovery of in-flight transactions after a WebLogic Server container failure.

To configure the SOA schemas for transactional recovery privileges:

Log on to SQL*Plus as a user with sysdba privileges. For example:

sqlplus "/ as sysdba"

Grant select on sys.dba_pending_transactions to soaha_soainfra.

Grant force any transaction to soaha_soainfra.

Note:

These privileges should be granted to the owner of the soainfra schema, as determined by the RCU operations.

This section describes the load balancer prerequisites for deploying an Oracle SOA Suite high availability environment.

Load Balancers

Oracle SOA Suite uses a hardware load balancer when deployed in a high availability configuration with two Oracle HTTP Servers as web tier components. The hardware load balancer should have the following features:

Ability to load-balance traffic to a pool of real servers through a virtual hostname: Clients access services using the virtual hostname (instead of using actual hostnames). The load balancer can then load balance requests to the servers in the pool.

Port translation configuration: The load balancer should have the ability to perform port translation, where it enables incoming requests received on one port to be routed to a server process running on a different port. For example, a request received on port 80 can be routed to port 7777.

Monitoring of ports (HTTP, HTTPS)

Virtual servers and port configuration: Ability to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements:

The load balancer should enable configuration of multiple virtual servers. For each virtual server, the load balancer should enable configuration of traffic management on more than one port. For example, the load balancer typically needs to be configured with a virtual server and ports for HTTP and HTTPS traffic.

The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the load balancer through the virtual server names.

Resource monitoring/port monitoring/process failure detection: The load balancer must be able to detect service and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your load balancer has the ability to automatically detect failures, you should use it.

Fault-tolerant mode: It is highly recommended that you configure the load balancer to be in fault-tolerant mode.

Other: It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client system.

Sticky routing capability: Ability to maintain sticky connections to components based on cookies or URL.

SSL acceleration: SSL acceleration is a method of offloading the processor-intensive public key encryption algorithms involved in SSL transactions to a hardware accelerator.

This feature is recommended but not required.

Table 5-5 shows some example virtual server names to use for the external load balancer in the Oracle SOA Suite high availability environment.

Table 5-5 Virtual Server Names for the External Load Balancer

Component

Virtual Server Name

Oracle SOA Suite

soa.mycompany.com

WebLogic Server Administration Console

admin.mycompany.com

Oracle Enterprise Manager Fusion Middleware Control

admin.mycompany.com

Virtual Server Names

This section describes the virtual server names that should be set up for the high availability deployments described in this chapter.

soa.mycompany.com

soa.mycompany.com is a virtual server name that acts as the access point for all HTTP traffic to the runtime SOA components, such as soa-infra and workflow. Traffic to the both SSL and non-SSL ports is configured and typically non-SSL is redirected to SSL. Clients access this service using the address soa.mycompany.com:443. This virtual server is defined on the load balancer.

admin.mycompany.com

This virtual server acts as the access point for all internal HTTP traffic that gets directed to the administration services. The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address admin.mycompany.com:80 and in turn forward these to ports 7777 on WEBHOST1 and WEBHOST2. The services accessed on this virtual host include the WebLogic Administration Server Console and Oracle Enterprise Manager.

In addition, ensure that the virtual server names are associated with IP addresses and are part of your Domain Name System (DNS). The computers on which Oracle Fusion Middleware is running must be able to resolve these virtual server names.

Port 7777 is not used by any service on the nodes. You can verify this by running the following command:

Unix:

netstat -an | grep LISTEN | grep "7777"

Windows:

netstat -an | findstr "LISTEN" | findstr ":7777"

If the ports are in use, make them available.

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct, and that you have write permissions for that directory.

Select Specify Custom Ports. If you specify a custom port, select Specify Ports using Configuration File, and use the Browse function to select the file.

Enter the Oracle HTTP Server port, for example, 7777.

Click Next.

In the Configuration Summary screen, ensure that the selections are correct, and click Install.

In the Installation Progress screen:

For UNIX systems, a dialog box appears prompting you to run the oracleRoot.sh script. Open a command window and run the script, following the prompts.

Click Next.

In the Configuration screen, several configuration assistants are launched in succession. When the configuration assistants are finished, the Configuration Completed screen appears.

In the Configuration Completed screen, click Finish to exit.

Repeat the steps for WEBHOST2 and configure your LBR with a pool containing both the WEBHOST1 and WEBHOST2 addresses.

5.13.1.10.1 Validating Oracle HTTP Server

To verify that Oracle HTTP Server is set up correctly, access the root URL context of the server by using the following URL in a browser:

HTTP://WEBHOST1:7777/

If Oracle HTTP Server is set up correctly, the Oracle FMW Welcome screen appears in the browser.

5.13.2 Installing Oracle Fusion Middleware Home

This section describes the procedure for installing Oracle Fusion Middleware on all nodes in the application tier that run Oracle WebLogic Server and Oracle SOA Suite. Repeat the procedures (described below for SOAHOST1) for installing WebLogic Server and Oracle SOA in SOAHOST2. The directory paths for binary files and domains used when installing new nodes must be exactly the same as those used for first node. If these paths and domains are not exactly the same as those used for the first node, failover will not work properly.

In the Choose Product Installation Directories screen, accept the following directory:

WL_HOME

Click Next.

In the Installation Summary screen, click Next.

In the Installation Complete screen, deselect Run QuickStart, and click Done.

5.13.2.2 Installing Oracle Fusion Middleware for Oracle SOA

On Linux platforms, if the /etc/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct and that you have write permissions for that directory.

If the /etc/oraInst.loc file does not exist, you can skip this step.

Start Oracle Fusion Middleware 11g Oracle SOA Suite Installer:

On UNIX:

SOAHOST1> runInstaller

On Windows:

SOAHOST1> setup.exe -jreLoc JRE_Location

When Oracle Fusion Middleware 11g Oracle SOA Suite Installer prompts you for a JRE/JDK location enter the Oracle SDK location created in the Oracle WebLogic Server installation, for example, MW_HOME/jrockit_160_<version>.

In the Specify Inventory Directory screen, do the following:

Enter HOME/oraInventory, where HOME is the home directory of the user performing the installation (this is the recommended location).

Enter the OS group for the user performing the installation.

Click Next.

Follow the instructions on the screen to run /createCentralInventory.sh as root.

Click OK.

In the Welcome screen, click Next.

In the Prerequisite Checks screen, verify that the checks complete successfully, and click Next.

5.13.3 Enabling VIP1 in SOAHOST1 and VIP2 in SOAHOST2

The SOA domain uses virtual hostnames as the listen addresses for the SOA managed servers. You must enable VIP mapping for each of these hostnames on the two SOA Machines, (VIP1 on SOAHOST1 and VIP2 on SOAHOST2), and must correctly resolve the virtual hostnames in the network system used by the topology (either by DNS Server or by hosts resolution).

Run Oracle Fusion Middleware Configuration Wizard from the SOA home directory to create a domain containing the Administration Server and Oracle SOA components. Ensure that the database where you installed the repository is running. For Oracle RAC databases, all the instances must be running.

Note:

Multiple SOA clusters are not allowed in the same WebLogic Server domain.

Note:

Oracle BPM requires the WL_HOME and ORACLE_HOME homes to be patched to the appropriate Oracle Fusion Middleware patch set level before you perform Oracle Fusion Middleware Configuration Wizard steps to extend a domain.

Change the directory to the location of the Oracle Fusion Middleware Configuration Wizard, located in the Middleware home, SOA directory:

SOAHOST1> cd ORACLE_COMMON_HOME/common/bin

Start Oracle Fusion Middleware Configuration Wizard:

For Linux:

SOAHOST1> ./config.sh

For Windows:

SOAHOST1> config.cmd

In the Welcome screen, select Create a New WebLogic Domain. Click Next.

In the Select Domain Source screen:

Select Generate a domain configured automatically to support the following products.

Service Name: Enter the service name of the database. For a GridLink data source, you must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

Note:

The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, example.com

Username prefix: Enter the prefix for the schemas. The user names in Table 5-6 assume that soaha was used as prefix for schema creation from RCU.

Password and Confirm Password: Enter the password for access to the schemas.

For a GridLink data source, enter the SCAN address in the Listener Address field and SCAN port in the Port field. Enter the ONS host and port information in the ONS Host and Port fields, respectively.

Note:

Oracle recommends that you use SCAN addresses to specify the host and port for both the TNS listener and the ONS listener in the WebLogic console. You do not need to update a GridLink data source containing SCAN addresses if you add or remove Oracle RAC nodes. Contact your network administrator for appropriately configured SCAN urls for your environment. See SCAN Addresses in the Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server guide.

In the Test JDBC Data Sources screen, the connections are tested automatically. The Status column shows the results. Ensure that all connections were successful. If not, click Previous to correct your entries.

Click Next when all the connections are successful.

In the Select Optional Configuration screen, select the following:

Administration Server

JMS Distributed Destinations

Managed Servers, Clusters and Machines

In the Configure the Administration Server screen, enter the following values:

Name: AdminServer

Listen Address: Enter the hostname for the VIP0 virtual IP

Listen Port: 7001

SSL listen port: N/A

SSL enabled: leave unchecked

Click Next.

Verify that the UDD settings are UDD for UMSJMSSystemResource, SOAJMSModule, and BPMJMSModule.

In the Configure Managed Servers screen, add the following managed servers:

Table 5-7 Configuring Managed Servers

Name

Listen Address

Listen Port

SSL Listen Port

SSL Enabled

WLS_SOA1

SOAHOST1VHN1

8001

n/a

No

WLS_SOA2

SOAHOST2VHN1

8001

n/a

No

Do not delete any server that appears. You can modify the servers. If you delete a server and add a new one, targeting fails.

Note:

Although the standard recommendation is to run custom applications and other systems in a separate WebLogic Managed Server, the creation of the custom WLS managed servers described in Figure 5-41 is not addressed here.

Click Next.

In the Configure Clusters screen, add the following cluster:

Name: SOA_Cluster

Cluster Messaging Mode: unicast

Multicast Address: N/A

Multicast Port: N/A

Cluster Address: Leave empty

Click Next.

In the Assign Servers to Clusters screen, assign the following servers to SOA_Cluster:

WLS_SOA1

WLS_SOA2

Click Next.

In the Configure Machines screen:

Delete the LocalMachine that appears by default.

Click the Unix Machine tab, and add the following machines:

Table 5-8 Configuring Machines

Name

Node Manager Listen Address

SOAHOST1

Hostname of SOAHOST1

SOAHOST2

Hostname of SOAHOST2

Leave all other fields as their default values, and click Next.

In the Assign Servers to Machines screen, assign servers to machines as follows:

The multicast and unicast addresses are different from the ones used by the WebLogic Server cluster for cluster communication. SOA guarantees that composites are deployed to members of a single WebLogic Server cluster even though the communication protocol for the two entities (the WebLogic Server cluster and the groups to which composites are deployed) are different.

5.13.5 Creating boot.properties for the Administration Server on SOAHOST1

This is an optional step for enabling the Administration Server to start without prompting you for the administrator username and password. Create a boot.properties file for the Administration Server on SOAHOST1.

5.13.7 Disabling Host Name Verification for the Administration Server and the WLS_SOAn Managed Servers

This step is required if you have not set up the appropriate certificates for hostname verification between the Administration Server and Node Manager. If SSL is not set up, you receive an error message unless you disable hostname verification.

You can re-enable hostname verification when you have set up SSL communication between the Administration Server and the Node Manager.

To disable hostname verification:

In the Administration Console, select Servers, and then AdminServer.

Select SSL, and then Advanced.

In the Change Center, click Lock & Edit.

Set Hostname Verification to None.

Save and activate the changes.

Repeat these steps for the WLS_SOA1 and WLS_SOA2 servers.

Restart the Administration Server.

5.13.8 Configuring Oracle Coherence for Deploying Composites

Although deploying composites uses multicast communication by default, Oracle recommends using unicast communication for SOA high availability. Use unicast if you disable multicast communication for security reasons.

Note:

An incorrect configuration of the Oracle Coherence framework that is used for deployment may prevent the SOA system from starting. The deployment framework must be properly customized for the network environment on which the SOA system runs. Oracle recommends the following configuration described in this section.

Enabling Communication within Clusters Using Unicast Communication

Multicast communication enables Oracle SOA Suite to discover all of the members of a cluster to which to it deploys composites dynamically. However, unicast communication does not enable nodes to discover other cluster members in this way. Consequently, you must specify the nodes that belong to the cluster. You do not need to specify all of the nodes of a cluster, however. You need only specify enough nodes so that a new node added to the cluster can discover one of the existing nodes. As a result, when a new node has joined the cluster, it is able to discover all of the other nodes in the cluster. Additionally, in a configuration where multiple IPs are available in the same node, you must configure Oracle Coherence to use a specific hostname to create Oracle Coherence cluster.

Tip:

To guarantee high availability during deployments of SOA composites, specify enough nodes so that at least one of them is running at any given time.

Specify the nodes using the tangosol.coherence.wka<n> system property, where <n> is a number between 1 and 9. You can specify up to 9 nodes. Start the numbering at 1. This numbering must be sequential and must not contain gaps. In addition, specify the hostname used by Oracle Coherence to create a cluster through the tangosol.coherence.localhost system property. This hostname should be the virtual hostname used by the SOA servers that maps the corresponding listener addresses (VIP1 and VIP2). Set this property by adding the -Dtangosol.coherence.localhost parameters to the Arguments field of the Administration Console's Server Start tab (Figure 5-45).

Figure 5-45 Setting the Hostname Using the Start Server Tab of the Administration Console

The Coherence cluster used for deployment uses port 8088 by default. This port can be changed by specifying a different port (for example, 8089) with the -Dtangosol.coherence.wkan.port and -Dtangosol.coherence.localport startup parameters. For example:

WLS_SOA1 (enter the following into the Arguments field on a single line, without a carriage return):

The multicast and unicast addresses are different from the ones used by the WebLogic Server cluster for cluster communication. SOA guarantees that composites are deployed to members of a single WebLogic Server cluster even though the communication protocol for the two entities (the WebLogic Server cluster and the groups to which composites are deployed) are different.

Do not copy the text from above to your Administration Console's arguments text field, which can result in HTML tags being inserted in the Java arguments. The text should not include any text or characters other than the ones shown above.

5.13.9 Starting the System in SOAHOST1

This section describes how to start Node Manager on SOAHOST1 and how to start and validate the WLS_SOA1 managed server on SOAHOST1.

5.13.9.1 Starting Node Manager on SOAHOST1

To start Node Manager on SOAHOST1:

Run the setNMProps.sh script, which is located in the ORACLE_COMMON_HOME/common/bin directory, to set the StartScriptEnabled property to true before starting Node Manager

SOAHOST1> cd ORACLE_COMMON_HOME/common/bin
SOAHOST1> ./setNMProps.sh

Note:

You must use the StartScriptEnabled property to avoid class loading failures and other problems.

Start Node Manager:

SOAHOST1> cd WL_HOME/server/bin
SOAHOST1> ./startNodeManager.sh

5.13.9.2 Starting and Validating the WLS_SOA1 Managed Server

To start the WLS_SOA1 managed server and check that it is configured correctly:

Start the WLS_SOA1 managed server using the Administration Console.

When WLS_SOA1 starts, the following URLs become available:

http://SOAHOST1VHN1:8001/b2bconsole

Verify login to B2B console

http://SOAHOST1VHN1:8001/integration/worklistapp

Verify login to worklist console

http://SOAHOST1VHN1:8001/wsm-pm

Verify the policy validator link.

http://SOAHOST1VHN1:8001/soa/composer

http://SOAHOST1VHN1:8001/soa-infra

http://SOAHOST1VHN1:8001/bpm/composer (ONLY FOR BPM Systems)

http://SOAHOST1VHN1:8001/bpm/workspace (ONLY FOR BPM Systems)

5.13.10 Propagating the Domain Configuration to SOAHOST2 with pack/unpack Utilities

To propagate the domain configuration to SOAHOST2 using pack/unpack utilities:

When you run the Fusion Middleware Configuration Wizard after a pack/unpack procedure, products that already exist in the original domain and appear in the Select Extension Source screen, are not selected or greyed out.

Templates created with the pack command are considered user-created templates. They are treated differently from default templates. User-created templates are self-contained and do not contain any information related to the creation of the original domain.

5.13.11 Extracting XEngine Files in the Second Node

To enable B2B's XEngine in the second node, it is required to extract the content of the ZEngine tar manually:

5.13.14 Validating Access Through Oracle HTTP Server

Verify that the SOA Servers status is reported as "Running" in the Administration Console. If the server is shown as "Starting" or "Resuming," wait for the server status to change to "Started." If another status is reported (such as "Admin" or "Failed"), check the server output log files for errors. Verify that you can access these URLS:

http://WEBHOST1:7777/wsm-pm

http://WEBHOST2:7777/wsm-pm

http://WEBHOST1:7777/soa-infra

http://WEBHOST2:7777/soa-infra

http://WEBHOST1:7777/soa/composer

http://WEBHOST2:7777/soa/composer

http://WEBHOST1:7777/integration/worklistapp

http://WEBHOST2:7777/integration/worklistapp

http://WEBHOST1:7777/sdpmessaging/userprefs-ui

http://WEBHOST2:7777/sdpmessaging/userprefs-ui

http://WEBHOST1:7777/b2bconsole

http://WEBHOST2:7777/b2bconsole

http://WEBHOST1:7777/bpm/composer (ONLY FOR BPM Systems)

http://WEBHOST2:7777/bpm/composer (ONLY FOR BPM Systems)

http://WEBHOST1:7777/bpm/workspace (ONLY FOR BPM Systems)

http://WEBHOST2:7777/bpm/workspace (ONLY FOR BPM Systems)

Verify these URLs also using your load balancing router address:

http://soa.mycompany.com:80/wsm-pm

http://soa.mycompany.com:80/soa-infra

http://soa.mycompany.com:80/soa/composer

http://soa.mycompany.com:80/integration/worklistapp

http://soa.mycompany.com:80/sdpmessaging/userprefs-ui

http://soa.mycompany.com:80/b2bconsole

http://soa.mycompany.com:80/bpm/composer (ONLY FOR BPM Systems)

http://soa.mycompany.com:80/bpm/workspace (ONLY FOR BPM Systems)

Follow these instructions to ensure that routing and failover from the HTTP Server to the SOA_CLuster is working correctly:

While WLS_SOA2 is running, stop WLS_SOA1 from the Administration Console.

Access the following URLs and verify the appropriate functionality:

WEBHOST1:7777/wsm-pm

WEBHOST1:7777/soa-infra

WEBHOST1:7777/soa/composer

WEBHOST1:7777/integration/worklistapp

WEBHOST1:7777/sdpmessaging/userprefs-ui

WEBHOST1:7777/b2bconsole

WEBHOST1:7777/bpm/composer (ONLY FOR BPM Systems)

WEBHOST1:7777/bpm/workspace (ONLY FOR BPM Systems)

Start WLS_SOA1 from the Administration Console.

Stop WLS_SOA2.

Access the URLs in Step 2 above again and verify the appropriate functionality:

5.13.15 Configuring JMS Persistence Store as Shared Across the Servers

Configure the location for all persistence stores to a directory visible from both nodes. Change all persistent stores to use this shared base directory.

To enable resume of pending JMS messages, you must specify a location on a persistent storage solution (NAS, SAN) that is available to other servers in the cluster. Therefore, the directory that you enter must be accessible by both WLS_SOA1 and WLS_SOA2. This directory must exist before the server is restarted.

Each server has a transaction log, which stores information about committed transactions coordinated by the server that may not have been completed. WebLogic Server uses the transaction log when recovering from system fails or network failures. To take advantage of the migration capability of the Transaction Recovery Service for servers in a cluster, you must store the transaction log in a location that is available to a server and its backup servers, preferably on a dual-ported SCSI disk, or a Storage Area Network (SAN). To configure the default persistent store:

In the Domain Structure tree, expand Environment and select Servers.

Select the server you want to modify.

Select the Configuration, Services tab.

Under Default Store, in the Directory field, enter the path to the folder where you want the default persistent store to store its data files.

To enable migration of the Transaction Recovery Service, you must specify a location on a persistent storage solution that is available to other servers in the cluster. Therefore, the directory that you enter must be accessible by both WLS_SOA1 and WLS_SOA2. This directory must exist before the server is restarted.

5.13.17 Setting the Front End HTTP Host and Port

You must set the front end HTTP host and port for Oracle WebLogic Server cluster:

In the Administration Console, in the Change Center section, click Lock & Edit.

In the left pane, select Environment > Clusters.

Select the SOA_Cluster.

Select HTTP.

Set the values for the following:

Frontend Host: soa.mycompany.com

Frontend HTTP Port: 80

Frontend HTTPS Port: leave it blank

Note:

Make sure this address is correct and available (the load balancing router is up). An incorrect value, for example, http:// in the address, or trailing / in the hostname, may prevent the SOA system from being accessible even when using the virtual IPs to access it.

Click Save.

To activate the changes, click Activate Changes in the Change Center section of the Administration Console.

If you have started the server before, notice this change requires a restart of the managed servers in the cluster.

Note:

The SOA system calculates the callback server URL as follows:

If a request to SOA originates from an external or internal service is originating a request to SOA, then SOA uses the callback server URL specified by the client.

If a request to an external or internal service originates from SOA, the callbackURL cannot be populated in the SOA request dynamically because SOA is the originator. Instead, callbackServerURL is used if it is specified as a binding property for the specific reference. (You can set this when modeling the composite or at runtime using the MBeans accessed through Oracle Enterprise Manager console.) This enables different service calls to have different callback server URLs. That is, a callback server URL from an external service is different than one to an internal service.

However, if the callbackServer URL is not specified as a binding property for that reference, then the system uses the callback server URL as specified in soa-infra-config. If the callback server URL is not specified in soa-infra-config, then the system uses the callback server URL as the front end host specified in WLS. If the front end host is not specified in WLS, the system uses the callback server URL as the local hostname as provided by WLS MBean APIs.

For SOA high availability installations frontended by Oracle HTTP Server, monitoring should be done on the Oracle HTTP Server ports of the real backend servers. This is the case when a deployment is using all the components deployed to the SOA Managed Server. A simple HTTP monitor that pings the HTTP/HTTPS port and expects a pre-determined response in turn should suffice. If only a specific SOA component is being used (such as B2B), then a monitor that does a deeper level check all the way to the managed server can be considered to validate the health of the component in use. Please check with your load balancer vendor on setting up the HTTP monitors with your load balancer.

If you do not set the front end HTTP host and port, you get the following message when trying to retrieve a document definition XSD from Oracle B2B:

An error occured while loading the document definitions.
java.lang.IllegalArgumentException: Cluster address must be set when
clustering is enabled.

You must set the WLS Cluster address for the SOA_Cluster if you are going to use direct binding to composites. To do this follow these steps:

In the Administration Console, in the Change Center section, click Lock & Edit.

In the left pane, choose Environment in the Domain Structure window and then choose Clusters. The Summary of Clusters page appears.

Select the SOA_Cluster cluster.

In the Configuration > General tab, enter the following in the Cluster Address field: SOAHOST1VHN1:8001,SOAHOST2VHN1:8001

Click Save.

To activate the changes, click Activate Changes in the Change Center section of the Administration Console.

Restart the servers to make the Frontend Host directive in the cluster effective.

Note:

For asynchronous request/response interactions over direct binding, the SOA composites need to provide the JNDI provider URL for the invoked service to look up the beans for callback.

If soa-infra config properties are not specified but the WebLogic Server cluster address is specified, the cluster address will be used to form the JNDI provider URL. This cluster address can be a single DNS name which maps to the clustered servers' IP-addresses or a comma separated list of server ip:port. Alternatively, if the soa-infra config property JndiProviderURL/SecureJndiProviderURL is explicitly set by users, it can be used also for the same purpose.

5.13.19 Deploying Applications

You can deploy SOA composite applications from Oracle Enterprise Manager Fusion Middleware Control Console with the Deploy SOA Composite wizard. Use the Deploy SOA Composite wizard to deploy any of the following:

A new SOA composite application for the first time

A new revision (for example, 2.0) alongside an older revision (for example, 1.0) without impacting the latter. The revision deployed last becomes the new default revision of that composite (unless you specify otherwise at a later step during deployment).

A bundle (ZIP file) containing multiple SOA composite application revisions (for example, revisions 2.0, 3.0, and 4.0) of a SOA composite application that already has a different revision that is currently deployed (for example, 1.0). This option enables you to deploy revisions 1.0, 2.0, 3.0, and 4.0 at the same time. The bundle can also contain revisions of different composites. There is no restriction that all revisions must be of the same composite application.

Deployment extracts and activates the composite application in SOA Service Infrastructure. Once an application is deployed, you can perform administration tasks, such as creating instances, configuring properties, monitoring performance, managing instance life cycles, and managing policies and faults.

Note:

To redeploy an existing revision of an application, do not use this wizard. Instead, use the Redeploy SOA Composite wizard.

To deploy applications:

Access the Deploy SOA Composite wizard through one of the following options:

In the Archive or Exploded Directory section, specify the archive of the SOA composite application to deploy. The archive contains the project files of the composite to be deployed (for example, HelloWorld_rev1.0.jar for a single archive or OrderBooking_rev1.0.zip for multiple archives).

In the Configuration Plan section, optionally specify the configuration plan to include with the archive. The configuration plan enables you to define the URL and property values to use in different environments. During process deployment, the configuration plan is used to search the SOA project for values that must be replaced to adapt the project to the next target environment.

Click Next.

The Select Target page appears.

Select the WebLogic Server or cluster to which to deploy the SOA composite application archive. You can deploy to multiple servers and clusters.

Click Next and review your selections.

Select whether or not to deploy the SOA composite application as the default revision. The default revision instantiates when a new request comes in.

The WLS_SOA1 managed server is configured to restart on SOAHOST2 in case of failure, and the WLS_SOA2 managed server is configured to restart on SOAHOST1 in case of failure. For this configuration the WLS_SOA1 and WLS_SOA2 servers listen on specific floating IPs that are failed over by WLS Server Migration.

The following steps enable server migration, enabling a managed server to failover to another node in the case of server or process failure.

Configuring server migration for the WebLogic managed servers consists of these steps:

5.13.20.1 Setting Up a User and Tablespace for the Server Migration Leasing Table

The first step to set up a user and tablespace for the server migration leasing table:

Note:

If other servers in the same domain are already configured with server migration, you can use the same tablespace and data sources. In this case, you don't need to recreate the data sources and multi data source for database leasing, however, you must retarget them to the clusters you're configuring for server migration.

Create a tablespace named leasing. For example, log on to SQL*Plus as the sysdba user and run the following command:

You create a data source to each of the Oracle RAC database instances during the process of setting up the multi data source, both for these data sources and the global leasing multi data source. When you create a data source:

Ensure that this is a non-XA data source.

Use Oracle's Driver (Thin) Version 9.0.1, 9.2.0, 10, 11.

Do not use any type of distributed transaction emulation/participation algorithm for the data source (do not choose the Supports Global Transactions option, or the Logging Last Resource, Emulate Two-Phase Commit, or One-Phase Commit options of the Supports Global Transactions option), and specify a service name for your database. Data sources do not require support for global transactions.

Target these data sources to the cluster(s).

Make sure the data source's connection pool initial capacity is set to 0 (zero). To do this, select Services, JDBC and then Datasources. In the Datasources screen, click the Datasource Name, click the Connection Pool tab, and enter 0 (zero) in the Initial Capacity field.

Ensure that an ONS daemon is running on your database servers at all times. To start the ONS daemon on a database server, run the onsctl command: start

Creating a GridLink Data Source

To create a GridLink Data Source:

Log in to the Administration Console.

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.

Service Name: Enter the service name of the database. For a GridLink data source, you must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

Note:

The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, example.com

Host Name - The DNS name or IP address of the server that hosts the database. For an Oracle GridLink service-instance connection, this must be the same for each data source in a given multi data source.

Port - The port on which the database server listens for connections requests.

On the Test GridLink Database Connection page, review the connection parameters and click Test All Listeners.

Oracle WebLogic attempts to create a connection from the Administration Server to the database. Results from the connection test appear at the top of the page. If the test is unsuccessful, correct any configuration errors and retry the test.

Click Next.

In the ONS Client Configuration page, do the following:

Select Fan Enabled to subscribe to and process Oracle FAN events.

In ONS host and port, enter a comma-separate list of ONS daemon listen addresses and ports for receiving ONS-based FAN events. You can use Single Client Access Name (SCAN) addresses to access FAN notifications.

In the Select Targets page, select Dept1_Cluster1 as the target and All Servers in the cluster.

Click Finish.

Click Activate Changes.

Creating a Multi Data Source

To create a multi data source:

Log into the Administration Console using the Admin credentials.

In the Domain Structure window, expand the Services node then expand the DataSource node.

Click Lock and Edit in the Change Center.

Click New then click Multi Data Sources.

Enter leasing as the name.

Enter jdbc/leasing as the JNDI name.

Select Failoveras algorithm (default). Click Next.

Select the target cluster(s). Click Next.

Select non-XA driver (the default). Click Next.

Click Create New Data Source.

Enter leasing-rac0 as the name. Enter jdbc/leasing-rac0 as the JNDI name. Enter oracle as the database type. For the driver type, select Oracle Driver (Thin) for RAC server-Instance connection Version 10,11.

Note:

When you create multi data sources for the leasing table, enter names in the format <MultiDS>-rac0, <MultiDS>-rac1, and so on.

Click Next.

Deselect Supports Global Transactions. Click Next.

Enter the service name, database name (that is, the RAC Node instance name such as racdb1, racdb2), host port, and password for your leasing schema. Click Next.

Click Create a New Data Source for the second instance of your Oracle RAC database, target it to the cluster(s), repeating the steps for the second instance of your Oracle RAC database.

Add the second data source to your multi data source.

Save and click Activate Changes.

5.13.20.3 Editing Node Manager's Properties File

You must edit the nodemanager.properties file to add the following properties for each node where you configure server migration:

Interface=eth0
NetMask=255.255.255.0
UseMACBroadcast=true

Interface: Specifies the interface name for the floating IP (such as eth0).

Note:

Do not specify the sub interface, such as eth0:1 or eth0:2. This interface is to be used without the :0 or :1. The Node Manager's scripts traverse the different:X enabled IPs to determine which to add or remove. For example, the valid values in Linux environments are eth0, eth1, or, eth2, eth3, ethn, depending on the number of interfaces configured.

Note:

In Windows, the Interface should be set to the Network Interface Name. For example: Interface="Local Area Connection".

NetMask: Net mask for the interface for the floating IP. The net mask should the same as the net mask on the interface; 255.255.255.0 is an example. The actual value depends on your network.

UseMACBroadcast: Specifies whether or not to use a node's MAC address when sending ARP packets, that is, whether or not to use the -b flag in the arping command.

Verify in Node Manager's output (shell where Node Manager starts) that these properties are being used or problems may arise during migration. You should see an entry similar to the following in Node Manager's output:

...
StateCheckInterval=500
Interface=eth0
NetMask=255.255.255.0
...

Note:

The following steps are not required if the server properties (start properties) are properly set and Node Manager can start the servers remotely.

Set the following property in the nodemanager.properties file:

StartScriptEnabled: Set this property to true to enable Node Manager to start the managed servers.

Start Node Manager on OIMHOST1 and OIMHOST2 by running the startNodeManager.sh script in the WL_HOME/server/bin directory.

Note:

When running Node Manager from a shared storage installation, multiple nodes start using the same nodemanager.properties file. However, each node may require different NetMask or Interface properties. In this case, specify individual parameters on a per-node basis using environment variables. For example, to use a different interface (eth3) in HOSTn, use the Interface environment variable as follows: HOSTn> export JAVA_OPTIONS=-DInterface=eth3 and start Node Manager after the variable is set in the shell.

5.13.20.4 Setting Environment and Superuser Privileges for the wlsifconfig.sh Script

To set environment and superuser privileges for the wlsifconfig.sh script:

Note:

On Windows, the script is wlsifconfig.cmd and users with the Administrator privilege can run it.

Ensure that your PATH environment variable includes these files:

Table 5-9 Files Required for the PATH Environment Variable

File

Located in this directory

wlsifconfig.sh

DOMAIN_HOME/bin/server_migration

wlscontrol.sh

WL_HOME/common/bin

nodemanager.domains

WL_HOME/common

Grant sudo configuration for the wlsifconfig.sh script.

Configure sudo to work without a password prompt.

For security reasons, Oracle recommends restricting to the subset of commands required to run the wlsifconfig.sh script. For example, perform the following steps to set the environment and superuser privileges for the wlsifconfig.sh script:

Grant sudo privilege to the WebLogic user (oracle) with no password restriction, and grant execute privilege on the /sbin/ifconfig and /sbin/arping binaries.

Ensure that the script is executable by the WebLogic user. The following example shows an entry inside /etc/sudoers granting sudo execution privilege for oracle and also over ifconfig and arping:

oracle ALL=NOPASSWD: /sbin/ifconfig,/sbin/arping

Note:

Ask the system administrator for the sudo and system rights as appropriate to this step.

5.13.20.5 Configuring Server Migration Targets

You first assign all the available nodes for the cluster's members and then specify candidate machines (in order of preference) for each server that is configured with server migration. To configure cluster migration in a migration in a cluster:

Log into the Administration Console.

In the Domain Structure window, expand Environment and select Clusters.

Click the cluster you want to configure migration for in the Name column.

Click the Migration tab. Click Lock and Edit.

In the Available field, select the machine to which to enable migration and click the right arrow.

Select the data source to use for automatic migration. In this case, select the leasing data source.

Click Save. Click Activate Changes.

Set the candidate machines for server migration. You must perform this task for all managed servers as follows:

In the Domain Structure window of the Administration Console, expand Environment and select Servers.

Tip:

Click Customize this table in the Summary of Servers page and move Current Machine from the Available window to the Chosen window to view the machine that the server runs on. This will be different from the configuration if the server migrates automatically.

Select the server that you want to configure migration for.

Click the Migration tab.

In the Available field, located in the Migration Configuration section, select the machines you want to enable migration to and click the right arrow.

Restart the administration server, Node Managers, and the servers for which server migration has been configured.

5.13.20.6 Testing the Server Migration

To verify that server migration is working properly, follow these steps:

From SOAHOST1:

Force stop the WLS_SOA1 managed server.

To do this, run this command:

SOAHOST1> kill -9<pid>

pid specifies the process ID of the managed server. You can identify the pid in the node by running this command:

SOAHOST1> ps -ef | grep WLS_SOA1

Note:

For Windows, you can terminate the Managed Server by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server, run the following command and identify the pid of the WLS_SOA1 Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager console: you should see a message indicating that WLS_SOA1's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_SOA1. Node Manager waits for a fence period of 30 seconds before trying this restart.

Once Node Manager restarts the server, stop it again. Now Node Manager should log a message indicating that the server will not be restarted again locally.

From SOAHOST2:

Watch the local Node Manager console. After 30 seconds since the last try to restart WLS_SOA1 on Node 1, Node Manager on Node 2 should prompt that the floating IP for WLS_SOA1 is being brought up and that the server is being restarted in this node.

On Windows, when you manually shut down multiple servers at the same time on the same machine and then, on another machine, attempt to start one of the servers that was shut down, the IP bind may not work. This happens because the original machine still has claim to the IP address even though netsh reports that the IP address has been removed.

To resolve this, you must check the network configuration either by using the ipconfig utility or Windows Network Configuration. Either of these may show that one of the virtual/floating IP addresses is still configured even though the servers are shut down. You can then use Windows Network Configuration to remove the IP address using the following procedure:

After a server is migrated, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager will start the managed server on the machine to which it was originally assigned.

5.13.21 Scaling the Topology

You can scale out or scale up this high availability topology. When you scale up the topology, you add new managed servers to nodes that are already running one or more managed servers. When you scale out the topology, you add new managed servers to new nodes.

In this case, you already have a node that runs a managed server configured with SOA components. The node contains a Middleware home, an Oracle HOME (SOA) and a domain directory for existing managed servers.

Use the existing installations (the Middleware home, and domain directories) to create new WLS_SOA servers; do not install SOA binaries in a new location or to run pack and unpack.

To scale up the topology:

Using the Administration Console, clone WLS_SOA1 into a new managed server. The source managed server to clone should be one that already exists on the node where you want to run the new managed server.

To clone a managed server:

Select Environment -> Servers from the Administration Console.

Select the managed server that you want to clone (for example, WLS_SOA1).

Select Clone.

Name the new managed server WLS_SOAn, where n is a number to identify the new managed server.

The rest of the steps assume that you are adding a new server to SOAHOST1, which is already running WLS_SOA1.

For the listen address, assign the virtual hostname to use for this new managed server. If you plan to use server migration as recommended for this server, this virtual hostname enables it to move to another node. The virtual hostname should be different from the one that the managed server uses that is already running.

To set the managed server listen address:

Log into the Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers.

Select the managed server whose listen address you want to update in the Names column of the table.

Set the Listen Address to SOAHOST1VHN1.

Click Save.

Save and activate the changes.

The changes do not take effect until the managed server restarts.

Update the cluster address to include the new server:

Select Environment -> Cluster in the Domain Structure window in the Administration Console.

Click on the SOA_Cluster server.

In the Change Center, click Lock & Edit.

Add the new server's address and port to the Cluster Address field. For example:

SOAHOST1VHN1:8001,SOAHOST2VHN1:8001,SOAHOST1VHN2:8001

Create JMS Servers for SOA, BPM (if applicable) and UMS on the new managed server.

Use the Administration Console to create a new persistent store for the new SOAJMSServer and name it, for example, SOAJMSFileStore_N. Specify the path for the store. This should be a directory on shared storage, as recommended in Section 5.13.1.3, "Shared Storage Prerequisites":

Note:

This directory must exist before the managed server is started or the start operation fails.

ORACLE_BASE/admin/DOMAIN_NAME/cluster_name/jms/SOAJMSFileStore_N

Create a new JMS Server for SOA, for example, SOAJMSServer_N. Use the SOAJMSFileStore_N for this JMSServer. Target the SOAJMSServer_N Server to the recently created Managed Server (WLS_SOAn).

This directory must exist before the managed server is started or the start operation fails.

You can also assign SOAJMSFileStore_N as store for the new UMS JMS Servers. For the purpose of clarity and isolation, individual persistent stores are used in the following steps.

Create a new JMS Server for UMS, for example, UMSJMSServer_N. Use the UMSJMSFileStore_N for this JMSServer. Target the UMSJMSServer_N Server to the recently created Managed Server (WLS_SOAn).

(ONLY FOR BPM Systems) Create a new persistence store for the new BPMJMSServer, for example, BPMJMSFileStore_N Specify the path for the store. This should be a directory on shared storage as recommended in Section 5.13.1.3, "Shared Storage Prerequisites":

ORACLE_BASE/admin/domain_name/cluster_name/jms/BPMJMSFileStore_N.

Note:

This directory must exist before the managed server is started or the start operation fails.

You can also assign SOAJMSFileStore_N as store for the new BPM JMS Servers. For the purpose of clarity and isolation, individual persistent stores are used in the following steps.

(ONLY FOR BPM Systems) Create a new JMS Server for BPM, for example, BPMJMSServer_N. Use the BPMJMSFileStore_N for this JMSServer. Target the BPMJMSServer_N Server to the recently created Managed Server (WLS_SOAn).

Update the SubDeployment targets for the SOA JMS Module to include the recently created SOA JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click SOAJMSModule (a hyperlink in the Names column of the table). The Settings page for SOAJMSModule appears. Click the SubDeployments tab. The subdeployment module for SOAJMS appears.

Note:

This subdeployment module name is a random name in the form of SOAJMSServerXXXXXX resulting from the Configuration Wizard JMS configuration for the first two servers, WLS_SOA1 and WLS_SOA2.

Click the SOAJMSServerXXXXXX subdeployment. Add the new JMS Server for SOA called SOAJMSServer_N to this subdeployment. Click Save.

Update the SubDeployment targets for the UMSJMSSystemResource to include the recently created UMS JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click UMSJMSSystemResource (represented as a hyperlink in the Names column of the table). The Settings page for UMSJMSSystemResource appears. Click the SubDeployments tab. The subdeployment module for UMSJMS appears.

Note:

This subdeployment module name is a random name in the form of UCMJMSServerXXXXXX resulting from the Config Wizard JMS configuration for the first two servers (WLS_SOA1 and WLS_SOA2).

Click the UMSJMSServerXXXXXX subdeployment. Add the new JMS Server for UMS called UMSJMSServer_N to this subdeployment. Click Save.

(ONLY FOR BPM Systems) Update the SubDeployment targets for BPMJMSModule to include the recently created BPM JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click BPMJMSSystemResource (represented as a hyperlink in the Names column of the table). The Settings page for BPMJMSSystemResource appears. Click the SubDeployments tab. The subdeployment module for BPMJMS appears.

Note:

This subdeployment module name is a random name in the form of BPMJMSServerXXXXXX resulting from the Config Wizard JMS configuration for the first two servers (WLS_SOA1 and WLS_SOA2).

Click the BPMJMSServerXXXXXX subdeployment. Add the new JMS Server for BPM called BPMJMSServer_N to this subdeployment. Click Save.

Only the localhost field needs to be changed for the server. Replace the localhost with the listen address of the new server added, for example:

Dtangosol.coherence.localhost=SOAHOST1VHNn

Configure TX persistent store for the new server. This should be a location visible from other nodes as indicated in the recommendations about shared storage.

From the Administration Console, select Server_name > Services tab. Under Default Store, in Directory, enter the path to the folder where you want the default persistent store to store its data files.

Disable hostname verification for the new managed server. Before starting and verifying the WLS_SOAn managed server, you must disable hostname verification. You can re-enable it after you have configured server certificates for the communication between the Oracle WebLogic Administration Server and the Node Manager in SOAHOSTn. If the source server from which the new one has been cloned had already disabled hostname verification, these steps are not required (the hostname verification settings is propagated to the cloned server).

To disable hostname verification:

In the Oracle Enterprise Manager Console, select the Administration Console.

Expand the Environment node in the Domain Structure window.

Click Servers. The Summary of Servers page appears.

Select WLS_SOAn in the Names column of the table. The Settings page for the server appears.

Click the SSL tab.

Click Advanced.

Set Hostname Verification to None.

Click Save.

Start and test the new managed server from the Administration Console.

Shut down the existing managed servers in the cluster.

Ensure that the newly created managed server, WLS_SOAn, is up.

Access the application on the newly created managed server (http://vip:port/soa-infra). The application should be functional.

Configure Server Migration for the new managed server.

Note:

Since this is a scale-up operation, the node should contain a Node Manager, an environment configured for server migration, and the floating IP for the new SOA managed server.

Configure server migration following these steps:

Log into the Administration Console.

In the left pane, expand Environment and select Servers.

Select the name of the new managed server for which you want to configure migration.

Click the Migration tab.

In the Available field, in the Migration Configuration section, select the machines to which to enable migration and click the right arrow. Select the same migration targets as for the servers that already exist on the node.

For example, for new managed servers on SOAHOST1, which is already running WLS_SOA1, select SOAHOST2. For new managed servers on SOAHOST2, which is already running WLS_SOA2, select SOAHOST1.

Make sure the appropriate resources are available to run the managed servers concurrently during migration.

Select the Automatic Server Migration Enabled option. This enables the Node Manager to start a failed server on the target node automatically.

Click Save.

Restart the Administration Server, managed servers, and Node Manager.

Test server migration for this new server. Follow these steps from the node where you added the new server:

Stop the WLS_SOAn managed server.

To do this, run kill -9 <pid> on the PID of the managed server. You can identify the PID of the node using ps -ef | grep WLS_SOAn.

Note:

For Windows, the Managed Server can be terminated by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server, run the following command and identify the pid of the WLS_SOAn Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager Console: you should see a message indicating that WLS_SOAn's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_SOAn. Node Manager waits for a fence period of 30 seconds before trying this restart.

Once Node Manager restarts the server, stop it again. Node Manager logs a message indicating that the server will not be restart again locally.

Note:

After a server migrates, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager starts the managed server on the machine it was originally assigned to.

5.13.21.2 Scaling Out the Topology (Adding Managed Servers to New Nodes)

When you scale out the topology, you add new managed servers configured with SOA to new nodes.

Before performing the steps in this section, check that you meet these requirements:

There must be existing nodes running managed servers configured with SOA within the topology.

The new node can access the existing home directories for WebLogic Server and SOA. (Use the existing installations in shared storage for creating a new WLS_SOA or WLS_WSM managed server. You do not need to install WebLogic Server or SOA binaries in a new location but you do need to run pack and unpack to bootstrap the domain configuration in the new node.)

When multiple servers share ORACLE_HOME or WL_HOME in different nodes, Oracle recommends keeping the Oracle Inventory and Middleware home list in those nodes updated for consistency in the installations and application of patches. To update the oraInventory in a node and attach an installation in a shared storage to it, use ORACLE_HOME/oui/bin/attachHome.sh. To update the Middleware home list to add or remove a WL_HOME, edit the user_home/bea/beahomelist file. See the following steps.

Follow these steps for scaling out the topology:

On the new node, mount the existing Middleware home, which should include the SOA installation and the domain directory, and ensure that the new node has access to this directory, just like the rest of the nodes in the domain.

To attach ORACLE_HOME in shared storage to the local Oracle Inventory, run the following command:

To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.

Log in to the Oracle WebLogic Administration Console.

Create a new machine for the new node that will be used, and add the machine to the domain.

Update the machine's Node Manager's address to map the IP of the node that is being used for scale out.

Use the Administration Console to clone WLS_SOA1 into a new managed server. Name it WLS_SOAn, where n is a number and assign it to the new machine.

Note:

These steps assume that you are adding a new server to node n, where no managed server was running previously.

For the listen address, assign the virtual hostname to use for this new managed server. If you are planning to use server migration as recommended for this server, this virtual hostname will enable it to move to another node. The virtual hostname should be different from the one used by the managed server that is already running.

Perform these steps to set the managed server listen address:

Log into the Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers.

Select the managed server whose listen address you want to update in the Names column of the table.

Set the Listen Address to SOAHOSTnVHN1.

Click Save.

Save and activate the changes.

The changes will not take effect until the managed server is restarted.

Update the cluster address to include the new server:

Select Environment -> Cluster from the Administration Console.

Click on the SOA_Cluster server. The Settings Screen for the SOA_Cluster appears.

In the Change Center, click Lock & Edit.

Add the new server's address and port to the Cluster Address field. For example:

SOAHOST1VHN1:8001,SOAHOST2VHN1:8001,SOAHOSTNVHN1:8001

Create JMS servers for SOA, BPM (if applicable), and UMS on the new managed server.

Use the Administration Console to create a new persistent store for the new SOAJMSServer and name it, for example, SOAJMSFileStore_N. Specify the path for the store. This should be a directory on shared storage as recommended in Section 5.13.1.3, "Shared Storage Prerequisites."

ORACLE_BASE/admin/domain_name/cluster_name/jms/SOAJMSFileStore _N

Note:

This directory must exist before the managed server is started or the start operation fails.

Create a new JMS Server for SOA, for example, SOAJMSServer_N. Use the SOAJMSFileStore_N for this JMSServer. Target the SOAJMSServer_N Server to the recently created managed server (WLS_SOAn).

Create a new persistence store for the new UMSJMSServer, and name it, for example, UMSJMSFileStore_N. Specify the path for the store. This should be a directory on shared storage as recommended in Section 5.13.1.3, "Shared Storage Prerequisites."

ORACLE_BASE/admin/domain_name/cluster_name/jms/UMSJMSFileStore _N

Note:

This directory must exist before the managed server is started or the start operation fails.

Note:

It is also possible to assign SOAJMSFileStore_N as the store for the new UMS JMS Servers. For the purpose of clarity and isolation, individual persistent stores are used in the following steps.

Create a new JMS Server for UMS: for example, UMSJMSServer_N. Use the UMSJMSFileStore_N for this JMS Server. Target the UMSJMSServer_N Server to the recently created managed server (WLS_SOAn).

(ONLY FOR BPM Systems) Create a new persistence store for the new BPMJMSServer, and name it, for example, BPMJMSFileStore_N. Specify the path for the store. This should be a directory on shared storage as recommended in Section 5.13.1.3, "Shared Storage Prerequisites."

ORACLE_BASE/admin/domain_name/cluster_name/jms/BPMJMSFileStore _N

Note:

This directory must exist before the managed server is started or the start operation fails.

Note:

It is also possible to assign SOAJMSFileStore_N as the store for the new BPM JMS Servers. For the purpose of clarity and isolation, individual persistent stores are used in the following steps.

(ONLY FOR BPM Systems) Create a new JMS Server for BPM: for example, BPMJMSServer_N. Use the BPMJMSFileStore_N for this JMS Server. Target the BPMJMSServer_N Server to the recently created managed server (WLS_SOAn).

Update the SubDeployment targets for the SOA JMS Module to include the recently created SOA JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click SOAJMSModule (a hyperlink in the Names column). In the Settings page for SOAJMSModule, open the SubDeployments tab. The subdeployment module for SOAJMS appears.

Note:

This subdeployment module name is a random name in the form of SOAJMSServerXXXXXX resulting from the Config Wizard JMS configuration for the first two servers (WLS_SOA1 and WLS_SOA2).

Click the SOAJMSServerXXXXXX subdeployment. Add the new JMS Server for SOA called SOAJMSServer_N to this subdeployment. Click Save.

Update the SubDeployment targets for UMSJMSSystemResource to include the recently created UMS JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click UMSJMSSystemResource (represented as a hyperlink in the Names column of the table). The Settings page for UMSJMSSystemResource appears. Open the SubDeployments tab. The subdeployment module for UMSJMS appears

Note:

This subdeployment module is a random name in the form of UMSJMSServerXXXXXX resulting from the Config Wizard JMS configuration for the first two servers (WLS_SOA1 and WLS_SOA2).

Click the UMSJMSServerXXXXXX subdeployment. Add the new JMS Server for UMS called UMSJMSServer_N to this subdeployment. Click Save.

(ONLY FOR BPM Systems) Update the SubDeployment Targets for BPMJMSModule to include the recently created BPM JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click BPMJMSSystemResource (represented as a hyperlink in the Names column of the table). The Settings page for BPMJMSSystemResource appears. Click the SubDeployments tab. The subdeployment module for BPMJMS appears.

Note:

This subdeployment module is a random name in the form of BPMJMSServerXXXXXX resulting from the Config Wizard JMS configuration for the first two servers (WLS_SOA1 and WLS_SOA2).

Click the BPMJMSXXXXXX subdeployment. Add the new JMS Server for BPM called BPMJMSServer_N to this subdeployment. Click Save.

Run the pack command on SOAHOST1 to create a template pack as follows:

Only the localhost field needs to be changed for the server. Replace the localhost with the listen address of the new server added, for example:

Dtangosol.coherence.localhost=SOAHOSTnVHN1

Configure TX persistent store for the new server. This should be a location visible from other nodes as indicated in the recommendations about shared storage.

From the Administration Console, select Server_name > Services tab. Under Default Store, in Directory, enter the path to the folder where you want the default persistent store to store its data files.

Disable hostname verification for the new managed server. Before starting and verifying the WLS_SOAN managed server, you must disable hostname verification. You can re-enable it after you have configured server certificates for the communication between the Oracle WebLogic Administration Server and the Node Manager in SOAHOSTn. If the source server from which the new one has been cloned had already disabled hostname verification, these steps are not required (the hostname verification setting is propagated to the cloned server).

Start the Node Manager on the new node. To start the Node Manager, use the installation in shared storage from the existing nodes, and start Node Manager by passing the hostname of the new node as a parameter as follows:

SOAHOSTN> WL_HOME/server/bin/startNodeManager <new_node_ip>

Start and test the new managed server from the Administration Console:

Shut down all the existing managed servers in the cluster.

Ensure that the newly created managed server, WLS_SOAn, is running.

Access the application on the newly created managed server (http://vip:port/soa-infra). The application should be functional.

Configure server migration for the new managed server.

Note:

Since this new node is using an existing shared storage installation, the node is already using a Node Manager and an environment configured for server migration that includes netmask, interface, wlsifconfig script superuser privileges. The floating IP for the new SOA Managed Server is already in the new node.

Configure server migration following these steps:

Log into the Administration Console.

In the left pane, expand Environment and select Servers.

Select the server (represented as hyperlink) for which you want to configure migration from the Names column of the table. The Setting page for that server appears.

Click the Migration tab.

In the Available field, in the Migration Configuration section, select the machines to which to enable migration and click the right arrow.

Note:

Specify the least-loaded machine as the migration target for the new server. The required capacity planning must be completed so that this node has enough available resources to sustain an additional managed server.

Select the Automatic Server Migration Enabled option. This enables the Node Manager to start a failed server on the target node automatically.

Click Save.

Restart the Administration Server, managed servers, and Node Manager.

Test server migration for this new server. Follow these steps from the node where you added the new server:

1. Abruptly stop the WLS_SOAn managed server.

2. To do this, run "kill -9 <pid>" on the PID of the managed server. You can identify the PID of the node using "ps -ef | grep WLS_SOAn".

Note:

For Windows, the Managed Server can be terminated by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server run the following command and identify the pid of the WLS_SOAn Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

3. Watch the Node Manager Console: you should see a message indicating that WLS_SOA1's floating IP has been disabled.

4. Wait for the Node Manager to try a second restart of WLS_SOAn. Node Manager waits for a fence period of 30 seconds before trying this restart.

5. Once Node Manager restarts the server, stop it again. Now Node Manager should log a message indicating that the server will not be restarted again locally.

Note:

After a server is migrated, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager will start the managed server on the machine to which it was originally assigned.

5.14 Configuring High Availability for Oracle Service Bus, with SOA Service Infrastructure and Component Service Engines

The procedures described in this section include setting up Oracle Service Bus, and also other Service engines contained in the Oracle SOA Service Infrastructure system (such as Oracle BPEL/Oracle PM, Oracle Mediator, Oracle Human Workflow and Oracle Decision Services, as well as Oracle B2B and Oracle User Messaging Service).

Oracle strongly recommends reading the release notes for any additional installation and deployment considerations prior to starting the setup process.

Note:

The architectures and deployment procedures defined in this guide enable simple clustered deployments. The procedures described in these chapters can be used as a building block to enable this and other similar high availability topologies for these Fusion Middleware components. It is also expected that production deployments will use other required procedures, such as associating security policies with a centralized LDAP server. For complete details of secured, multi-tiered architectures, and deployment procedures, please refer to the Enterprise Deployment Guide for the component you are configuring.

Figure 5-47 represents the example architecture that the configuration steps in this section create.

Figure 5-47 shows a two-node Oracle Service Bus cluster running on two Oracle WebLogic Servers. The Oracle WebLogic Servers are frontended by Oracle HTTP Servers, which load balance incoming requests. A load balancer front ends the system and distributes incoming requests from clients to the two Oracle HTTP Servers. This configuration uses an Oracle RAC database to store reporting data from the JMS reporting provider, and two phase commit logs using LLR with the JTA transaction manager (only used by the reporting data provider). It may also contain heartbeat data from servers in the cluster (used in the WebLogic Server whole server migration feature). The persistent store for transaction and JMS stores is on shared storage. Virtual IP addresses (VIPs) provide manual failover for the Administration Server and for Oracle Service Bus managed servers (for server migration).

Note:

To keep the Oracle Service Bus high availability configuration figure above readable, the Oracle SOA Service Infrastructure high availability configuration is not shown. See Figure 5-5 to view the Oracle SOA Service Infrastructure high availability figure.

SQL>select version from sys.product_component_version where product like 'Oracle%';

5.14.1.2 VIP and IP Prerequisites

As shown in Table 5-10, you configure the Administration Server and the SOA and OSB managed servers to listen on different virtual IPs. This requires the provisioning of the corresponding VIP in the node and related host names in the DNS system in your network. Ensure that the different VIPS are available and are reachable from SOAHOST1, SOAHOST2, and the client machines before running the installation.

Table 5-10 Virtual Hosts

Virtual IP

VIP Maps to...

Description

VIP0

SOAHOST1VHN0

SOAHOST1VHN0 is the virtual hostname that is the listen address for the Administration Server and fails over with manual failover of the Administration Server. It is enabled on the node where the Administration Server process is running (SOAHOST1 by default).

VIP1

SOAHOST1VHN1

SOAHOST1VHN1 is the virtual hostname that maps to the listen address for WLS_SOA1 and fails over with server migration of this managed server. It is enabled on the node where the WLS_SOA1 process is running (SOAHOST1 by default).

VIP2

SOAHOST2VHN1

SOAHOST2VHN1 is the virtual hostname that maps to the listen address for WLS_SOA2 and fails over with server migration of this managed server. It is enabled on the node where the WLS_SOA2 process is running (SOAHOST2 by default).

VIP3

OSBHOST1VHN1

OSBHOST1VHN1 is the virtual hostname that maps to the listen address for WLS_OSB1 and fails over with server migration of this managed server. It is enabled on the node where the WLS_OSB1 process is running (SOAHOST1 by default).

VIP4

OSBHOST2VHN1

OSBHOST2VHN1 is the virtual hostname that maps to the listen address for WLS_OSB2 and fails over with server migration of this managed server. It is enabled on the node where the WLS_OSB2 process is running (SOAHOST2 by default).

5.14.1.3 Shared Storage Prerequisites

For proper recovery in case of failure, store both JMS and transaction logs in a location that is accessible to all the nodes that can resume the operations after a failure in a managed server. This requires a shared storage location that can be referenced by multiple nodes. Table 5-11 lists the contents of shared storage.

Common location but Individual store per server (for example: WseeFileStore_auto_1)

WLS_OSB2

JMS Stores

VOL1

ORACLE_BASE/admin/domain_name/osb_cluster_name/jms

Common location but Individual store per server (for example: WseeFileStore_auto_2)

The shared storage can be a NAS or SAN device. Specifically for NFS mounted systems, different issues related to file locking and abrupt node failures have been detected. Check the Oracle Fusion Middleware Release Notes and with your storage vendor for the main recommended parameters to be used as mount options. The following is an example command based on a NAS device. Your options may be different from the ones specified in this section:

ORACLE_HOME: This environment variable and related directory path refers to the location where Oracle FMW SOA Suite or Oracle Service Bus is installed.

ORACLE_COMMON_HOME: This environment variable and related directory path refers to the Oracle home that contains the binary and library files required for the Oracle Enterprise Manager Fusion Middleware Control and Java Required Files (JRF).

There can be only one Oracle Common home within each Middleware home.

DOMAIN Directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored.

ORACLE_INSTANCE: An Oracle instance contains one or more system components, such as Oracle Web Cache, Oracle HTTP Server, or Oracle Internet Directory. An Oracle instance directory contains updateable files, such as configuration files, log files, and temporary files.

The values used and recommended for consistency for this directories are:

Ensure that the following initialization parameter is set to the required value. It is checked by Repository Creation Assistant.

Table 5-12 Required Initialization Parameters

Parameter

Required Value

Parameter Class

PROCESSES

300 or greater

Static

To check the value of the initialization parameter using SQL*Plus, you can use the SHOW PARAMETER command.

As the SYS user, issue the SHOW PARAMETER command as follows:

SQL> SHOW PARAMETER processes

Set the initialization parameter using the following command:

SQL> ALTER SYSTEM SET processes=300 SCOPE=SPFILE

Restart the database.

Note:

The method that you use to change a parameter's value depends on whether the parameter is static or dynamic, and on whether your database uses a parameter file or a server parameter file. See the Oracle Database Administrator's Guide for details on parameter files, server parameter files, and how to change parameter values.

Host Name: Enter the name of the node that is running the database. You can use the RAC SCAN address as the host name. For an Oracle RAC database, specify the VIP name or one of the node names as the hostname: SOADBHOST1VIRTUAL.

Port: The port number for the database: 1521

Service Name: Enter the service name of the database: soaha.mycompany.com

Username: SYS

Password: Enter the password for the SYS user.

Role: SYSDBA

Click Next.

If you receive the following warning message:

The database you are connecting is with non-UTF8 charset, if you are going to use this DB for multilingual support, you may have data loss. If you are not using for multilingual support you can continue, otherwise we strongly recommend using UTF-8 database.

Click Ignore or Stop.

In the Select Components screen, select Create a New Prefix, and enter a prefix to use for the database schemas, for example, SOAHA

Write down the schema names so they are available in later procedures.

Select the following schemas:

Under AS Common Schemas, select Metadata Services.

Under SOA and BPM Infrastructure, select SOA Infrastructure and User Messaging Service (the OSB Reporting Provider schema is part of SOA Service Infrastructure).

Click Next.

In the Schema Passwords screen, enter passwords for the main and additional (auxiliary) schema users, and click Next.

In the Map Tablespaces screen, choose the tablespaces for the selected components, and click Next.

You need the appropriate database privileges to enable the Oracle WebLogic Server transaction manager to query for transaction state information and issue the appropriate commands, such as commit and rollback, during recovery of in-flight transactions after a WebLogic Server container failure.

To configure the SOA schemas for transactional recovery privileges:

Log on to SQL*Plus as a user with sysdba privileges. For example:

sqlplus "/ as sysdba"

Grant select on sys.dba_pending_transactions to soaha_soainfra.

Grant force any transaction to soaha_soainfra.

Note:

These privileges should be granted to the owner of the soainfra schema, as determined by the RCU operations.

This section describes the load balancer prerequisites for deploying an Oracle SOA Suite high availability environment.

Load Balancers

Oracle SOA Suite uses a hardware load balancer when deployed in a high availability configuration with two Oracle HTTP Servers as web tier components. The hardware load balancer should have the following features:

Ability to load-balance traffic to a pool of real servers through a virtual hostname: Clients access services using the virtual hostname (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.

Port translation configuration: The load balancer should have the ability to perform port translation, where it enables incoming requests received on one port to be routed to a server process running on a different port. For example, a request received on port 80 can be routed to port 7777.

Monitoring of ports (HTTP, HTTPS)

Virtual servers and port configuration: Ability to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements:

The load balancer should enable configuration of multiple virtual servers. For each virtual server, the load balancer should enable configuration of traffic management on more than one port. For example, the load balancer typically needs to be configured with a virtual server and ports for HTTP and HTTPS traffic.

The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the load balancer through the virtual server names.

Resource monitoring/port monitoring/process failure detection: The load balancer must be able to detect service and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your load balancer has the ability to automatically detect failures, you should use it.

Fault-tolerant mode: It is highly recommended that you configure the load balancer to be in fault-tolerant mode.

Other: It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client machine.

Sticky routing capability: Ability to maintain sticky connections to components based on cookies or URL.

SSL acceleration: SSL acceleration is a method of offloading the processor-intensive public key encryption algorithms involved in SSL transactions to a hardware accelerator.

This feature is recommended but not required.

Table 5-13 shows some example virtual server names to use for the external load balancer in the Oracle SOA Suite high availability environment.

Table 5-13 Virtual Server Names for the External Load Balancer

Component

Virtual Server Name

Oracle SOA Suite

soa.mycompany.com

WebLogic Server Administration Console

admin.mycompany.com

Oracle Enterprise Manager Fusion Middleware Control

admin.mycompany.com

Oracle Service Bus Web Console

admin.mycompany.com

Virtual Server Names

This section describes the virtual server names that should be set up for the high availability deployments described in this chapter.

soa.mycompany.com

soa.mycompany.com is a virtual server name that acts as the access point for all HTTP traffic to the runtime SOA components, such as soa-infra and workflow. Traffic to the both SSL and non-SSL ports is configured and typically non-SSL is redirected to SSL. Clients access this service using the address soa.mycompany.com:443. This virtual server is defined on the load balancer.

admin.mycompany.com

This virtual server acts as the access point for all internal HTTP traffic that gets directed to the administration services. The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address admin.mycompany.com:80 and in turn forward these to ports 7777 on WEBHOST1 and WEBHOST2. The services accessed on this virtual host include the WebLogic Administration Server Console, Oracle Enterprise Manager, and the Oracle Service Bus Web Console.

In addition, ensure that the virtual server names are associated with IP addresses and are part of your Domain Name System (DNS). The computers on which Oracle Fusion Middleware is running must be able to resolve these virtual server names.

Port 7777 is not used by any service on the nodes. You can verify this by running the following command:

Unix:

netstat -an | grep LISTEN | grep "7777"

Windows:

netstat -an | findstr "LISTEN" | findstr ":7777"

If the ports are in use, make them available.

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct, and that you have write permissions for that directory.

Select Specify Custom Ports. If you specify a custom port, select Specify Ports using Configuration File, and use the Browse function to select the file.

Enter the Oracle HTTP Server port, for example, 7777.

Click Next.

In the Configuration Summary screen, ensure that the selections are correct and click Install.

In the Installation Progress screen:

For UNIX systems, a dialog box appears prompting you to run the oracleRoot.sh script. Open a command window and run the script, following the prompts.

Click Next.

In the Configuration screen, several configuration assistants launch in succession. When the configuration assistants are finished, the Configuration Completed screen appears.

In the Configuration Completed screen, click Finish to exit.

Repeat the steps for WEBHOST2 and configure your LBR with a pool containing both the WEBHOST1 and WEBHOST2 addresses.

5.14.1.10.1 Validating Oracle HTTP Server

To verify that Oracle HTTP Server is set up correctly, access the root URL context of the server by using the following URL in a browser:

HTTP://WEBHOST1:7777/

If Oracle HTTP Server is set up correctly, the Oracle FMW Welcome screen appears in the browser.

5.14.2 Installing Oracle Fusion Middleware Home

This section describes the procedure for installing Oracle Fusion Middleware on all nodes in the application tier that run Oracle WebLogic Server and Oracle SOA Suite. Repeat the procedures (described below for SOAHOST1) for installing WebLogic Server and Oracle SOA in SOAHOST2. The directory paths for binary files and domains used when installing new nodes must be exactly the same as those used for first node. If these paths and domains are not exactly the same as those used for the first node, failover will not work properly.

In the Choose Product Installation Directories screen, accept the following directory:

WL_HOME

Click Next.

In the Installation Summary screen, click Next.

In the Installation Complete screen, deselect Run QuickStart, and click Done.

5.14.2.2 Installing Oracle Fusion Middleware for Oracle SOA

On Linux platforms, if the /etc/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct and that you have write permissions for that directory.

If the /etc/oraInst.loc file does not exist, you can skip this step.

Start Oracle Fusion Middleware 11g Oracle SOA Suite Installer:

On UNIX:

SOAHOST1> runInstaller

On Windows:

SOAHOST1> setup.exe -jreLoc JRE_Location

When Oracle Fusion Middleware 11g Oracle SOA Suite Installer prompts you for a JRE/JDK location enter the Oracle SDK location created in the Oracle WebLogic Server installation, for example, MW_HOME/jrockit_160_<version>.

In the Specify Inventory Directory screen, do the following:

Enter HOME/oraInventory, where HOME is the home directory of the user performing the installation (this is the recommended location).

Enter the OS group for the user performing the installation.

Click Next.

Follow the instructions on the screen to run /createCentralInventory.sh as root.

Click OK.

In the Welcome screen, click Next.

In the Prerequisite Checks screen, verify that the checks complete successfully, and click Next.

In the Specify Installation Location screen:

For Middleware Home, enter MW_HOME.

For Oracle Home Directory, enter soa.

Click Next.

In the Installation Summary screen, click Install.

In the Installation Complete screen, click Finish.

5.14.2.3 Installing Oracle Service Bus

To install Oracle Service Bus, follow these steps:

Start the Oracle Fusion Middleware 11g Oracle Service Bus Installer:

On UNIX:

SOAHOST1> runinstaller

On Windows:

SOAHOST1> setup.exe -jreLoc JRE_LOCATION

When Oracle Fusion Middleware 11g Oracle Service Bus Installer prompts you for a JRE/JDK location enter the Oracle SDK location created in the Oracle WebLogic Server installation, for example, MW_HOME/jrockit_160_<version>.

5.14.3 Enabling VIP1 and VIP3 in SOAHOST1 and VIP2 and VIP4 in SOAHOST2

The SOA domain uses virtual hostnames as the listen addresses for the SOA and OSB managed servers. You must enable VIP mapping for each of these hostnames on the two SOA Machines, (VIP1 and VIP3 on SOAHOST1 and VIP2 and VIP4 on SOAHOST2), and must correctly resolve the virtual hostnames in the network system used by the topology (either by DNS Server or by hosts resolution).

Run Oracle Fusion Middleware Configuration Wizard from the MW_HOME/Oracle_common/common/bin directory to create a domain containing the Administration Server and Oracle SOA and OSB components. Ensure that the database where you installed the repository is running. For Oracle RAC databases, all instances must be running.

Note:

Multiple SOA clusters are not allowed in the same WebLogic Server domain.

Multiple OSB clusters are not allowed in the same WebLogic Server domain.

One single cluster holding both SOA and OSB is not allowed in a single WebLogic Server domain.

Change the directory to the location of the Oracle Fusion Middleware Configuration Wizard, located in the Middleware home, oracle_common/common/bin directory:

SOAHOST1> cd ORACLE_HOME/oracle_common/common/bin

Start Oracle Fusion Middleware Configuration Wizard:

For Linux:

SOAHOST1> ./config.sh

For Windows:

SOAHOST1> config.cmd

In the Welcome screen, select Create a New WebLogic Domain, and click Next.

In the Select Domain Source screen:

Select Generate a domain configured automatically to support the following products.

Service Name: Enter the service name of the database. For a GridLink data source, you must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

Note:

The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, example.com

Username prefix: Enter the prefix for the schemas. The user names shown in Table 5-14 assume that soaha was used as prefix for schema creation from RCU.

Password and Confirm Password: Enter the password for access to the schemas.

For a GridLink data source, enter the SCAN address in the Listener Address field and SCAN port in the Port field. Enter the ONS host and port information in the ONS Host and Port fields, respectively.

Note:

Oracle recommends that you use SCAN addresses to specify the host and port for both the TNS listener and the ONS listener in the WebLogic console. You do not need to update a GridLink data source containing SCAN addresses if you add or remove Oracle RAC nodes. Contact your network administrator for appropriately configured SCAN urls for your environment. See SCAN Addresses in the Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server guide.

In the Test JDBC Data Sources screen, the connections are tested automatically. The Status column displays the results. Ensure that all connections were successful. If not, click Previous to correct your entries.

Click Next when all the connections are successful.

In the Select Optional Configuration screen, select the following:

Administration Server

JMS Distributed Destinations

Managed Servers, Clusters and Machines

JMS File Store

In the Configure the Administration Server screen, enter the following values:

Name: AdminServer

Listen Address: Enter the hostname for the VIP0 virtual IP

Listen Port: 7001

SSL listen port: N/A

SSL enabled: leave unchecked

Click Next.

Verify that the JMS Distributed Destination Type is set to UDD.

In the Configure Managed Servers screen, add the following managed servers:

Table 5-15 Configuring Managed Servers

Name

Listen Address

Listen Port

SSL Listen Port

SSL Enabled

WLS_SOA1

SOAHOST1VHN1

8001

n/a

No

WLS_SOA2

SOAHOST2VHN1

8001

n/a

No

WLS_OSB1

OSBHOST1VHN1

8011

n/a

No

WLS_OSB2

OSBHOST2VHN1

8011

n/a

No

Do not delete any server that appears. You can modify the servers. If you delete a server and add a new one, targeting fails.

Note:

The standard recommendation is to run custom applications and other systems in a separate WebLogic Managed Server. However, this section does not cover creating custom WLS managed servers as Figure 5-47 describes.

Click Next.

In the Configure Clusters screen, add the following clusters:

Table 5-16 Adding the SOA and OSB Clusters

Name

Cluster Messaging Mode

Multicast Address

Multicast Port

Cluster Address

SOA_Cluster

unicast

N/A

N/A

SOAHOST1VHN1:8001, SOAHOST2VHN1:8001

OSB_Cluster

unicast

N/A

N/A

OSBHOST1VHN1:8011, OSBHOST2VHN1:8011

Click Next.

In the Assign Servers to Clusters screen, assign the following servers to SOA_Cluster:

WLS_SOA1

WLS_SOA2

Assign the following servers to the OSB_Cluster:

WLS_OSB1

WLS_OSB2

Click Next.

In the Configure Machines screen:

Delete the LocalMachine that appears by default.

Click the Unix Machine tab, and add the following machines:

Table 5-17 Configuring Machines

Name

Node Manager Listen Address

SOAHOST1

Hostname of SOAHOST1

SOAHOST2

Hostname of SOAHOST2

OSBHOST1

Hostname of OSBHOST1

OSBHOST2

Hostname of OSBHOST2

Leave all other fields as their default values, and click Next.

In the Assign Servers to Machines screen, assign servers to machines as follows:

In the Configure JMS Files Stores screen, enter a directory location for each store that is accessible from all nodes that the servers can be failed over to. As recommended in Section 5.14.1.3, "Shared Storage Prerequisites," use a directory like the following on a shared device:

5.14.5 Creating boot.properties for the Administration Server on SOAHOST1

This is an optional step for enabling the Administration Server to start without prompting you for the administrator username and password. Create a boot.properties file for the Administration Server on SOAHOST1.

5.14.6.2 Validating the Administration Server

Verify that the WLS_SOA1 and WLS_SOA2 managed servers are listed. Also verify that the WLS_OSB1 and WLS_OSB2 managed servers are listed.

Verify that the SOA_Cluster cluster and the OSB_Cluster cluster is listed.

Verify that you can access Enterprise Manager at http://vip0:7001/em.

Verify that you can access the Oracle Service Bus Console at http://vip0:7001/sbconsole.

5.14.7 Disabling Host Name Verification for the Administration Server and the WLS_SOAn/WLS_OSBn Managed Servers

This step is required if you have not set up the appropriate certificates for hostname verification between the Administration Server and Node Manager. If SSL is not set up, you receive an error message unless you disable hostname verification.

You can re-enable hostname verification when you have set up SSL communication between the Administration Server and the Node Manager.

To disable hostname verification:

In Administration Console, select Servers, and then AdminServer.

Select SSL, and then Advanced.

In the Change Center, click Lock & Edit.

Set Hostname Verification to None. Save and activate the changes.

Repeat these steps for the WLS_SOA1, WLS_SOA2, WLS_OSB1, and WLS_OSB2 servers.

Restart the Administration Server.

5.14.8 Configuring Oracle Coherence for Deploying Composites

Although deploying composites uses multicast communication by default, Oracle recommends using unicast communication for SOA high availability. Use unicast if you disable multicast communication for security reasons.

Note:

An incorrect configuration of the Oracle Coherence framework that is used for deployment may prevent the SOA system from starting. The deployment framework must be properly customized for the network environment on which the SOA system runs. Oracle recommends the following configuration described in this section.

Enabling Communication within Clusters Using Unicast Communication

Multicast communication enables Oracle SOA Suite to discover all of the members of a cluster to which to it deploys composites dynamically. However, unicast communication does not enable nodes to discover other cluster members in this way. Consequently, you must specify the nodes that belong to the cluster. You do not need to specify all of the nodes of a cluster, however. You need only specify enough nodes so that a new node added to the cluster can discover one of the existing nodes. As a result, when a new node has joined the cluster, it is able to discover all of the other nodes in the cluster. Additionally, in a configuration where multiple IPs are available in the same node, you must configure Oracle Coherence to use a specific hostname to create Oracle Coherence cluster.

Tip:

To guarantee high availability during deployments of SOA composites, specify enough nodes so that at least one of them is running at any given time.

Specify the nodes using the tangosol.coherence.wka<n> system property, where <n> is a number between 1 and 9. You can specify up to 9 nodes. Start the numbering at 1. This numbering must be sequential and must not contain gaps. In addition, specify the hostname used by Oracle Coherence to create a cluster through the tangosol.coherence.localhost system property. This hostname should be the virtual hostname used by the SOA servers that maps the corresponding listener addresses (VIP1 and VIP2). Set this property by adding the -Dtangosol.coherence.localhost parameters to the Arguments field of Administration Console's Server Start tab (Figure 5-51).

Figure 5-51 Setting the Hostname Using the Start Server Tab of Administration Console

The Coherence cluster used for deployment uses port 8088 by default. This port can be changed by specifying a different port (for example, 8089) with the -Dtangosol.coherence.wkan.port and -Dtangosol.coherence.localport startup parameters. For example:

WLS_SOA1 (enter the following into the Arguments field on a single line, without a carriage return):

The multicast and unicast addresses are different from the ones used by the WebLogic Server cluster for cluster communication. SOA guarantees that composites are deployed to members of a single WebLogic Server cluster even though the communication protocol for the two entities (the WebLogic Server cluster and the groups to which composites are deployed) are different.

Do not copy the text from above to your Administration Console's arguments text field, which can result in HTML tags being inserted in the Java arguments. The text should not include any text or characters other than the ones shown above.

5.14.9 Configure Oracle Coherence for the Oracle Service Bus Result Cache

The default configuration for the Coherence infrastructure that Oracle Service Bus result cache uses is multicast. Oracle recommends using unicast communication for the Oracle Service Bus result cache.

To enable unicast for the Oracle Service Bus result cache Coherence infrastructure:

Log into the Administration Console. In the Change Center, click Lock & Edit.

In the Domain Structure window, expand the Environment node.

Click Servers.

Click the name of the server (represented as a hyperlink) in the Name column of the table. The settings page for the selected server appears.

Click the Server Start tab.

Enter the following for WLS_OSB1 on a single line, no carriage returns:

When you run the Fusion Middleware Configuration Wizard after a pack/unpack procedure, products already in the original domain that appear in the Select Extension Source screen are not selected or greyed out.

Templates that you create with the pack command are considered user-created templates and treated differently than default templates. User-created templates are self-contained and do not have any information related to the creation of the original domain.

Repeat the steps above for OSBHOST1 and OSBHOST2.

5.14.13 Extracting XEngine Files in the Second Node

To enable B2B's XEngine in the second node, you must extract ZEngine tar content manually:

Values such as soa.mycompany.com:443, 7777, admin.mycompany:80, and you@youraddress are examples only. Enter values based on the actual environment.

Note:

Each HTTP proxy service in Oracle Service Bus exposes its own context root, which is the full path of the proxy service (default). For example, ordersubmit.proxy in the project OrderProcessing and in folder submission exposes /OrderProcessing/submission/ordersubmit as context path. You must add these url contexts in Oracle HTTP Server for each service.

5.14.16 Validating Access Through Oracle HTTP Server

Verify that the SOA Servers status is Running in the Administration Console. If the server status is Starting or Resuming, wait for the server status to change to Started. If another status is reported Admin, Failed), check the server output log files for errors. Verify that you can access these URLS:

http://WEBHOST1:7777/wsm-pm

http://WEBHOST2:7777/wsm-pm

http://WEBHOST1:7777/soa-infra

http://WEBHOST2:7777/soa-infra

http://WEBHOST1:7777/soa/composer

http://WEBHOST2:7777/soa/composer

http://WEBHOST1:7777/integration/worklistapp

http://WEBHOST2:7777/integration/worklistapp

http://WEBHOST1:7777/sdpmessaging/userprefs-ui

http://WEBHOST2:7777/sdpmessaging/userprefs-ui

http://WEBHOST1:7777/b2bconsole

http://WEBHOST2:7777/b2bconsole

http://WEBHOST1:7777/sbinspection.wsil

http://WEBHOST2:7777/sbinspection.wsil

Verify these URLs also using your load balancing router address:

http://soa.mycompany.com:80/wsm-pm

http://soa.mycompany.com:80/soa-infra

http://soa.mycompany.com:80/soa/composer

http://soa.mycompany.com:80/integration/worklistapp

http://soa.mycompany.com:80/sdpmessaging/userprefs-ui

http://soa.mycompany.com:80/b2bconsole

http://soa.mycompany.com:80/sbinspection.wsil

To ensure that routing and failover from the HTTP Server to the SOA_CLuster is working correctly:

While WLS_SOA2 is running, stop WLS_SOA1 from the Administration Console.

Access the following URLs and verify the appropriate functionality:

WEBHOST1:7777/wsm-pm

WEBHOST1:7777/soa-infra

WEBHOST1:7777/soa/composer

WEBHOST1:7777/integration/worklistapp

WEBHOST1:7777/sdpmessaging/userprefs-ui

WEBHOST1:7777/b2bconsole

Start WLS_SOA1 from Administration Console.

Stop WLS_SOA2.

Access the URLs in Step 2 above again and verify the appropriate functionality:

To ensure that routing and failover from the HTTP Server to the OSB_Cluster is working correctly:

While WLS_OSB2 is running, stop WLS_OSB1 from the Administration Console.

Access the following URL and verify the appropriate functionality:

WEBHOST1:7777/sbinspection.wsil

Start WLS_OSB1 from the Administration Console.

Stop WLS_OSB2.

Access the URL in Step 2 above again and verify the appropriate functionality:

5.14.17 Setting the Front End HTTP Host and Port

You must set the front end HTTP host and port for Oracle WebLogic Server cluster:

In the Administration Console, in the Change Center section, click Lock & Edit.

In the left pane, select Environment > Clusters.

Select the SOA_Cluster.

Select HTTP.

Set the values for the following:

Frontend Host: soa.mycompany.com

Frontend HTTP Port: 80

Frontend HTTPS Port: leave it blank

Note:

Make sure this address is correct and available (the load balancing router is up). An incorrect value, for example, http:// in the address, or trailing / in the hostname, may prevent the SOA system from being accessible even when using the virtual IPs to access it.

Click Save.

To activate the changes, click Activate Changes in the Change Center section of the Administration Console.

Repeat steps 1 through 7 for the OSB_Cluster.

If you have started the server before, notice this change requires a restart of the managed servers in the cluster.

Note:

The SOA system calculates the callback server URL as follows:

If a request to SOA originates from an external or internal service is originating a request to SOA, then SOA uses the callback server URL specified by the client.

If a request to an external or internal service originates from SOA, the callbackURL cannot be populated in the SOA request dynamically because SOA is the originator. Instead, callbackServerURL is used if it is specified as a binding property for the specific reference. (You can set this when modeling the composite or at runtime using the MBeans accessed through Oracle Enterprise Manager console.) This enables different service calls to have different callback URLs. That is, a callback server URL from an external service is different than one to an internal service.

However, if the callbackServer URL is not specified as a binding property for that reference, then the system uses the callback server URL as specified in soa-infra-config. If the callback server URL is not specified in soa-infra-config, then the system uses the callback server URL as the front end host specified in WLS. If the front end host is not specified in WLS, the system uses the callback server URL as the local hostname as provided by WLS MBean APIs.

For SOA high availability installations frontended by Oracle HTTP Server, monitoring should be done on the Oracle HTTP Server ports of the real backend servers. This is the case when a deployment is using all the components deployed to the SOA Managed Server. A simple HTTP monitor that pings the HTTP/HTTPS port and expects a pre-determined response in turn should suffice. If only a specific SOA component is being used (such as B2B), then a monitor that does a deeper level check all the way to the managed server can be considered to validate the health of the component in use. Please check with your load balancer vendor on setting up the HTTP monitors with your load balancer.

If you do not set the front end HTTP host and port, you get the following message when trying to retrieve a document definition XSD from Oracle B2B:

An error occured while loading the document definitions.
java.lang.IllegalArgumentException: Cluster address must be set when
clustering is enabled.

Each server has a transaction log, which stores information about committed transactions coordinated by the server that may not have been completed. WebLogic Server uses the transaction log when recovering from system fails or network failures. To take advantage of the migration capability of the Transaction Recovery Service for servers in a cluster, you must store the transaction log in a location that is available to a server and its backup servers (the nodes the WebLogic Server can failover to), preferably on a dual-ported SCSI disk, or a Storage Area Network (SAN). To configure the default persistent store:

In the Domain Structure tree, expand Environment and select Servers.

Select the server you want to modify.

Select the Configuration, Services tab.

Under Default Store, in the Directory field, enter the path to the folder where you want the default persistent store to store its data files.

To enable migration of the Transaction Recovery Service, you must specify a location on a persistent storage solution that is available to other servers in the cluster. Therefore, the directory that you enter must be accessible by both WLS_SOA1 and WLS_SOA2 for the SOA_Cluster and WLS_OSB1 and WLS_OSB2 for the OSB_Cluster. This directory must exist before the server is restarted.

5.14.19 Deploying Applications

You can deploy SOA composite applications from Oracle Enterprise Manager Fusion Middleware Control Console with the Deploy SOA Composite wizard. Use the Deploy SOA Composite wizard to deploy any of the following:

A new SOA composite application for the first time

A new revision (for example, 2.0) alongside an older revision (for example, 1.0) without impacting the latter. The revision deployed last becomes the new default revision of that composite (unless you specify otherwise at a later step during deployment).

A bundle (ZIP file) containing multiple SOA composite application revisions (for example, revisions 2.0, 3.0, and 4.0) of a SOA composite application that already has a different revision that is currently deployed (for example, 1.0). This option enables you to deploy revisions 1.0, 2.0, 3.0, and 4.0 at the same time. The bundle can also contain revisions of different composites. There is no restriction that all revisions must be of the same composite application.

Deployment extracts and activates the composite application in SOA Service Infrastructure. Once an application is deployed, you can perform administration tasks, such as creating instances, configuring properties, monitoring performance, managing instance life cycles, and managing policies and faults.

Note:

If you want to redeploy an existing revision of an application, do not use this wizard. Instead, use the Redeploy SOA Composite wizard.

To deploy applications:

Access the Deploy SOA Composite wizard through one of the following options:

In the Archive or Exploded Directory section, specify the archive of the SOA composite application to deploy. The archive contains the project files of the composite to be deployed (for example, HelloWorld_rev1.0.jar for a single archive or OrderBooking_rev1.0.zip for multiple archives).

In the Configuration Plan section, optionally specify the configuration plan to include with the archive. The configuration plan enables you to define the URL and property values to use in different environments. During process deployment, the configuration plan is used to search the SOA project for values that must be replaced to adapt the project to the next target environment.

Click Next.

Select the WebLogic Server or cluster to which to deploy the SOA composite application archive. You can deploy to multiple servers and clusters.

Click Next.

Review your selections.

Select whether or not to deploy the SOA composite application as the default revision. The default revision instantiates when a new request comes in.

The WLS_SOA1 managed server is configured to restart on SOAHOST2 in case of failure, and the WLS_SOA2 managed server is configured to restart on SOAHOST1 in case of failure. For this configuration the WLS_SOA1 and WLS_SOA2 servers listen on specific floating IPs that are failed over by WLS Server Migration. To configure server migration for the WLS_SOAn managed servers, follow these steps:

Setting Environment and Superuser Privileges for the wlsifconfig.sh script

Grant sudo privilege to the WebLogic user ('oracle') with No Password restriction, and grant execute privilege on /sbin/ifconfig and /sbin/arping binaries.

Note:

On Windows, the script name is wlsifconfig.cmd. Users that have the administrator privilege can run it.

Make sure the script is executable by the WebLogic user ('oracle'). The following is an example of an entry inside /etc/sudoers granting sudo execution privilege for oracle and also over the ifconfig and arping:

To verify that Server Migration is working properly, follow these steps:

From SOAHOST1:

Force stop the WLS_SOA1 managed server.

To do this, run this command:

SOAHOST1> kill -9<pid>

pid specifies the process ID of the managed server. You can identify the pid in the node by running this command:

SOAHOST1> ps -ef | grep WLS_SOA1

Note:

For Windows, the Managed Server can be terminated by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of Managed server run the following command and identify the pid of the WLS_SOA1 Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager console: you should see a message indicating that WLS_SOA1's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_SOA1. Node Manager waits for a fence period of 30 seconds before trying this restart.

Once Node Manager restarts the server, stop it again. Now Node Manager should log a message indicating that the server will not be restarted again locally.

From SOAHOST2:

Watch the local Node Manager console. After 30 seconds since the last try to restart WLS_SOA1 on Node 1, Node Manager on Node 2 should prompt that the floating IP for WLS_SOA1 is being brought up and that the server is being restarted in this node.

Access the soa-Service Infrastructure console in the same IP.

To verify migration from the Administration Console:

Log into Administration Console.

Click on Domain on the left pane.

Click on the Monitoring tab and then on the Migration sub-tab.

The Migration Status table provides information on the status of the migration.

On Windows, when you manually shut down multiple servers at the same time on the same machine and, on another machine, attempt to start one of the servers that you shut down, the IP bind may not work. This happens because the original machine still has claim to the IP address, even though netsh has reported that the IP address has been removed.

To resolve this, you must check the network configuration either by using the ipconfig utility or Windows Network Configuration. Either of these may show that one of the virtual/floating IP addresses is still configured even though the servers have been shut down. You can then use Windows Network Configuration to remove the IP address using the following procedure:

After a server migrates, to fail it back to its original node/machine, stop the managed server from the Administration Console and then start it again. The appropriate Node Manager starts the managed server on the machine it was originally assigned to.

The WLS_OSB1 managed server is configured to restart on OSBHOST2 in case of failure, and the WLS_OSB2 managed server is configured to restart on OSBHOST1 in case of failure. For this configuration the WLS_OSB1 and WLS_OSB2 servers listen on specific floating IPs that WLS Server Migration fails over. To configure server migration for the WLS_OSBn managed servers, follow these steps:

Note:

If a tablespace, schema multi data source, and data sources are already set up for SOA (as Section 5.14.20, "Configuring Server Migration for the WLS_SOA Servers" describes), then the OSB Cluster can reuse the existing leasing schema and data source and Step 1 and Step 2 below are not required. Instead, target the data sources and multi data sources created for SOA to the OSB Cluster.

Step 1 Set Up User and Tablespace for the Server Migration Leasing Table

Create a tablespace called leasing.

Example: Log on to SQL*Plus as the sysdba user and run the following command:

Copy the leasing.ddl file, located in the WL_HOME/server/db/oracle/817 or WL_HOME/server/db/oracle/920 directories, to your database node.

Connect to the database as the leasing user.

Run the leasing.ddl script in SQL*Plus.

SQL> @copy_location/leasing.ddl;

Step 2 Create a Multi Data Source from Administration Console

You create a data source to each of the Oracle RAC database instances during the process of setting up the multi data source, both for these data sources and the global leasing multi data source. When you create a data source:

Ensure that this is a non-xa data source.

The names of the multi data sources are in the following format: <MultiDS>-rac0, <MultiDS>-rac1

Use Oracle's Driver (Thin) Version 9.0.1, 9.2.0, 10, 11.

Do not use any type of distributed transaction emulation/participation algorithm for the data source; do not choose the Supports Global Transactions option, or the Logging Last Resource, Emulate Two-Phase Commit, or One-Phase Commit options of the Supports Global Transactions option, and specify a service name for your database. Data sources do not require support for global transactions.

Target these data sources to the cluster.

Make sure the datasources' connection pool initial capacity is set to 0. To do this, select Services, JDBC, and then Datasources. In the Datasources screen, click the Datasource Name, then click the Connection Pool tab, and enter 0 in the Initial capacity field.

To create a multi data source:

From the Domain Structure window in the Administration Console, expand the Services node, then click Data Sources.

The nodemanager.properties file is located in the WL_HOME/common/nodemanager directory. For server migration to work properly, you must add the properties listed in this section:

Interface=eth0

Note:

Do not specify the sub interface, such as eth0:1 or eth0:2. This interface is to be used without the :0, or :1. The Node Manager's scripts traverse the different :X enabled IPs to determine which to add or remove. For example, the valid values in Linux environments are eth0, eth1, or, eth2, eth3, ethn, depending on the number of interfaces configured.

This property specifies the interface name for the floating IP (eth0, for example, on Linux).

Note:

For Windows, the Interface should be set to the Network Interface Name. For example: Interface="Local Area Connection".

Be sure that the interface provided is the public interface for this node. On multi home nodes, this interface should be the one on which the floating IP can be enabled.

NetMask=255.255.255.0

This property specifies the net mask for the interface for the floating IP. The netmask provided (255.255.255.0) is just an example. The actual value depends on your network.

UseMACBroadcast=true

This property specifies whether or not to use a node's MAC address when sending ARP packets, that is, whether or not to use the -b flag in the arping command.

After starting Node Manager, verify in Node Manager's output (shell where Node Manager is started) that these properties are being used, or problems may arise during migration. You should see something like this in the Node Manager's output:

StateCheckInterval=500

Interface=eth0 (Linux) or Interface="Local Area Connection" (Windows)

NetMask=255.255.255.0

UseMACBroadcast=true

Note:

The steps in this section are not required if the server properties (start properties) have been properly set and the Node Manager can start the servers remotely.

Set the following property in the nodemanager.properties file.

StartScriptEnabled

Set this property to true.

Start the Node Manager on OSBHOST1 and OSBHOST2 by running the startNodeManager.sh script located in the WL_HOME/server/bin directory.

Validate the changes to the nodemanager.properties file by checking the nodemanager.log file.

Step 4 Set Environment and Superuser Privileges for the wlsifconfig.sh script

Grant sudo privilege to the WebLogic user ('oracle') with No Password restriction, and grant execute privilege on /sbin/ifconfig and /sbin/arping binaries.

Note:

On Windows, the script is named wlsifconfig.cmd and it can be run by users with the administrator privilege.

Verify that the WebLogic user ('oracle') can run the script. The following is an example of an entry inside /etc/sudoers granting sudo execution privilege for oracle and also over the ifconfig and arping:

Configuring Cluster Migration sets the DataSourceForAutomaticMigration property to true. To configure cluster migration in a migration in a cluster:

Log into the Administration Console

In the left pane, expand Environment and select Clusters.

Select the cluster you want to configure migration for (OSB_Cluster).

Click Migration.

In the Change Center, click Lock & Edit.

In the Available field, select the machine to which to enable migration and click the right arrow. In this case, select OSBHOST1 and OSBHOST2.

Select the leasing data source.

Click Save.

Set the Candidate Machines for Server Migration for all the managed servers.

In the left pane of Administration Console, expand Environment and select Servers.

Select the server for which you want to configure migration.

Click the Migration tab.

In the Available field, located in the Migration Configuration section, select the machines to which to enable migration and click the right arrow. For WLS_OSB1, select OSBHOST2. For WLS_OSB2, select OSBHOST1.

Select the checkbox for Automatic Server Migration Enabled. This enables the Node Manager to start a failed server on the target node automatically.

Click Save and activate the changes.

These changes require restart of the servers and Node Managers to be effective.

Step 6 Test Server Migration

To verify that Server Migration is working properly, follow these steps:

From OSBHOST1:

Force stop the WLS_OSB1 managed server.

To do this, run this command:

OSBHOST1> kill -9<pid>

pid specifies the process ID of the managed server. You can identify the pid in the node by running this command:

OSBHOST1> ps -ef | grep WLS_OSB1

Note:

For Windows, the Managed Server can be terminated by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server run the following command and identify the pid of the WLS_OSB1 Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager console: you should see a message indicating that WLS_OSB1's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_OSB1. Node Manager waits for a fence period of 30 seconds before trying this restart.

Once Node Manager restarts the server, stop it again. Now Node Manager should log a message indicating that the server will not be restarted again locally.

From OSBHOST2:

Watch the local Node Manager console. After 30 seconds since the last try to restart WLS_OSB1 on Node 1, Node Manager on Node 2 should prompt that the floating IP for WLS_OSB1 is being brought up and that the server is being restarted in this node.

Access the sbinspection.wsil urlconsole in the same IP.

To verify migration from the Administration Console:

Log into the Administration Console.

Click on Domain on the left pane.

Click on the Monitoring tab and then on the Migration sub-tab.

The Migration Status table provides information on the status of the migration.

On Windows, when you manually shut down multiple servers at the same time, on the same machine, and then, on another machine, attempt to start one of the servers that was shut down, the IP bind may not work. This happens because the original machine still has claim to the IP address, even though netsh has reported that the IP address has been removed.

To resolve this, you must check the network configuration either by using the ipconfig utility or Windows Network Configuration. Either of these may show that one of the virtual/floating IP addresses is still configured even though the servers have been shut down. You can then use Windows Network Configuration to remove the IP address using the following procedure:

After a server is migrated, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager will start the managed server on the machine to which it was originally assigned.

5.14.22 Scaling the Topology

You can scale out or scale up this high availability topology. When you "scale up" the topology, you add new managed servers to nodes that are already running one or more managed servers. When you "scale out" the topology, you add new managed servers to new nodes.

In this case, you already have a node that runs a managed server configured with SOA and Oracle Service Bus components. The node contains a Middleware home, an Oracle HOME (SOA and Oracle Service Bus) and a domain directory for existing managed servers.

You can use the existing installations (the Middleware home, and domain directories) for creating new WLS_OSB servers. There is no need to install SOA or Oracle Service Bus binaries in a new location, or to run pack and unpack.

Using the Administration Console, clone WLS_OSBn into a new managed server. The source managed server to clone should be one that already exists on the node where you want to run the new managed server.

To clone a managed server:

Select Environment -> Servers from the Administration Console.

Select the managed server that you want to clone (for example, WLS_OSB1).

Select Clone.

Name the new managed server WLS_OSBn, where n is a number to identify the new managed server.

The rest of the steps assume that you are adding a new server to OSBHOST1, which is already running WLS_OSB1.

For the listen address, assign the virtual hostname to use for this new managed server. If you plan to use server migration as recommended for this server, this virtual hostname enables it to move to another node. The virtual hostname should be different from those that other managed servers use (may be in the same or different domain) that are running in the nodes that the OSB/SOA domain uses.

To set the managed server listen address:

Log into the Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers.

Select the managed server whose listen address you want to update in the Names column of the table.

Set the Listen Address to OSBHOST1VHN1. Click Save.

Save and activate the changes.

The changes take effect when the managed server restarts.

Update the cluster address to include the new server:

Select Environment -> Cluster from the Administration Console.

Click on the OSB_Cluster server.

In the Change Center, click Lock & Edit.

Add the new server's address and port to the Cluster Address field. For example:

OSBHOST1VHN1:8011,OSBHOST2VHN1:8011,OSBHOST1VHN2:8011

If your Oracle Service Bus configuration includes one or more business services that use JMS request/response functionality, then you must also perform the following procedure using the Oracle Service Bus Console after adding the new managed server to the cluster:

In the Change Center, click Create to create a session.

Using the Project Explorer, locate and select a business service that uses JMS request/response. Business services of this type display Messaging Service as their Service Type.

At the bottom of the View Details page, click Edit.

If there is a cluster address in the endpoint URI, add the new server to the cluster address.

On the Edit a Business Service - Summary page, click Save.

Repeat the previous steps for each remaining business service that uses JMS request/response.

In the Change Center, click Activate.

Restart the managed server.

Restart the Administration Server.

The business services are now configured for operation in the extended domain.

Note:

For business services that use a JMS MesageID correlation scheme, you must edit the connection factory settings to add an entry to the table mapping managed servers to queues. For information on how to configure queues and topic destinations, see "JMS Server Targeting" in Oracle Fusion Middleware Configuring and Managing JMS for Oracle WebLogic Server.

If your Oracle Service Bus configuration includes one or more proxy services that use JMS endpoints with cluster addresses, then you must also perform the following procedure using the Oracle Service Bus Console after adding the new managed server to the cluster:

In the Change Center, click Create to create a session.

Using the Project Explorer, locate and select a proxy service that uses JMS endpoints with cluster addresses.

At the bottom of the View Details page, click Edit.

If there is a cluster address in the endpoint URI, add the new server to the cluster address.

On the Edit a Proxy Service - Summary page, click Save.

Repeat the previous steps for each remaining proxy service that uses JMS endpoints with cluster addresses.

In the Change Center, click Activate.

Restart the managed server.

The proxy services are now configured for operation in the extended domain.

Update the Oracle Service Bus result cache Coherence configuration for the new server:

Log into the Administration Console. In the Change Center, click Lock & Edit.

In the Domain Structure window, expand the Environment node.

Click Servers. The Summary of Servers page appears.

Click the name of the server (represented as a hyperlink) in the Name column of the table. The settings page for the selected server appears.

Click the Server Start tab.

Enter the following for WLS_OSBn (on a single line, without a carriage returns):

The above configuration assumes that servers WLS_OSB1 and WLS_OSB2 are running when WLS_OSBn is started. This will enable WLS_OSBn to join the coherence cluster started by either WLS_OSB1 or WLS_OSB2 using the WKA addresses specified. Also, make sure WLS_OSB1 and WLS_OSB2 are started before WLS_OSBn is started when all three servers are restarted. This will ensure WLS_OSBn join the cluster started by one of WLS_OSB1 or WLS_OSB2. If you want to have a configuration where the order in which the servers are started does not matter, you must add the host and port for WLS_OSBn as WKA for WLS_OSB1 and WLS_OSB2, and also add WLS_OSBn as WKA for WLS_OSBn.

Save and activate the changes (this change requires the Oracle Service Bus servers to be restarted to be effective).

Create JMS Servers and persistent stores for OSB reporting/internal destinations on the new managed server.

Use the Administration Console to create a new persistent store for the new wlsbJMSServer and name it, for example, FileStore_auto_x. Specify the path for the store on shared storage, as Section 5.14.1.3, "Shared Storage Prerequisites" recommends:

Create a new JMS Server for OSB, for example, wlsbJMSServer_auto_x. Use the FileStore_auto_x for this JMSServer. Target the wlsbJMSServer_auto_x Server to the recently created Managed Server, WLS_OSBn.

Update the SubDeployment targets for the jmsresources OSB JMS Module to include the recently created OSB JMS Server, wlsbJMSServer_auto_x. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click jmsResources, represented as a hyperlink in the Names column of the table. The Settings page for jmsResources opens. Click the SubDeployments tab. The subdeployment module for jmsresources appears.

Note:

This subdeployment module name for destinations is a random name in the form of wlsbJMSServerXXXXXX. It comes from the Configuration Wizard JMS configuration for the first two servers, WLS_OSB1 and WLS_OSB2.

Click on the wlsbJMSServerXXXXXX subdeployment and update the targets to include the new wlsbJMSServer_auto_x server.

Create JMS Servers, persistent stores, and destinations for OSB JAX-RPC on the new managed server.

Note:

WebLogic Advanced Web Services for JAX-RPC Extension uses regular (non-distributed) destinations to ensure that a locally processed request on a service gets enqueued only to a local member.

Use the Administration Console to create a new persistent store for the new WseeJMSServer and name it, for example, Wsee_rpc_JMSFileStore_N. Specify the path for the store. This should be a directory on shared storage, as recommended inSection 5.14.1.3, "Shared Storage Prerequisites":

Note:

This directory must exist before the managed server is started or the start operation fails.

Create a new JMS Server for OSB JAX-RPC, for example, OSB_rpc_JMSServer_N. Use the Wsee_rpc_JMSFileStore_N for this JMSServer. Target the OSB_rpc_JMSServer_N Server to the recently created Managed Server (WLS_OSBn).

Update the WseeJMSModule OSB JMS Module with destinations and the recently created OSB JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click WseeJmsModule (represented as a hyperlink in the Names column of the table). The Settings page for WseeJmsModule appears. Follow steps d through v to complete this step.

In the Change Center, click Lock and Edit. Click New.

Select Queue.

Click Next.

Enter DefaultCallbackQueue-WseeJmsServer_auto_n as name for the queue.

Enter weblogic.wsee.DefaultCallbackQueue-WseeJmsServer_auto_n as the JNDI name.

Click Next.

Click Create a New Subdeployment.

Accept the default name.

Click OK.

Select OSB_rpc_JMSServer_n as the target.

Click Finish.

Activate the changes.

Update the local JNDI name for the destination by following steps q though u.

In the Change Center, click Lock & Edit.

In the Settings for the WseeJmsModule page, click on the DefaultCallbackQueue-WseeJmsServer_auto_n destination.

In the general Configuration tab, click on Advanced.

Update the local JNDI name to weblogic.wsee.DefaultCallbackQueue.

Activate the changes.

Repeat steps d through n for the DefaultQueue-WseeJmsServer_auto_n queue, using weblogic.wsee.DefaultQueue-WseeJmsServer_auto_n as the JNDi name and weblogic.wsee.DefaultQueue as the local JNDI name.

Create a new SAF agent and target it to the newly added managed server, as follows:

Select persistent store Wsee_rpc_JMSFileStore_N (persistent store created for OSB JAX-RPC). Target the SAF Agent to the new managed server and activate changes.

Configure TX persistent store for the new server. This should be a location visible from other nodes as indicated in the recommendations about shared storage.

From the Administration Console, select Server_name > Services tab. Under Default Store, in Directory, enter the path to the folder where you want the default persistent store to store its data files.

Disable hostname verification for the new managed server. Before starting and verifying the WLS_OSBn managed server, you must disable hostname verification. You can re-enable it after you have configured server certificates for the communication between the Oracle WebLogic Administration Server and the Node Manager in OSBHOSTn. If the source server from which the new one has been cloned had already disabled hostname verification, these steps are not required (the hostname verification settings is propagated to the cloned server).

Select WLS_OSBn in the Names column of the table. The Settings page for the server appears.

Click the SSL tab.

Click Advanced.

Set Hostname Verification to None.

Click Save.

If it is not already started, start the Node Manager on the node. To start the Node Manager, use the installation in shared storage from the existing nodes as follows:

OSBHOSTN> WL_HOME/server/bin/startNodeManager

Start and test the new managed server from the Administration Console.

Shut down the existing managed servers in the cluster.

Ensure that the newly created managed server, WLS_OSBn, is up.

Access the application on the newly created managed server (http://vip:port/sbinspection.wsil).

Configure Server Migration for the new managed server.

Note:

Since this is a scale-up operation, the node should already contain a Node Manager, an environment configured for server migration, and the floating IP for the new OSB managed server.

Configure server migration following these steps:

Log into the Administration Console.

In the left pane, expand Environment and select Servers.

Select the name of the new managed server for which you want to configure migration.

Click the Migration tab.

In the Available field, in the "Migration Configuration" section, select the machines to which to enable migration and click the right arrow. Select the same migration targets as for the servers that already exist on the node.

For example, for new managed servers on OSBHOST1, which is already running WLS_OSB1, select OSBHOST2. For new managed servers on OSBHOST2, which is already running WLS_OSB2, select OSBHOST1.

Make sure the appropriate resources are available to run the managed servers concurrently during migration.

Select the "Automatic Server Migration Enabled" option. This enables the Node Manager to start a failed server on the target node automatically.

Click Save.

Restart the Administration Server, managed servers, and Node Manager.

Test server migration for this new server. Follow these steps from the node where you added the new server:

Stop the WLS_OSBn managed server.

To do this, run "kill -9 <pid>" on the PID of the managed server. You can identify the PID of the node using "ps -ef | grep WLS_OSBn".

Note:

For Windows, you can stop the Managed Server by using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server run the following command and identify the pid of the WLS_OSBn Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager Console. You will see a message indicating that WLS_OSBn's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_OSBn. Node Manager waits for a fence period of 30 seconds before trying this restart.

After Node Manager restarts the server, stop it again. Node Manager logs a message indicating that the server does not restart again locally.

Note:

After a server migrates, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager starts the managed server on the machine to which it was originally assigned.

5.14.22.2 Scaling Out the Topology (Adding Managed Servers to New Nodes)

When you scale out the topology, you add new managed servers configured with OSB to new nodes.

Follow the steps in this section for scaling out the topology for the OSB server.

Before performing the steps in this section, check that you meet these requirements:

There must be existing nodes running managed servers configured with OSB within the topology.

The new node optionally can access the existing home directories for WebLogic Server and OSB installation. (Use the existing installations in shared storage for creating a new WLS_OSB managed server. You do not need to install WebLogic Server or OSB binaries in every new location in this case, but you do need to run pack and unpack to bootstrap the domain configuration in the new node, unless you are scaling the OSB server to machines containing other servers of the same domain (the SOA servers).)

When multiple servers in different nodes share an ORACLE_HOME or WL_HOME, Oracle recommends keeping the Oracle Inventory and Middleware home list in those nodes updated for consistency in the installations and application of patches. To update the oraInventory in a node and attach an installation in a shared storage to it, use ORACLE_HOME/oui/bin/attachHome.sh. To update the Middleware home list to add or remove a WL_HOME, edit the user_home/bea/beahomelist file. See the following steps.

To scale out the topology:

On the new node, mount the existing Middleware home. It should include the OSB and SOA (if homes are shared) installation and the domain directory. Ensure that the new node has access to this directory, just like other nodes in the domain.

To attach ORACLE_HOME in shared storage to the local Oracle Inventory, run this command:

To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.

Log in to the Oracle WebLogic Administration Console.

Create a new machine for the new node that will be used, and add the machine to the domain.

Update the machine's Node Manager's address to map the IP of the node that is being used for scale out.

Use the Administration Console to clone WLS_OSB1 into a new managed server. Name it WLS_OSBn, where n is a number and assign it to the new machine.

Note:

These steps assume that you are adding a new server to node n, where no managed server was running previously.

For the listen address, assign the virtual hostname to use for this new managed server. If you plan to use server migration as recommended for this server, this virtual hostname enables it to move to another node. The virtual hostname should be different from those in use by other managed servers (may be in the same or different domain) that are running in the nodes used by the OSB/SOA domain.

Perform these steps to set the managed server listen address:

Log into the Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers.

Select the managed server whose listen address you want to update in the Names column of the table.

Set the Listen Address to OSBHOSTnVHN1.

Click Save.

Save and activate the changes.

The changes will not take effect until the managed server is restarted.

Update the cluster address to include the new server:

Select Environment -> Cluster from the Administration Console.

Click on the OSB_Cluster server. The Settings Screen for the OSB_Cluster appears.

In the Change Center, click Lock & Edit.

Add the new server's address and port to the Cluster Address field. For example:

OSBHOST1VHN1:8011,OSBHOST2VHN1:8011,OSBHOSTNVHN1:8011

Create JMS servers and persistent stores for OSB reporting/internal destinations on the new managed server.

Use the Administration Console to create a new persistent store for the new WseeJMSServer and name it, for example, OSB_rep_JMSFileStore_N. Specify the path for the store. This should be a directory on shared storage as recommended in Section 5.14.1.3, "Shared Storage Prerequisites."

Note:

This directory must exist before the managed server is started or the start operation fails.

Create a new JMS Server for OSB, for example, OSB_rep_JMSServer_N. Use the OSB_rep_JMSFileStore_N for this JMSServer. Target the OSB_rep_JMSServer_N Server to the recently created managed server (WLS_OSBn).

Update the SubDeployment targets for the jmsresources OSB JMS Module to include the recently created OSB JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click jmsresources (represented as a hyperlink in the Names column of the table). The Settings page for jmsResources appears. Open the SubDeployments tab. The subdeployment module for jmsresources appears.

Note:

This subdeployment module name is a random name in the form of wlsbJMSServerXXXXXX resulting from the Configuration Wizard JMS configuration for the first two servers (WLS_OSB1 and WLS_OSB2).

Click the wlsbJMSServerXXXXXX subdeployment and update the targets to include the new OSB_rep_JMSServer_N server.

Create JMS Servers, persistent stores and destinations for OSB JAX-RPC on the new managed server.

Note:

WebLogic Advanced Web Services for JAX-RPC Extension uses regular (non-distributed) destinations to ensure that a locally processed request on a service gets enqueued only to a local member.

Use the Administration Console to create a new persistent store for the new WseeJMSServer and name it, for example, Wsee_rpc_JMSFileStore_N. Specify the path for the store. This should be a directory on shared storage as recommended inSection 5.14.1.3, "Shared Storage Prerequisites":

Note:

This directory must exist before the managed server is started or the start operation fails.

Create a new JMS Server for OSB JAX-RPC, for example, OSB_rpc_JMSServer_N. Use the Wsee_rpc_JMSFileStore_N for this JMSServer. Target the OSB_rpc_JMSServer_N Server to the recently created Managed Server (WLS_OSBn).

Update the WseeJMSModule OSB JMS Module with destinations and the recently created OSB JMS Server. To do this, expand the Services node and then expand the Messaging node. Choose JMS Modules from the Domain Structure window of the Administration Console. The JMS Modules page appears. Click WseeJmsModule (represented as a hyperlink in the Names column of the table). The Settings page for WseeJmsModule appears. Follow steps d through v to complete this step.

In the Change Center, click Lock and Edit. Click New.

Select Queue.

Click Next.

Enter DefaultCallbackQueue-WseeJmsServer_auto_n as name for the queue.

Enter weblogic.wsee.DefaultCallbackQueue-WseeJmsServer_auto_n as the JNDI name.

Click Next.

Click Create a New Subdeployment.

Accept the default name.

Click OK.

Select OSB_rpc_JMSServer_n as the target.

Click Finish.

Activate the changes.

Update the local JNDI name for the destination by following steps q though u.

In the Change Center, click Lock & Edit.

In the Settings for the WseeJmsModule page, click on the DefaultCallbackQueue-WseeJmsServer_auto_n destination.

In the general Configuration tab, click on Advanced.

Update the local JNDI name to weblogic.wsee.DefaultCallbackQueue.

Activate the changes.

Repeat steps d through n for the DefaultQueue-WseeJmsServer_auto_n queue, using weblogic.wsee.DefaultQueue-WseeJmsServer_auto_n as the JNDi name and weblogic.wsee.DefaultQueue as the local JNDI name.

Create a new SAF agent and target it to the newly added managed server, as follows:

Select persistent store Wsee_rpc_JMSFileStore_N (persistent store created for OSB JAX-RPC). Target the SAF Agent to the new managed server and activate changes.

If your Oracle Service Bus configuration includes one or more business services that use JMS request/response functionality, then you must also perform the following procedure using the Oracle Service Bus Console after adding the new managed server to the cluster:

In the Change Center, click Create to create a session.

Using the Project Explorer, locate and select a business service that uses JMS request/response. Business services of this type display Messaging Service as their Service Type.

At the bottom of the View Details page, click Edit.

If there is a cluster address in the endpoint URI, add the new server to the cluster address.

On the Edit a Business Service - Summary page, click Save.

Repeat the previous steps for each remaining business service that uses JMS request/response.

In the Change Center, click Activate.

Restart the managed server.

Restart the Administration Server.

The business services are now configured for operation in the extended domain.

Note:

For business services that use a JMS MesageID correlation scheme, you must edit the connection factory settings to add an entry to the table mapping managed servers to queues. For information on how to configure queues and topic destinations, see "JMS Server Targeting" in Oracle Fusion Middleware Configuring and Managing JMS for Oracle WebLogic Server.

If your Oracle Service Bus configuration includes one or more proxy services that use JMS endpoints with cluster addresses, then you must also perform the following procedure using the Oracle Service Bus Console after adding the new managed server to the cluster:

In the Change Center, click Create to create a session.

Using the Project Explorer, locate and select a proxy service that uses JMS endpoints with cluster addresses.

At the bottom of the View Details page, click Edit.

If there is a cluster address in the endpoint URI, add the new server to the cluster address.

On the Edit a Proxy Service - Summary page, click Save.

Repeat the previous steps for each remaining proxy service that uses JMS endpoints with cluster addresses.

In the Change Center, click Activate.

Restart the managed server.

The proxy services are now configured for operation in the extended domain.

Update the Oracle Service Bus result cache Coherence configuration for the new server:

Log into the Administration Console. In the Change Center, click Lock & Edit.

In the Domain Structure window, expand the Environment node.

Click Servers. The Summary of Servers page appears.

Click the name of the server (represented as a hyperlink) in the Name column of the table. The settings page for the selected server appears.

Click the Server Start tab.

Enter the following for WLS_OSBn (on a single line, without a carriage returns):

The preceding configuration assumes that servers WLS_OSB1 and WLS_OSB2 are running when WLS_OSBn starts. This will enable WLS_OSBn to join the coherence cluster started by either WLS_OSB1 or WLS_OSB2 using the WKA addresses specified. Also, make sure WLS_OSB1 and WLS_OSB2 are started before WLS_OSBn is started when all three servers are restarted. This will ensure WLS_OSBn join the cluster started by one of WLS_OSB1 or WLS_OSB2. If you want to have a configuration where the order in which the servers are started does not matter, you must add the host and port for WLS_OSBn as WKA for WLS_OSB1 and WLS_OSB2, and also add WLS_OSBn as WKA for WLS_OSBn.

Save and activate the changes (this change requires the Oracle Service Bus servers to be restarted to be effective).

Run the pack command on SOAHOST1 to create a template pack as follows:

Configure TX persistent store for the new server. This should be a location visible from other nodes as indicated in the recommendations about shared storage.

From the Administration Console, select Server_name > Services tab. Under Default Store, in Directory, enter the path to the folder where you want the default persistent store to store its data files.

Disable hostname verification for the new managed server. Before starting and verifying the WLS_OSBn managed server, you must disable hostname verification. You can re-enable it after you have configured server certificates for the communication between the Oracle WebLogic Administration Server and the Node Manager in OSBHOSTn. If the source server from which the new one has been cloned had already disabled hostname verification, these steps are not required (the hostname verification setting is propagated to the cloned server).

Start the Node Manager on the new node. To start the Node Manager, use the installation in shared storage from the existing nodes, and start Node Manager by passing the hostname of the new node as a parameter as follows:

OSBHOSTN> WL_HOME/server/bin/startNodeManager <new_node_ip>

Start and test the new managed server from the Administration Console:

Shut down all the existing managed servers in the cluster.

Ensure that the newly created managed server, WLS_OSBn, is running.Access the application on the newly created managed server (http://vip:port/sbinspection.wsil). The application should be functional.

Configure server migration for the new managed server.

Note:

Since this new node is using an existing shared storage installation, the node is already using a Node Manager and an environment configured for server migration that includes netmask, interface, wlsifconfig script superuser privileges. The floating IP for the new OSB Managed Server is already in the new node.

Configure server migration following these steps:

Log into the Administration Console.

In the left pane, expand Environment and select Servers.

Select the server (represented as hyperlink) for which you want to configure migration from the Names column of the table. The Setting page for that server appears.

Click the Migration tab.

In the Available field, in the Migration Configuration section, select the machines to which to enable migration and click the right arrow.

For example, for new managed servers on OSBHOST1, which is already running WLS_OSB1, select OSBHOST2. For new managed servers on OSBHOST2, which is already running WLS_OSB2, select OSBHOST1.

Note:

It is recommended to specify the least-loaded machine as the migration target for the new server. The required capacity planning must be completed so that this node has enough available resources to sustain an additional managed server.

Select the Automatic Server Migration Enabled option. This enables the Node Manager to start a failed server on the target node automatically.

Click Save.

Restart the Administration Server, managed servers, and Node Manager.

Test server migration for this new server. Follow these steps from the node where you added the new server:

Abruptly stop the WLS_OSBn managed server.

To do this, run "kill -9 <pid>" on the PID of the managed server. You can identify the PID of the node using "ps -ef | grep WLS_OSBn".

Note:

For Windows, terminate the Managed Server using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server, run the following command and identify the pid of the WLS_OSBn Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

Watch the Node Manager Console for a message indicating that WLS_OSBn's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_OSBn. Node Manager waits for a fence period of 30 seconds before trying this restart.

Once Node Manager restarts the server, stop it again. Now Node Manager should log a message indicating that the server will not be restarted again locally.

Note:

After a server is migrated, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager will start the managed server on the machine to which it was originally assigned.

5.15 Configuring High Availability for Oracle BAM

This section describes how to set up a 2 node Highly Available configuration for BAM. This includes the BAM Server and the BAM Web Applications.

Oracle strongly recommends reading the release notes for any additional installation and deployment considerations prior to starting the setup process.

Note:

The architectures and deployment procedures defined in this guide enable simple clustered deployments. The procedures described in these chapters can be used as a building block to enable this and other similar high availability topologies for these Fusion Middleware components. It is also expected that production deployments will use other required procedures, such as associating security policies with a centralized LDAP server. For complete details of secured, multi-tiered architectures, and deployment procedures, please refer to the Enterprise Deployment Guide for the component you are configuring.

Figure 5-54 represents the example architecture that configuration steps in this section create.

Figure 5-54 shows a two-node cluster running Oracle BAM Web Applications. One of them is also running Oracle BAM Server. The Oracle WebLogic Servers are frontended by Oracle HTTP Servers, which loads balance incoming requests to BAM Web Applications. A load balancer front ends the system and distributes requests from clients to the two HTTP Servers. An Oracle RAC database stores metadata and BAM schemas.

5.15.1 Preparing the Environment: Prerequisite Steps Before Setting up a High Availability Configuration for Oracle BAM

The following sections provide prerequisite steps before setting up an Oracle BAM high availability configuration:

5.15.1.1 Database Prerequisites

Oracle BAM requires the presence of a supported database.

To check if your database is certified or to see all certified databases, refer to the "Certified Databases" section in the Certification Document:

SQL>select version from sys.product_component_version where product like 'Oracle%';

5.15.1.2 VIP and IPs Prerequisites

As shown in Table 5-19, you configure the Administration and the BAM Server in BAMHOST1 to listen on two different virtual IPs, as shown in the architecture diagram. This requires the provisioning of the corresponding VIP in the box and related host names in the DNS system in your network. Ensure that the different VIPS are available and are reachable from BAMHOST1 andBAMHOST2 before running the installation.

Table 5-19 Virtual Hosts

Virtual IP

VIP Maps to...

Description

VIP0

BAMHOST1VHN0

BAMHOST1VHN0 is the virtual hostname that is the listen address for the Administration Server and fails over with manual failover of the Administration Server. It is enabled on the node where the Administration Server process is running (BAMHOST1 by default).

VIP1

BAMHOST1VHN1

BAMHOST1VHN1 is the virtual hostname that maps to the listen address for WLS_BAM1 and fails over with server migration of this managed server. It is enabled on the node where WLS_BAM1 process is running (BAMHOST1 by default).

5.15.1.3 Installing and Configuring the Database Repository

This section describes how to install and configure the database repository.

Ensure that the following initialization parameter is set to the required value. It is checked by Repository Creation Assistant.

Table 5-20 Required Initialization Parameters

Parameter

Required Value

Parameter Class

PROCESSES

300 or greater

Static

To check the value of the initialization parameter using SQL*Plus, you can use the SHOW PARAMETER command.

As the SYS user, issue the SHOW PARAMETER command as follows:

SQL> SHOW PARAMETER processes

Set the initialization parameter using the following command:

SQL> ALTER SYSTEM SET processes=300 SCOPE=SPFILE

Restart the database.

Note:

The method that you use to change a parameter's value depends on whether the parameter is static or dynamic, and on whether your database uses a parameter file or a server parameter file. See the Oracle Database Administrator's Guide for details on parameter files, server parameter files, and how to change parameter values.

Host Name: Enter the name of the node that is running the database. For an Oracle RAC database, specify the VIP name, or one of the node names as the hostname: BAMDBHOST1VIRTUAL.

Port: The port number for the database: 1521

Service Name: Enter the service name of the database: bamha.mycompany.com

Username: SYS

Password: Enter the password for the SYS user.

Role: SYSDBA

Click Next.

If you receive the following warning message:

The database you are connecting is with non-UTF8 charset, if you are going to use this DB for multilingual support, you may have data loss. If you are not using for multilingual support you can continue, otherwise we strongly recommend using UTF-8 database.

Click Ignore or Stop.

In the Select Components screen, select Create a New Prefix, and enter a prefix to use for the database schemas, for example, BAMHA

Write down the schema names so they are available in later procedures.

Select the following schemas:

AS Common Schemas:

Metadata Services

SOA Infrastructure:

Business Activity Monitoring

User Messaging

Click Next.

In the Schema Passwords screen, enter passwords for the main and additional (auxiliary) schema users, and click Next.

In the Map Tablespaces screen, choose the tablespaces for the selected components, and click Next.

This section describes the Load Balancer prerequisites for deploying an Oracle BAM high availability environment.

Load Balancers

Oracle BAM uses a hardware load balancer when deployed in a high availability configuration with two Oracle HTTP Servers as web tier components. The hardware load balancer must have the following features:

Ability to load-balance traffic to a pool of real servers through a virtual hostname: Clients access services using the virtual hostname (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.

Port translation configuration: The load balancer must have the ability to perform port translation, where it enables incoming requests received on one port to be routed to a server process running on a different port. For example, a request received on port 80 can be routed to port 7777.

Monitoring of ports (HTTP/HTTPS)

Virtual servers and port configuration: Ability to configure virtual server names and ports on your external load balancer, and the virtual server names and ports must meet the following requirements:

The load balancer must allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, the load balancer typically needs to be configured with a virtual server and ports for HTTP and HTTPS traffic.

The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the load balancer through the virtual server names.

Resource monitoring/port monitoring/ process failure detection: The load balancer must be able to detect service and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your load balancer has the ability to automatically detect failures, you should use it.

Fault-tolerant mode: It is highly recommended that you configure the load balancer to be in fault-tolerant mode.

Other: It is highly recommended that you configure the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client machine.

Sticky routing capability: Ability to maintain sticky connections to components based on cookies or URL.

SSL acceleration: SSL acceleration is a method of offloading the processor-intensive public key encryption algorithms involved in SSL transactions to a hardware accelerator.

This feature is recommended but not required.

Table 5-21 shows some example virtual server names to use for the external load balancer in the Oracle BAM high availability environment:

This section describes the virtual server names that should be set up for the high availability deployments described in this section:

bam.mycompany.com

bam.mycompany.com is a virtual server name that acts as the access point for all HTTP traffic to the runtime BAM Web components. Traffic to the both SSL and non-SSL ports is configured and typically non-SSL is redirected to SSL. Clients access this service using the address bam.mycompany.com:443. This virtual server is defined on the load balancer.

admin.mycompany.com

This virtual server acts as the access point for all internal HTTP traffic that gets directed to the administration services. The incoming traffic from clients is non-SSL enabled. Thus, the clients access this service using the address admin.mycompany.com:80 and in turn forward these to ports 7777 on WEBHOST1 and WEBHOST2. The services accessed on this virtual host include the WebLogic Administration Server Console and Oracle Enterprise Manager.

In addition, ensure that the virtual server names are associated with IP addresses and are part of your Domain Name System (DNS). The computers on which Oracle Fusion Middleware is running must be able to resolve these virtual server names.

Port 7777 is not used by any service on the nodes. You can verify this by running the following command:

Unix:

netstat -an | grep LISTEN | grep ":7777"

Windows:

netstat -an | findstr "LISTEN" | findstr ":7777"

If the ports are in use, make them available.

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct, and that you have write permissions for that directory.

Select Specify Ports using Configuration File and copy the staticports.ini template file from your installation disk (the file is located in the /Disk1/stage/Response directory) to your user's home. Then use the Browse button to select this file.

Click View/Edit File to open the staticports.ini file in an editor.

Change the Oracle HTTP Server port in that file to 7777.

Save the file.

Click Next.

In the Configuration Summary screen, ensure that the selections are correct, and click Install.

In the Installation Progress screen:

For UNIX systems, a dialog box appears prompting you to run the oracleRoot.sh script. Open a command window and run the script, following the prompts.

Click Next.

In the Configuration screen, several configuration assistants are launched in succession. When the configuration assistants are finished, the Configuration Completed screen appears.

In the Configuration Completed screen, click Finish to exit.

Repeat the steps for WEBHOST2 and configure your load balancing router with a pool containing both the WEBHOST1 and WEBHOST2 addresses.

5.15.2.1 Validating Oracle HTTP Server

To verify that Oracle HTTP Server is set up correctly, access the root URL context of the server by using the following URL a browser:

HTTP://WEBHOST1:7777/

If Oracle HTTP Server is set up correctly, the Hello World page appears in the browser.

5.15.3 Installing Oracle Fusion Middleware Home

This section describes how to install Oracle Fusion Middleware on all nodes in the application tier that run Oracle WebLogic Server and Oracle Fusion Middleware BAM.

In the Choose Product Installation Directories screen, accept the following directory:

WL_HOME

Click Next.

In the Installation Summary screen, click Next.

In the Installation Complete screen, deselect Run QuickStart, and click Done.

5.15.3.2 Installing Oracle BAM Using the Oracle SOA Suite Installer

Oracle BAM is installed as part of the Oracle SOA Suite. The 11g Oracle SOA Suite Installer is used to install the required Oracle BAM binaries on all the nodes in the application tier:

On Linux platforms, if the /etc/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct and that you have write permissions for that directory.

Run Oracle Fusion Middleware Configuration Wizard from the SOA home directory to create a domain containing the Administration Server and Oracle BAM components. Ensure that the database where you installed the repository is running. For Oracle RAC databases, all the instances must be running.

Change the directory to the location of Oracle Fusion Middleware Configuration Wizard, located in the SOA/BAM home directory:

BAMHOST1> cd ORACLE_HOME/common/bin

Start Oracle Fusion Middleware Configuration Wizard

For UNIX:

BAMHOST1> ./config.sh

For Windows:

BAMHOST1> config.cmd

In the Welcome screen, select Create a New WebLogic Domain, and click Next.

In the Select Domain Source screen, select Generate a domain configured automatically to support the following products, and select the following products (Basic WebLogic Server domain - 10.3.6.0 is selected by default):

Oracle Business Activity Monitoring

Oracle WSM-PM (selected by default)

Enterprise Manager

JRF (selected by default)

Click Next.

In the Specify Domain Name and Location screen, make the following entries:

Domain Name: bamdomain

Domain Location: accept the default

Application Location: accept the default

Click Next.

In the Configure Administrator Username and Password screen, enter the username and password to be used for the domain's administrator. Click Next.

In the Configure Server Start Mode and JDK screen, make the following selections:

Ensure that the following data source appears on the screen. The user names shown in Table 5-22, assume that bamha was used as the prefix for schema creation from RCU.

Table 5-22 Configuring Values for Data Sources

Data Source

User Name

BAMDataSource

bamha_orabam

OraSDPMDataSource

bamha_orasdpm

mds-owsm

bamha_mds

For the RAC configuration for component schemas, select Convert to GridLink or Convert to RAC multi data source. For single database configuration, select Don't convert. If you select Don't Convert, continue to Step 10.

Click Next.

In the Configure JDBC Component Schema screen, enter values for the following fields, specifying connect information for the Oracle RAC database that was seeded with RCU:

Service Name: Enter the service name of the database. For a GridLink data source, you must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

Note:

The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, example.com

Username: Enter the complete user name (including the prefix) for the schemas.

Password: enter the password to use to access the schemas.

For a GridLink data source, enter the SCAN address in the Listener Address field and SCAN port in the Port field. Enter the ONS host and port information in the ONS Host and Port fields, respectively.

Note:

Oracle recommends that you use SCAN addresses to specify the host and port for both the TNS listener and the ONS listener in the WebLogic console. You do not need to update a GridLink data source containing SCAN addresses if you add or remove Oracle RAC nodes. Contact your network administrator for appropriately configured SCAN urls for your environment. See SCAN Addresses in the Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server guide.

In the Test JDBC Data Sources screen, the connections are tested automatically. The Status column shows the results. Ensure that all connections were successful. If not, click Previous to correct your entries.

In the Configure the Administration Server screen, enter the following values:

Name: AdminServer

Listen Address: VIP0

Listen Port: 7001

SSL listen port: N/A

SSL enabled: leave unchecked

Click Next.

In the Configure Managed Servers screen, add the following managed servers:

Table 5-23 Configuring Managed Servers

Name

Listen Address

Listen Port

SSL Listen Port

SSL Enabled

WLS_BAM1

BAMHOST1VHN1

8001

n/a

No

WLS_BAM2

BAMHOST2 (BAM Server WLS_BAM2 does not use server migration)

8001

n/a

No

Note:

Although the standard recommendation is to run custom applications and other systems in a separate WebLogic Managed Server, the creation of custom WLS managed servers described in Figure 5-54 is not addressed here.

Click Next.

In the Configure Clusters screen, add the following cluster:

Name: BAM_Cluster

Cluster Messaging Mode: unicast

Multicast Address: N/A

Multicast Port: N/A

Cluster Address: Leave empty

Click Next.

In the Assign Servers to Clusters screen, assign the following servers to BAM_Cluster:

WLS_BAM1

WLS_BAM2

Click Next.

In the Configure Machines screen:

Delete the LocalMachine that appears by default.

Click the Unix Machine tab, and add the following machines:

Table 5-24 Configuring Machines

Name

Node Manager Listen Address

BAMHOST1

BAMHOST1

BAMHOST2

BAMHOST2

Leave all other fields to their default values, and click Next.

In the Assign Servers to Machines screen, assign servers to machines as follows:

BAMHOST1: AdminServer, WLS_BAM1

BAMHOST2: WLS_BAM2

Click Next.

In the Target Deployments to Cluster or Services screen, ensure the following targets:

User Messaging Deployments should be targeted only to BAM_Cluster. (The usermessaging-xmpp, usermessaging-smpp, and usermessaging-voicexml applications are optional.)

wsm-pm should be targeted only to BAM_Cluster.

The DMS Application should be targeted to BAM_Cluster and Administration Server.

The oracle.rules.*, oracle.sdp.* and oracle.bam.* deployments should be targeted only to BAM_Cluster.

The oracle.wsm.seedpolicies library should be targeted only to the BAM_Cluster.

In the Target Services to Cluster or Servers screen, ensure the following targets:

WSM Startup Class should be targeted only to BAM_Cluster.

mds-wsm, mds-wsm-rac0, and mds-wsm-rac1 should be targeted to both BAM_Cluster and AdminServer.

OrasDPMDatasource, OrasDPMDatasource-rac0, and OrasDPMDatasource-rac1 should be targeted to the BAM_Cluster.

OWSM Startup Class is only targeted to BAM_Cluster *DMS Startup is targeted both to BAM_Cluster and AdminServer*mds-wsm, mds-wsm-rac0, and mds-wsm-rac1 should be targeted to both BAM_Cluster and AdminServer

mds-soa, mds-soa-rac0, and mds-soa-rac1 should be targeted to both BAM_Cluster and AdminServer.

In the Configuration Summary screen, click Create.

In the Creating Domain screen, click Done.

5.15.6 Creating boot.properties for the Administration Server and for WLS_BAM1 on BAMHOST1

This is an optional step for enabling the Administration Server to start without prompting you for the administrator username and password. Create a boot.properties file for the Administration Server on BAMHOST1.

5.15.8 Disabling Host Name Verification for the Servers

This step is required if you have not set up the appropriate certificates to authenticate the different nodes with the Administration Server. If you have not configured the server certificates, you receive errors when managing the different WebLogic Servers. To avoid these errors, disable hostname verification while setting up and validating the topology, and enable them again after the high availability topology configuration is complete.

To disable hostname verification:

In the Administration Console, select Administration Server, SSL, and then Advanced.

Set Hostname Verification to None.

In the Administration Console, select WLS_BAM1, SSL, and then Advanced.

Set Hostname Verification to None.

In the Administration Console, select WLS_BAM2, SSL, and then Advanced.

Set Hostname Verification to None.

Restart the Administration Server for the changes to take affect.

Save and activate the changes.

5.15.9 Configuring a JMS Persistence Store for BAM UMS

Configure the location for all of the persistence stores as a directory that is visible from both BAMHOST1 and BAMHOST2. This is required in order to enable the resume of transactions when a server is migrated to a different node. By using a shared location for the persistent stores from both nodes, it is guaranteed that no messages are lost should a failover take place.

In the Domain Structure window, expand the Services node and then click the Persistence Stores node.

The Summary of Persistence Stores page appears.

Select the a persistent store (represented as a hyperlink) from the Name column of the table.

The Settings page for the persistence store appears.

In the Configuration tab, enter the location on a persistent storage solution (such as NAS or SAN) that is available to other servers in the cluster in the Directory field. Specifying this location enables pending JMS messages to be sent.

Each server has a transaction log that stores information about committed transactions that are coordinated by the server that may not have been completed. The WebLogic Server uses this transaction log for recovery from system fails or network failures. To leverage the migration capability of the Transaction Recovery Service for the servers within a cluster, store the transaction log in a location accessible to a server and its backup servers.

Note:

Preferably, this location should be a dual-ported SCSI disk or on a Storage Area Network (SAN).

To set the location for the default persistence store:

Log into the Administration Console.

In the Domain Structure window, expand the Environment node and then click the Servers node.

The Summary of Servers page appears.

Click the name of the server (represented as a hyperlink) in Name column of the table.

The settings page for the selected server appears and defaults to the Configuration tab.

Click the Services tab.

In the Default Store section of the page, enter the path to the folder where the default persistent stores will store its data files.

Save and Activate the changes.

Note:

To enable migration of the Transaction Recovery Service, specify a location on a persistent storage solution that is available to other servers in the cluster. This directory must also exist before you restart the server.

5.15.11 Untargeting the BAM Server System from BAMHOST2

Because the BAM server component in BAM is a singleton, you must untarget it from one of the WLS_BAM servers before you configure it for server migration. Otherwise the system would use two active BAM Servers which could cause different data inconsistencies. This way, BAM Web applications run in both BAMHOST1 and BAMHOST2, but BAM Server is initially active only in BAMHOST1.

In this step, you remove the BAM server runtime from WLS_BAM2. To untarget the BAM server artifacts from WLS_BAM2:

Log into the Oracle WebLogic Administration Console at http://BAMHOST1VHN0:7001/console.

5.15.13 Starting Node Manager on BAMHOST1 and BAMHOST2

To start Node Manager on BAMHOST1 and BAMHOST2:

Run the setNMProps.sh script, located in the ORACLE_COMMON_HOME/common/bin directory.

This script sets the StartScriptEnabled property to true before starting Node Manager, and allows staring servers from the Administration Console (the environment variables required for the BAM Servers are set in the default start script in the domain directory):

BAMHOST1> cd ORACLE_COMMON_HOME/common/bin
BAMHOST1> ./setNMProps.sh

Note:

You must use the StartScriptEnabled property to avoid class loading failures and other problems.

Start Node Manager using the following command:

BAMHOST1> cd WL_HOME/server/bin
BAMHOST1> ./startNodeManager.sh

Repeat step1 and 2 for Node Manager in BAMHOST2.

5.15.14 Starting the Oracle BAM System

To start the WLS_BAM1 managed server on BAMHOST1:

Start the WLS_BAM1 managed servers:

Log into the Administration Console using the following URL:

http://bamhost1vhn0:7001/console

In the Domain Structure window, expand the Environment node and then select Servers.

Log into the Administration Console at http://bamhost1vhn0:7001/console.

In the Domain Structure window, expand the Environment node and then select Servers.

The Summary of Servers page appears.

Click the Control tab.

Select WLS_BAM2 from the Servers column of the table.

Click Start.

Access HTTP://BAMHOST2:9001/OracleBAM to verify status of WLS_BAM2.

If the managed servers fails to start with the following message:

Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
The Connection descriptor used by the client was <db_connect_string>

Verify that the PROCESSES initialization parameter for the database is set to a high enough value.

5.15.15 Configuring Oracle RAC Failover for the WLS_BAM Servers

Oracle BAM enables customizing the behavior of a BAM server when Oracle RAC is used as the repository for the BAM schemas and a failure occurs in the database. The properties that allow this customization can be adjusted depending on the application and based on the desired expected behavior for each BAM system. The properties are configured in the Fusion Middleware Control Console System MBean Browser, or in the corresponding Oracle BAM-specific XML configuration file.

If you want to completely disable Oracle BAM's failover to the database in an Oracle RAC configuration set UseDBFailover to false in the Fusion Middleware Control Console System MBean Browser for your BAM server. The default value of this property is true, therefore, failover is provided. You can also increase or decrease the number of retries in the access to the database when there is a database instance failure (for BAM to retry the in flight transactions). To adjust the number of retries change the MaxDBNodeFailoverRetries in the Fusion Middleware Control Console System MBean Browser. The default value for MaxDBNodeFailoverRetries is 5 times. See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite for more details on the properties that can be configured for Oracle BAM.

5.15.16 Configuring the BAM Web Applications to Use the BAM Server in BAMHOST1

To configure the OracleBamWeb (WLS_BAM1) and OracleBamWeb (WLS_BAM2) applications to use the BAM server in BAMHOST1:

Access Oracle Enterprise Manager Fusion Middleware Control at the following URL

http://bamhost1vhn0:7001/em

Expand BAM in the navigation tree.

Right-click OracleBamWeb(WLS_BAM1).

Select BAM Web Properties from the context menu.

The BAM Web Properties page appears.

Define the following properties:

Enter WEBHOST:7777 for the application URL.

Enter BAMHOST1VHN1 for the server name.

Modify the listening port for the BAM server using the Mbean browser. To do this, follow these steps:

5.15.18 Validating Access through Oracle HTTP Server

Verify that the BAM servers' status is reported as "Running" in the Admin Console. If the server is shown as "Starting" or "Resuming," wait for the server status to change to "Started." If another status is reported (such as "Admin" or "Failed"), check the server output log files for errors. Verify that you can access these URLs, where 'WEBHOSTN' specifies the name of each Oracle HTTP Server host (for example, WEBHOST1, WEBHOST2):

http://WEBHOST1:7777/wsm-pm

http://WEBHOST2:7777/wsm-pm

http://WEBHOST1:7777/sdpmessaging/userprefs-ui

http://WEBHOST2:7777/sdpmessaging/userprefs-ui

http://WEBHOST1:7777/OracleBAM

http://WEBHOST2:7777/OracleBAM

Verify these URLs also using the load balancing router address:

http://bam.mycompany.com:80/wsm-pm

http://bam.mycompany.com:80/sdpmessaging/userprefs-ui

http://bam.mycompany.com:80/OracleBAM

Verify the following URLs to ensure that routing and failover from the HTTP Server to the BAM cluster is working correctly:

While WLS_BAM2 is running, stop WLS_BAM1 from Administration Console.

Access the following URLs and verify the appropriate functionality:

http://WEBHOST1:7777/wsm-pm

http://WEBHOST1:7777/sdpmessaging/userprefs-ui

http://WEBHOST1:7777/OracleBAM

Start WLS_BAM1 from Administration Console.

Stop WLS_BAM2.

Access the URLs from step 2 above again and verify the appropriate functionality.

5.15.19 Configuring Server Migration for the WLS_BAM Servers

The high availability architecture for Oracle BAM uses server migration to protect singleton services against failures. The WLS_BAM1 managed server is configured so that it can be restarted on BAMHOST2 in case of failure. For this configuration, WLS_BAM1 listens on a specific floating IP address that is failed over by Oracle WebLogic Server Migration. To configure server migration for the WLS_BAM1 managed servers, follow these steps:

Note:

If server migration was configured previously for SOA, the BAM system can use the same data sources and database schemas. In that case, steps 1 through 5 may not be required, but you must also target the corresponding server-migration/leasing datasources to the BAM Cluster.

5.15.19.1 Setting Up the User and Tablespace for the Server Migration Leasing Table

To create the user and tablespace:

Create a tablespace called leasing. For example, log on to SQL*Plus as the sysdba user and run the following command:

Example: Log on to SQL*Plus as the sysdba user and run the following command:

Copy the leasing.ddl file located in the WL_HOME/server/db/oracle directory to your database node.

Connect to the database as the leasing user.

Run the leasing.ddl script in SQL*Plus.

SQL> @copy_location/leasing.ddl;

5.15.19.2 Creating a GridLink or Multi Data Source from the Administration Console

You create a data source to each of the Oracle RAC database instances during the process of setting up the multi data source, both for these data sources and the global leasing multi data source. When you create a data source:

Ensure that this is a non-xa data source.

Use Oracle's Driver (Thin) Version 9.0.1, 9.2.0, 10, 11.

Do not use any type of distributed transaction emulation/participation algorithm for the data source (do not choose the Supports Global Transactions option, or the Logging Last Resource, Emulate Two-Phase Commit, or One-Phase Commit options of the Supports Global Transactions option), and specify a service name for your database. Data sources do not require support for global transactions.

Target these data sources to the cluster(s).

Make sure the datasources' connection pool initial capacity is set to 0. To do this, select Services, JDBC, and then Datasources. In the Datasources screen, click the Datasource Name, then click the Connection Pool tab, and enter 0 in the Initial capacity field.

Ensure that an ONS daemon is running on your database servers at all times. To start the ONS daemon on a database server, run the onsctl command: start

Creating a GridLink Data Source

To create a GridLink Data Source:

Log in to the Administration Console.

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.

Service Name: Enter the service name of the database. You must enter the RAC service name in lower-case letters followed by the domain name example.com. For example, <mydbservice>.example.com

Note:

The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, example.com

Host Name - The DNS name or IP address of the server that hosts the database. For an Oracle GridLink service-instance connection, this must be the same for each data source in a given multi data source.

Port - The port on which the database server listens for connections requests.

On the Test GridLink Database Connection page, review the connection parameters and click Test All Listeners.

Oracle WebLogic creates a connection from the Administration Server to the database. Results from the connection test appear at the top of the page. If the test is unsuccessful, correct any configuration errors and retry the test.

Click Next.

In the ONS Client Configuration page, do the following:

Select Fan Enabled to subscribe to and process Oracle FAN events.

In ONS host and port, enter a comma-separate list of ONS daemon listen addresses and ports for receiving ONS-based FAN events. You can use Single Client Access Name (SCAN) addresses to access FAN notifications.

When creating the multi data sources for the leasing table, enter names in the format of <MultiDS>-rac0, <MultiDS>-rac1, and so on.

Click Next.

Deselect Supports Global Transactions.

Click Next.

Enter the service name, database name (the RAC Node instance name, for example: racdb1,racdb2), host port, and password for your leasing schema.

Click Next.

Click Test Configuration and verify the connection works.

Target the data source to the BAM cluster.

Select the data source and add it to the right screen.

Click Create a New Data Source second instance of your Oracle RAC database, target it to BAM_Cluster, repeating the steps for the second instance of your Oracle RAC database.

Add the second data source to your multi data source.

Click Save.

Click Activate Changes.

5.15.19.3 Edit the Node Manager's Properties File

The nodemanager.properties file is in the WL_HOME/server/bin directory.

Interface=eth0
NetMask=255.255.255.0
UseMACBroadcast=true

Interface=eth0

This property specifies the interface name for the floating IP (eth0, for example, on Linux).

Note:

Do not specify the sub interface, such as eth0:1 or eth0:2. This interface is to be used without the:0 or :1. The Node Manager's scripts traverse the different :X enabled IPs to determine which to add or remove. For example, the valid values in Linux environments are eth0, eth1, or, eth2, eth3, ethn, depending on the number of interfaces configured.

Note:

For Windows, the Interface should be set to the Network Interface Name. For example: Interface="Local Area Connection".

NetMask

This property specifies the net mask for the interface for the floating IP.

UseMACBroadcast

This property specifies whether or not to use a node's MAC address when sending ARP packets, that is, whether or not to use the -b flag in the arping command.

Perform this configuration in the two nodes BAMHOST1 and BAMHOST2. Verify in the output of Node Manager (the shell where the Node Manager is started) that these properties are in use. Otherwise, problems may occur during migration. The output should be similar to the following:

5.15.19.4 Set Environment and Superuser Privileges for the wlsifconfig.sh Script

To set the environment and superuser privileges for the wlsifconfig.sh script:

Note:

On Windows, the script is named wlsifconfig.cmd and it can be run by users with the administrator privilege.

Ensure that your PATH environment variable includes the files listed in Table 5-26.

Table 5-26 Files Required to be in the PATH Environment Variable

File

Located in this directory

wlsifconfig.sh

MW_HOME/user_projects/domains/bamdomain/bin/server_migration

wlscontrol.sh

WL_HOME/common/bin

nodemanager.domains

WL_HOME//common

Grant sudo configuration for the wlsifconfig.sh script.

Configure sudo to work without a password prompt.

For security reasons, sudo should be restricted to the subset of commands required to run the wlsifconfig.sh script. For example, to set the environment and superuser privileges for the wlsifconfig.sh script, complete these steps:

Grant sudo privilege to the WebLogic user ('oracle') with no password restriction, and grant execute privilege on the /sbin/ifconfig and /sbin/arping binaries.

Make sure the script is executable by the WebLogic user ('oracle'). The following example shows an entry inside /etc/sudoers granting sudo execution privilege for oracle and also over ifconfig and arping:

Ask the system administrator for the sudo and system rights as appropriate to this step.

5.15.19.5 Configure Server Migration Targets

Configuring Cluster Migration sets the DataSourceForAutomaticMigration property to true. To configure cluster migration in a cluster:

Log into the Administration Console.

In the Domain Structure window, expand Environment and select Clusters.

Click the cluster for which you want to configure migration (BAM_Cluster) in the Name column of the table.

Click the Migration tab. In the Change Center, click Lock & Edit.

In the Available field, select the machine to which to enable migration and click the right arrow. In this case, select BAMHOST1 and BAMHOST2.

Select the data source to be used for automatic migration. In this case select the leasing data source.

Click Save.

Set the Candidate Machines for Server Migration.

You must perform this task for WLS_BAM1:

In Domain Structure window of the Administration Console, expand Environment and select Servers.

Select the server which you want to configure migration for. Click the Migration tab

In the Available field, located in the Migration Configuration section, select the machines to which to enable migration and click the right arrow. Select BAMHOST2 for WLS_BAM1.

Select Automatic Server Migration Enabled.

This enables the Node Manager to start a failed server on the target node automatically.

Click Save and Activate the changes.

Restart the Administration Server and the WLS_BAM1 server.

5.15.19.6 Test Server Migration

To verify that Server Migration is working properly, follow these steps:

From BAMHOST1:

Force stop the WLS_BAM1 managed server using the following command:

BAMHOST1> kill -9 <pid>

pid specifies the process ID of the managed server. You can identify the pid in the node by running the following command:

BAMHOST1> ps -ef | grep WLS_BAM1

Note:

For Windows, terminate the Managed Server using the taskkill command. For example:

taskkill /f /pid <pid>

Where <pid> is the process Id of the Managed Server.

To determine the process Id of the Managed Server, run the following command and identify the pid of the WLS_BAM1 Managed Server.

MW_HOME\jrockit_160_<version>\bin\jps -l -v

In the Node Manager console, a message indicates that WLS_BAM1's floating IP has been disabled.

Wait for the Node Manager to try a second restart of WLS_BAM1.

Node Manager waits for a fence period of thirty seconds before trying this restart.

After Node Manager restarts the server, stop it again.

Node Manager should now log a message indicating that the server will not be restarted again locally.

From BAMHOST2:

Watch the local Node Manager console.

After thirty seconds since the last try to restart WLS_BAM1 on Node 1, Node Manager on BAMHOST2 should prompt that the floating IP for WLS_BAM1 is being brought up and that the server is being restarted in this node.

Access the Oracle BAM console using BAMHOST1VHN1/OracleBAM and WEBHOST1:7777/OracleBAM.

Verification From the Administration Console

To verify Migration in the Administration Console:

Log into the Administration Console. Click on Domain on the left console.

Click the Monitoring tab and then on the Migration tab.

The Migration Status table provides information on the migration status.

Note:

To fail a server back to its original node/machine after it migrates, use the Oracle WebLogic Administration Console to stop and restart the managed server. The appropriate Node Manager starts the managed server on the machine that it was originally assigned to.

5.15.20 Configuring Clients Connecting to the BAM System

When accessing a BAM server using the BAM Adapter (RMI), the virtual hostname of the BAM server (BAMHOST1VNH1) should be used for the connection. SOAP requests come through HTTP, therefore it is required to use the load balancing router addresses when using the BAM Adapter in this situation.

Scripting on this page enhances content navigation, but does not change the content in any way.