6.1 Oracle ADF and High Availability Concepts

Oracle ADF is an end-to-end application framework that builds on Java Platform, Enterprise Edition (Java EE) standards and open-source technologies to simplify and accelerate implementing service-oriented applications. Oracle ADF is suitable for enterprise developers who want to create applications that search, display, create, modify, and validate data using web, wireless, desktop, or web services interfaces. Used in tandem, Oracle JDeveloper 11g and Oracle ADF provide an environment that covers the full development lifecycle from design to deployment, with drag-and-drop data binding, visual UI design, and team development features built in.

6.1.1 Understanding Oracle ADF

In line with community best practices, applications built using the Fusion web technology stack achieve a clean separation of business logic, page navigation, and user interface by adhering to a model-view-controller (MVC) architecture supported by the following layers.

6.1.1.1 Oracle ADF Components

The core module in the framework is Oracle ADF Model, a declarative data binding facility that implements the JSR-227 specification. This specification provides an API for accessing declarative data binding metadata. The Oracle ADF Model layer enables a unified approach to bind any user interface to any business service, without the need to write code.

The other modules that make up a Fusion web application technology stack are:

Oracle ADF Business Components, which simplifies building business services.

Oracle ADF Faces, which offers a rich library of AJAX-enabled UI components for Web applications built with JavaServer Faces (JSF).

Oracle ADF Controller, which integrates JSF with Oracle ADF Model. The ADF Controller extends the standard JSF controller by providing additional functionality, such as reusable task flows that pass control not only between JSF pages, but also between other activities, for instance method calls or other task flows.

6.1.1.1.1 ADF Business Components

ADF Business Components are prebuilt application objects that accelerate the job of delivering and maintaining high-performance, richly functional, database-centric services. When building service-oriented Java EE applications, developers implement the core business logic as one or more business services. These backend services provide clients with a way to query, insert, update, and delete business data as required while enforcing appropriate business rules. ADF Business Components provides a ready-to-use implementation of Java EE design patterns and best practices.

Oracle ADF Business Components provides the following key components to simplify building database-centric business services:

Entity object

An entity object represents a row in a database table and simplifies modifying its data by handling all data manipulation language (DML) operations. It can encapsulate business logic to ensure that business rules are consistently enforced. Developers can associate an entity object with others to reflect relationships in the underlying database schema to create a layer of business domain objects to reuse in multiple applications.

View object

A view object represents a SQL query and simplifies working with its results. Developers use the SQL language to join, project, filter, sort, and aggregate data into the shape required by the end-user task being represented in the user interface. This includes the ability to link a view object with other entity objects to create master-detail hierarchies of any complexity. When end users modify data in the user interface, view objects collaborate with entity objects to consistently validate and save the changes.

Application module

An application module is the transactional component that UI clients use to work with application data. It defines an updateable data model and top-level procedures and functions (called service methods) related to a logical unit of work related to an end-user task.

6.1.1.1.2 ADF Model Layer

In the model layer, Oracle ADF Model implements the JSR-227 service abstraction called the data control. Data controls abstract the implementation technology of a business service by using standard metadata interfaces to describe the service's operations and data collections, including information about the properties, methods, and types involved. In Oracle JDeveloper, developers can view that information as icons that they can easily drag and drop onto a page. When the developer drags the representation of the service onto the page, Oracle JDeveloper automatically creates the bindings from the page to the services. At runtime, the ADF Model layer reads the information describing the application's data controls and data bindings from appropriate XML files and implements the two-way connection between the user interface and the application's business service.

Oracle ADF provides out-of-the-box data control implementations for the most common business service technologies. Using Oracle JDeveloper and Oracle ADF together provides a declarative, drag-and-drop data binding experience for building user interfaces. Along with support for ADF Business Components application modules, ADF Model also provides support for the following service technologies:

6.1.1.1.3 ADF Controller

In the controller layer, where handling page flow of the web applications is a key concern, ADF Controller provides an enhanced navigation and state management model on top of JSF. JDeveloper supports declarative creation of task flows that can manage application control between different types of activities, such as pages, methods on managed beans, declarative case statements, or calls to other task flows.

6.1.1.1.4 ADF Faces Rich Client

ADF Faces rich client (ADF Faces for short), is a set of standard JSF components that include built-in AJAX functionality. AJAX is a combination of asynchronous JavaScript, dynamic HTML (DHTML), XML, and XmlHttpRequest communication channel. This combination allows requests to be made to the server without fully rerendering the page. While AJAX allows rich client-like applications to use standard internet technologies, JSF provides server-side control, which reduces the dependency on an abundance of JavaScript often found in typical AJAX applications.

ADF Faces provides over 100 rich components, including hierarchical data tables, tree menus, in-page dialogs, accordions, dividers, and sortable tables. ADF Faces also provides ADF Data Visualization components, which are Flash- and SVG-enabled components capable of rendering dynamic charts, graphs, gauges, and other graphics that can provide a real-time view of underlying data. Each component also supports customization and skinning, along with internationalization and accessibility.

To achieve these front-end capabilities, ADF Faces components use a rendering kit that handles displaying the component and also provides the JavaScript objects needed for the rich functionality. This built-in support enables developers to build rich applications without needing extensive knowledge of the individual technologies on the front or back end.

6.1.1.2 Oracle ADF Single Node Architecture

You can install the Oracle ADF runtime to the Oracle WebLogic Server using either the Oracle JDeveloper Installer or the Oracle Fusion Middleware Application Developer Installer. The Application Developer Installer also lets you optionally install Fusion Middleware Control to provide web-based administration support for all Managed Servers in the domain. The Oracle JDeveloper installer does not install Fusion Middleware Control. Both of these options are described in the deployment chapter of the Oracle Fusion Middleware Fusion Developer's Guide for Oracle Application Development Framework.

When you use the Application Developer Installer to install the Oracle ADF runtime, it results in the creation of an Oracle Application Developer home directory (by default, Oracle_APPDEV1) located under the Middleware home. After the administrator uses the domain configuration wizard to create an Application Developer domain (base_domain) based on the JRF domain template, the administrator can configure the topology of the server. In a typical set up, the domain contains an Administration server containing the WLS Administration Console and Fusion Middleware Control. Typically, the Oracle ADF runtime libraries (part of the Java Required Files) get deployed to the Managed Servers, in addition to the user-facing custom Fusion web applications. To provide customization and personalization features, an optional MDS repository may also be installed, and needs to be configured separately. Figure 6-3 shows a basic single-node Oracle ADF architecture.

6.1.1.3 Oracle ADF External Dependencies

If the Fusion web application involves customization using Oracle Metadata Services (MDS), you should register your MDS repository with the Oracle WebLogic Server domain before you deploy the application. For information about registering MDS, see the Oracle Fusion Middleware Administrator's Guide.

Then, when you deploy the application, JDeveloper prompts you to choose the target metadata repository or shared metadata repository. You will be able to choose from the list of metadata repositories registered with the Oracle WebLogic Administration Server.

To ensure that you receive the metadata repository prompt, the application's adf-config.xml file must define a cust-config element in the mds-config section. This element specifies an ordered and named list of customization classes. A customization class is the interface that MDS uses to define which customization applies to the base definition metadata. In JDeveloper, you can use the overview editor for the adf-config.xml file to define a cust-config element.

6.1.1.4 Oracle ADF Log File

The operations performed by the Fusion web application are logged directly to the WebLogic Managed Server where the application is running:

DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log

The log files for the different WebLogic Managed Servers are also available from the Oracle WebLogic Server Administration Console. To verify the logs, access the Oracle WebLogic Server Administration Console http://<admin_server_host>:<port>/console and click on Diagnostics-Log Files.

This log's granularity and logging properties can be changed using Oracle Enterprise Manager Fusion Middleware Control (Fusion Middleware Control). Fusion Middleware Control is a Web browser-based, graphical user interface that you can use to monitor and administer a farm. To receive high availability warning diagnostic messages for Oracle ADF, the level should be set to FINE, as described in Section 6.1.4.3, "Troubleshooting Oracle ADF Replication and Failover Issues."

6.1.2 Oracle ADF High Availability Considerations

Fusion web applications built on the Oracle ADF technology stack are Java EE applications (and J2EE applications). You should observe the same best practices for Fusion web applications as you would any other Java EE application in a high availability environment.

6.1.2.1 Oracle ADF Scope and Session State

At runtime, ADF objects such as the binding container and managed beans are instantiated. Each of these objects has a defined life span set by its scope attribute.

There are six types of scopes in a Fusion web application:

Application scope: The object is available for the duration of the application.

Session scope: The object is available for the duration of the session.

Page flow scope: The object is available for the duration of a bounded task flow.

Request scope: The object is available from the time an HTTP request is made until a response is sent back to the client.

Backing bean scope: Used for managed beans for page fragments and declarative components only, the object is available from the time an HTTP request is made until a response is sent back to the client.

View scope: The object is available until the view ID for the current view activity changes. This scope can be used to hold values for a given page. However, unlike request scope, which can be used to store a value needed from one page to the next, anything stored in view scope will be lost once the view ID changes.

When the Fusion web application runs in a clustered environment, a portion of the application's state is serialized and copied to another server or a data store at the end of each request so that the state is available to other servers in the cluster.

When you are designing an application to run in a clustered environment, you must:

Ensure that all managed beans with a life span longer than one request are serializable (that is, they implement the java.io.Serializable interface). Specifically, beans stored in session scope, page flow scope, and view scope need to be serializable.

Ensure that Oracle ADF is aware of changes to managed beans stored in ADF scopes (view scope and page flow scope) and enable the tracking of changes to ADF memory scopes.

When a value within a managed bean in either view scope or page flow scope is modified, the application needs to notify Oracle ADF so that it can ensure the bean's new value is replicated.

Without additional code, Oracle ADF will be unaware of this change and will not know that a new value needs to be replicated within the cluster. To inform Oracle ADF that an object in an ADF scope has been modified and that replication is needed, use the markScopeDirty() method, as shown in Example 6-2. The markScopeDirty() method accepts only viewScope and pageFlowScope as parameters.

Example 6-2 Additional Code to Notify Oracle ADF of Changes to an Object

This code is needed for any request that modifies an existing object in one of the ADF scopes. If the scope itself is modified by the scope's put(), remove(), or clear() methods, it is not necessary to notify Oracle ADF.

To enable ADF Controller to track changes to ADF memory scopes and replicate the page flow scope and view scope within the server cluster, you can enable the<adf-scope-ha-support> parameter in the adf-config.xml file, as described in Section 6.1.3.3, "Configuring adf-config.xml."

If you are using Oracle HTTP Server, the server is configured with the WebLogicCluster directive to balance among all available application instances.

If you are using a hardware load balancer, the load balancer is:

Routing traffic to all available instances

Configured correctly with a health monitor to mark unavailable instances

Configured to support persistence of session state

Expected Behavior for Application Failover

If the environment has been configured correctly, application users do not notice when an application instance in a cluster becomes unavailable. The sequence of events in an application failover is, for example, as follows:

A user makes a request and is routed by a hardware load balancer to instance A of the application.

Instance A of the application becomes unavailable because of node failure, process failure, or network failure.

The hardware load balancer marks instance A as unavailable.

The user makes a subsequent request. The request is routed to instance B.

Instance B is configured as a replication partner of Instance A and has the user's session state.

The application resumes using the session state on Instance B and the user continues working without interruption.

6.1.2.3 Oracle ADF Active Data Services

The Fusion technology stack includes the Active Data Service (ADS), which allows you to bind ADF Faces components to an active data source using the ADF Model layer. In JDeveloper, you configure individual components in your JSF pages to display active data. When you configure components to use active data, data is pushed to the client whenever a change event is raised by the data source. The data is pushed from the server to the client and displayed by the browser.

To support failover for pages configured to display active data, it is necessary to enable failover for the ADF Business Components application module and to enable ADF Controller to track changes to the ADF memory scopes. With these ADF settings configured, when failover occurs, the ADF server will detect the failover and request all pages configured to display active data to refresh themselves, after that ADS restarts and data is pushed to the client.

6.1.2.4 Configuring the ADF Application Module for Oracle RAC

When configuring the ADF application module to access a highly available database system, such as redundant databases or Oracle Real Application Clusters (Oracle RAC) as the backend, the data source must be container-defined. In this scenario, it is required to use a multi data source; however, from the standpoint of the application module configuration, the naming convention for the multi data source is the same as it is an non-multi data source. This ensures that the correct data source will be used at runtime. For details about configuring multi data sources for high availability applications, see Section 4.1.3, "Configuring Multi Data Sources with Oracle RAC.".

6.1.3 Configuring Oracle ADF for High Availability

To support automatic replication and failover for web applications within a clustered environment, Oracle WebLogic Server supports mechanisms for replicating HTTP session state across clusters. You can configure Oracle ADF to ensure the Fusion web application's state can be restored from any server in the cluster.

6.1.3.1 Configuring Application Modules

An application module is the transactional component that UI clients use to work with application data. It defines an updateable data model and top-level procedures and functions (called service methods) related to a logical unit of work related to an end-user task. An application module supports passivating, or storing, its transaction state as a snapshot in the database. It also supports the reverse operation of activating the transaction state from one of these saved snapshots.

To enable support for ADF Business Components failover, set the jbo.dofailover parameter to true so that the application module state is saved on release. This allows Oracle ADF to restore the application module state from a snapshot saved from a previous checkin. By contrast, when the failover feature is disabled, which it is by default, then application module state is saved only when the application is reused by a subsequent user session and only when the application module pool cannot find an unused application module.

You can set this parameter in your application module configuration on the Pooling and Scalability tab of the Edit Business Components Configuration dialog.

To configure application modules for high availability:

Launch JDeveloper and open the application.

In the Application Navigator, expand the project that contains the data model and locate the application module.

6.1.3.2 Configuring weblogic.xml

To enable support for replicating HTTP session state, you must assign a value to the persistent-store-type element in the Oracle WebLogic Server weblogic.xml file. The value replicated_if_clustered ensures that the in-effect persistent store type will be replicated so that sessions on the clustered environment are stored in accordance with the value set for the cluster of servers to which this server belongs.

To configure the weblogic.xml file for high availability:

Launch JDeveloper and open the application.

In the Application Navigator, expand the project that contains the web application and expand the WEB-INF folder.

Double-click the weblogic.xml file, and click the Source tab to edit the file.

In the file, add the persistent-store-type definition to the session-descriptor element:

6.1.3.3 Configuring adf-config.xml

When you are designing an application to run in a clustered environment, you must ensure that Oracle ADF is aware of changes to managed beans stored in ADF scopes (view scope and page flow scope).

When a value within a managed bean in either view scope or page flow scope is modified, the application needs to notify Oracle ADF so that it can ensure the bean's new value is replicated.

To enable ADF Controller to track changes to ADF memory scopes and replicate the page flow scope and view scope within the server cluster, you must set the ADF Controller parameter <adf-scope-ha-support> in the application's adf-config.xml file to true. For example, when set to true for an application and that application adds or removes a bean from a page flow scope during a request, the change will automatically replicated within a cluster.

The adf-config.xml file is the central configuration file for all ADF components. The file contains sections to configure the runtime behavior for ADF components, including, ADF Controller.

To configure the adf-config.xml file for high availability:

Launch JDeveloper and open the application.

In the Application Navigator, expand the Application Resources.

Select Descriptors, and then select the ADF META-INF node.

Double-click the adf-config.xml file, and click the Source tab to edit the file.

The org.apache.myfaces.trinidad.CHECK_FILE_MODIFICATION parameter must not be set to true when running in a high availability environment. Setting this context parameter to true can lead to errors after failover occurs.

6.1.4 Troubleshooting Oracle ADF High Availability

6.1.4.1 Troubleshooting Oracle ADF Development Issues

When you develop the Fusion web application in Oracle JDeveloper, the integrated development environment provides support for detecting potential High Availability issues. The warnings that JDeveloper provides are generated by the audit framework and will be triggered to display in the JDeveloper source editors. The warnings the editors display are based on the audit rules for High Availability applications.

The High Availability audit rules that JDeveloper enables by default are:

ADF Controller Configuration - High Availability for ADF Scopes is not Enabled simply warns the developer that the adf-scope-ha-support flag in the adf-config.xml file is set is not set to true. This audit rule fires only when the <adf-controller-config> element is present the ADF application-level configuration file (adf-config.xml).

ADF Page Flows - Bean in Scope Map is Modified warns the developer when the some code calls a setter method on a bean to indicate that the code did not subsequently call the ControllerContext.markScopeDirty() method. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

ADF Page Flows - EL Bean is Modified warns the developer when some code evaluates an EL expression that mutates a bean to indicate that the code did not subsequently call the ControllerContext.markScopeDirty() method. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

ADF Page Flows - Managed Bean Class Not Serializable warns the developer that a managed bean has a non-serializable class defined in viewScope, pageFlowScope, or sessionScope. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

You can modify the High Availability audit rule settings using the Preference dialog in JDeveloper. From the JDeveloper toolbar, choose Tools - Preferences, under Audit - Profiles expand ADF Controller Configuration or ADF Pages Flows and make the desired audit rule selections.

You can also trigger the audit by choosing Build - Auditproject.jpr from the JDeveloper toolbar.

6.1.4.2 Troubleshooting Oracle ADF Deployment Issues

Fusion web applications are deployed when the managed server is first started. Use the Oracle WebLogic Server Administration Console first to check that all application deployments were successful:

Click Deployments in the left hand pane. The right hand pane shows the application deployments and their status. The state of all applications, assuming all the servers are running, should be ACTIVE.

If an application deployment has failed, the server logs may provide some indication of why the application was not deployed successfully. The server logs are located in the DOMAIN_HOME/servers/server_name/logs directory. Common issues include:

Unavailability of external resources, such as database resources. Examine the error, fix it, and attempt to redeploy the application.

The appropriate applications or libraries are not targeted correctly to the right managed server or Cluster.

6.1.4.3 Troubleshooting Oracle ADF Replication and Failover Issues

State Replication is most prominent in failover scenarios. A user working on one server may discover that, upon failover:

Windows may be closed or state might be reset.

Screens may require a reset.

The application may be redirecting to the logon screen.

The following steps provide guidance in troubleshooting and diagnosing state replication issues.

Replication occurs within the context of a cluster. For failover to be successful, there must be at least one other healthy member of the cluster available. You can check cluster status in one of two ways:

Check the cluster status using the Oracle WebLogic Server Administration Console - In the Left-hand pane, click on Servers. Verify the state of all servers in the cluster.

Check the cluster status using weblogic.Admin utility The weblogic.Admin command can be used to query the state of all servers in a specific cluster. For example:

There are 2 server(s) in cluster: Spaces_Cluster
The alive servers and their respective states are listed below:
Application Server---RUNNING
Managed Server---RUNNING

Check cluster communications.

Although Cluster members may all be running, there may be communication issues which prevent them from communicating replication information to each other. There are two types of cluster communication configurations. Troubleshooting depends on the cluster type:

Checking Unicast cluster communications - For Unicast clusters, managed servers must be able to access each other's hosts and each other's default listening port.

Ensure that all individual managed servers have their Listen Address set correctly. You can find this setting by selecting Configuration, General for each managed server.

Checking Multicast cluster communications - For multicast clusters, servers must be able to intercept the same multicast traffic. Ensure that multicast is configured correctly by running the WebLogic utility utils.MulticastTest on each machine. For example:

$ java utils.MulticastTest -H

Confirm Oracle WebLogic Server application configuration.

Oracle WebLogic Server is not configured by default for failover. In-memory replication takes place only with the proper setting in the weblogic.xml file:

Set an appropriate value for the weblogic-application.xml deployment descriptor parameter inactive-connection-timeout-seconds on the element <connection-check-params> pool-params.

When enabling application module state passivation, a failure can occur when Oracle WebLogic Server is configured to forcibly release connections back into the pool. The failure creates an exception "Connection has already been closed" that gets saved to the server log. The user interface does not show this exception.

Set inactive-connection-timeout-seconds to several minutes. In most cases, this setting avoids forcing the inactive connection timeout and passivation failure. Adjust the setting as needed for your environment.

Confirm Oracle ADF Controller configuration.

Oracle ADF is not configured by default to replicate changes to ADF objects in ADF memory scopes. ADF object replication is supported only with the proper setting in the ADF application-level configuration file (adf-config.xml):

By default the ADF log shows high-level messages (INFO level). The default logging often reports problems with serialization and replication without the need to enable more detailed log messages. For more information about the log, see Section 6.1.1.4, "Oracle ADF Log File."

Enable log messages for ADF high availability applications.

Configure the ADF logger to output runtime messages for high availability. By default the ADF log display high-level messages (INFO level). You enable high availability diagnostics for ADF Controller by setting the logging level in Fusion Middleware Control to FINE.

When enabled, the logger outputs a warning if the adfc:adf-scope-ha-support setting in the adf-config.xml file is not set. For more information about the ADF logger, see Section 6.1.1.4, "Oracle ADF Log File."

Enable debug.

Check the server logs for any unusual messages on managed server startup. In particular, if the managed server is unable to locate other members of the cluster. The server logs are located in the DOMAIN_HOME/servers/SERVER_NAME/logs directory.

For further debugging, enable the flags DebugCluster, DebugClusterAnnouncements, DebugFailOver, DebugReplication, and DebugReplicationDetails. Each flag can be enabled with the weblogic.Admin utility:

Enable server checking to ensure no unserializable state content on session attributes is detected. This check is disabled by default to reduce runtime overhead. Serialization checking is supported by the Java server system property org.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION.

Table 6-1 shows the options that can be used with the property. Use commas to delimit the options.

Table 6-1 CHECK_STATE_SERIALIZATION Options

Option

Description

tree

Checks whether the entire component tree is serializable. This is the fastest component check. Most testing should be performed with this flag enabled.

component

Checks each component individually for serializability. This option is much slower than "tree." It is typically turned on only after testing with "tree" has reported an error. This option is used to narrow down the problematic component.

property

Checks each component attribute individually for serializability. This is slower than "component" and is used to narrow the down the specific problematic component attribute after a failure has been detected in the "tree" or "component" modes.

session

Checks that all attributes in the JSF Session Map that are marked as Serializable are in fact serializable.

application

Checks that all attributes in the JSF Application Map that are marked as Serializable are in fact serializable.

beans

Checks that any serializable object in the appropriate map has been marked as dirty if the serializable content of the object has changed during the request.

all

Checks everything.

For high availability testing, start off by validating that the Session and JSF state is serializable by launching the application server with the system property:

-Dorg.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION=session,tree

Add the beans option to check that any serializable object in the appropriate map has been marked as dirty if the serialized content of the object has changed during the request:

ORACLE_HOME: This environment variable and related directory path refers to the location where Oracle Fusion Middleware SOA Suite is installed.

ORACLE_COMMON_HOME: This environment variable and related directory path refers to the Oracle home that contains the binary and library files required for the Oracle Enterprise Manager Fusion Middleware Control and Java Required Files (JRF).

DOMAIN directory: This directory path refers to the location where the Oracle WebLogic domain information (configuration artifacts) is stored.

ORACLE_INSTANCE: An Oracle instance contains one or more system components, such as Oracle Web Cache, Oracle HTTP Server, or Oracle Internet Directory. An Oracle instance directory contains updateable files, such as configuration files, log files, and temporary files.

The values used and recommended for consistency for this directories are:

ORACLE_BASE: /u01/app/oracle

MW_HOME (application tier): ORACLE_BASE/product/fmw

WL_HOME: MW_HOME/wlserver_10.3

ORACLE_HOME: MW_HOME/adf

6.2.2 Using RCU to Load Fusion Middleware Schemas in the Database

This step is required only if your ADF application needs to use any of schemas that are part of Oracle Fusion Middleware. Typically, this is done if the ADF application uses MDS repository, in which case you must install the 11g (11.1.1) Oracle Fusion Middleware Metadata Store into a Real Application Clusters (Oracle RAC) database before you install Oracle Fusion Middleware. Oracle Fusion Middleware provides a tool, the Oracle Fusion Middleware Repository Creation Utility (RCU), to create the component schemas in an existing database.

Use the latest version of RCU to install the 11g (11.1.1) Oracle Fusion Middleware Repository into a Real Application Clusters database.

Host Name: Enter the name of the node that is running the database. For an Oracle RAC database, specify the VIP name, or one of the node names as the host name: ADFDBHOST1VIRTUAL.

Port: The port number for the database: 1521

Service Name: Enter the service name of the database: adfha.mycompany.com

Username: SYS

Password: Enter the password of the SYS user.

Role: SYSDBA

Click Next.

If you receive the following message, click Ignore or Stop:

The database you are connecting is with non-UTF8 charset, if you are going to use this database for multilingual support, you may have data loss. If you are not using for multilingual support you can continue, otherwise, Oracle strongly recommends using a UTF-8 database.

In the Select Components screen, select Create a New Prefix, and enter a prefix to use for the database schemas, for example, ADFHA.

Write down the schema names so they are available in later procedures.

Select the following schemas:

AS Common Schemas

Metadata Services

Click Next.

In the Schema Passwords screen, enter passwords for the main and additional (auxiliary) schema users, and click Next.

In the Map Tablespaces screen, choose the tablespaces for the selected components, and click Next.

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct, and that you have write permissions for that directory.

To install Oracle WebLogic Server on all nodes in the application tier:

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct and that you have write permissions for that directory.

In the Configure Administrator Username and Password screen, enter the username and password to be used for the domain's administrator, and click Next.

In the Configure Server Start Mode and JDK screen, make the following selections:

WebLogic Domain Startup Mode: select Production Mode

JDK Selection: select Oracle JRockit 1.6.0_14 SDK

Click Next.

In the Select Optional Configuration screen, select the following:

Administration Server

Managed Servers, Clusters and Machines

Click Next.

In the Customize Server and Cluster Configuration screen, select Yes, and click Next.

In the Configure the Administration Server screen, enter the following values:

Name: AdminServer

Listen Address: APPHOST1

Listen Port: 7001

SSL listen port: NA

SSL enabled: Leave unchecked

Click Next.

In the Configure Managed Servers screen, add the following managed servers:

Managed Server Name

Listen Address

Listen Port

SSL Listen Port

SSL Enabled

WLS_ADF1

Hostname of APPHOST1

8889

NA

unchecked

WLS_ADF2

Hostname of APPHOST2

8889

NA

unchecked

Click Next.

In the Configure Clusters screen, add the following cluster:

Name: ADF_CLUSTER

Cluster Messaging Mode: unicast

Cluster Address Enabled: Leave blank

Click Next.

In the Assign Servers to Clusters screen, assign the following servers to the Cluster:

ADF_CLUSTER:

WLS_ADF1

WLS_ADF2

Click Next.

In the Configure Machines screen:

Delete the LocalMachine that appears by default.

Click the Unix Machine tab, and add the following machines:

Name

Node Manager Listen Address

APPHOST1

Hostname of APPHOST1

APPHOST2

Hostname of APPHOST2

Click Next.

In the Assign Servers to Machines screen, assign servers to machines as follows:

APPHOST1: AdminServer, WLS_ADF1

APPHOST2: WLS_ADF2

In the Configuration Summary screen, click Create.

In the Creating Domain screen, click Done.

6.2.6.1 Creating boot.properties for the Administration Server and Managed Servers on APPHOST1

This is an optional step for enabling the Administration Server to start without prompting you for the administrator username and password. Create a boot.properties file for the Administration Server and for the managed servers on APPHOST1.

6.2.7.2 Validating the Administration Server

To verify that the Administration Server is properly configured, follow these steps:

In a Web browser, go to http://VIP1:7001/console.

Log in as the administrator.

Verify that the WLS_ADF1 and WLS_ADF2 managed servers are listed.

Verify that the ADF_Cluster cluster is listed.

Verify that you can access Enterprise Manager at http://VIP1:7001/em.

6.2.7.3 Disabling Host Name Verification for the Administration Server and Managed Servers for APPHOST1 and APPHOST2

This step is required if you have not set up SSL communication between the Administration Server and the Node Manager. If SSL is not set up, you receive an error message unless you disable host name verification.

You can re-enable host name verification when you have set up SSL communication between the Administration Server and the Node Manager.

6.2.7.4 Starting Node Manager on APPHOST1

Run the setNMProps.sh script, which is located in the ORACLE_COMMON_HOME/common/bin directory, to set the StartScriptEnabled property to true before starting Node Manager:

OAHOST1> cd ORACLE_COMMON_HOME/common/bin
APPHOST1> ./setNMProps.sh

Note:

You must use the StartScriptEnabled property to avoid class loading failures and other problems.

Start Node Manager:

APPHOST1> cd WL_HOME/server/bin
APPHOST1> ./startNodeManager.sh

6.2.8 Installing Oracle WebLogic Server and Oracle ADF on APPHOST2

Repeat the procedures for installing WebLogic Server and Oracle ADF for APPHOST2, start with Section 6.2.3, "Installing Oracle HTTP Server on WEBHOST1." The directory paths for binary files and domains used when installing new nodes must be exactly the same as those used for the first node. If these paths and domains are not exactly the same as those used for the first node, failover does not occur.

6.2.9 Propagating the Domain Configuration to APPHOST2 with pack/unpack Utilities

Follow these steps to propagate the domain configuration to APPHOST2 using pack/unpack utilities:

6.2.9.2 Starting Node Manager on APPHOST2

6.2.9.3 Configuring the ADF Application for Replication

Use the procedures in this section to configure your application for replication.

Clustering Requirement

The application must be deployed to an Oracle WebLogic Cluster. This automatically establishes a replication channel for the multiple instances of the application.

Note:

In a Unicast cluster, the default replication channel is configured using the Listen address of each managed server. Therefore, the Listen address should be configured to be a specific IP address or host name, instead of being configured to listen on Any.

Oracle ADF Replication

It is essential that Oracle ADF is configured properly. The following tag should be present in the adf-config.xml file, one of the Application Resources, for a stateful application:

Enable Oracle HTTP Server to route to the Administration Server that contains Oracle WebCenter Managed Servers by setting the WebLogicCluster parameter to the list of nodes in the cluster. Follow these steps:

Add the following lines to the OHS_HOME/instances/ohs_instance1/config/OHS/ohs1/mod_wl_ohs.conf file:

6.2.10 Scaling the Topology

You can scale out and scale up an Oracle ADF topology. When you "scale up" the topology, you add new managed servers to nodes that are already running one or more managed servers. When you "scale out" the topology, you add new managed servers to new nodes.

In this case, you already have a node that runs a managed server configured with Oracle ADF. The node contains a Middleware home in shared storage.

You can use the existing installations (Middleware home, and domain directories) for creating new servers. There is no need to install Oracle Fusion Middleware binaries in a new location, or to run pack and unpack.

Follow these steps for scaling up the topology:

Using the Oracle WebLogic Server Administration Console, clone WLS_ADF1 into a new managed server. The source managed server to clone should be one that already exists on the node where you want to run the new managed server.

To clone a managed server:

Select Environment > Servers from the Administration Console.

Select the managed server that you want to clone.

Select Clone.

Name the new managed server WLS_ADFn, where n is a number to identify the new managed server.

For the listen address, assign the host name or IP to use for this new managed server.

Ensure that the port number for this managed server is available on this node.

ADF managed servers are clustered and the new managed server will also be part of that cluster.

The new node can access the existing home directories for WebLogic Server. You use the existing installations in shared storage for creating a new managed server. There is no need to install WebLogic Server binaries in a new location, although you must run pack and unpack to create a managed server domain.

Follow these steps for scaling out the topology:

On the new node, mount the existing Middleware home, and ensure that the new node has access to this directory, just like the rest of the nodes in the domain.

To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.

Log into the Oracle WebLogic Administration Console.

Create a new machine for the new node that will be used, and add the machine to the domain.

Update the machine's Node Manager's address to map the IP of the node that is being used for scale out.

Use the Oracle WebLogic Server Administration Console to clone WLS_ADF1 into a new managed server. Name it WLS_ADFn, where n is a number and assign it to the new machine.

For the listen address, assign the host name or IP to use for the new managed server.

Perform these steps to set the managed server listen address:

Log into the Oracle WebLogic Server Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers. The Summary of Servers page is displayed.

Select the managed server whose listen address you want to update in the Names column of the table. The Setting page for that managed server is displayed.

Set the Listen Address to APPHOSTn where APPHOSTn is the DNS name of your new machine.

Click Save.

Save and activate the changes.

The changes will not take effect until the managed server is restarted.

Run the pack command on APPHOST1 to create a template pack as follows:

Start the Node Manager on the new node. To start the Node Manager, use the installation in shared storage from the existing nodes, and start Node Manager by passing the host name of the new node as a parameter as follows:

APPHOSTn> WL_HOME/server/bin/startNodeManager <new_node_ip>

Start and test the new managed server from the Oracle WebLogic Server Administration Console:

Shut down all the existing managed servers in the cluster.

Ensure that the newly created managed server is running.

Access the application on the newly created managed server. The application should be functional.

6.3 Oracle WebCenter and High Availability Concepts

The information in this section guides you through the issues and considerations necessary for designing an Oracle WebCenter high availability cluster.

6.3.1 Understanding Oracle WebCenter

Oracle WebCenter combines the standards-based, declarative development of Java Server Faces (JSF), the flexibility and power of portals, and a set of integrated Web 2.0 services.

6.3.1.1 Oracle WebCenter Components

Oracle WebCenter includes the following components.

Oracle WebCenter Spaces offers a single, integrated, Web-based environment for social networking, communication, collaboration, and personal productivity through a robust set of services and applications.

A browser-based, community-focused application targeting the business user.

A personal space for each user, providing a private work area for storing personal content, keeping notes, viewing and responding to business process assignments, maintaining a list of online buddies, emailing, and so on. The focus of a personal space is personal productivity.

Oracle WebCenter Analytics provides users with the ability to view reports on the various user activities within an instance of Oracle WebCenter Spaces, for example:

Logins

Page views

Portlet views

Document views

Search metrics

Response times

These reports can be broken out by parameters such as User properties, GroupSpaces, and time.

Oracle WebCenter Activity Graph provides users with the ability to analyze the statistics collected by Oracle WebCenter Analytics. The output of an Oracle WebCenter Activity Graph analysis is the collected scores for objects and users, which are used to give recommendations. The scores are stored in the Oracle WebCenter Activities database.

Oracle WebCenter Personalization Server is a lightweight service accessed by client applications via RESTful web services. New JDev tooling supports creation of the property definitions and scenarios used by Oracle WebCenter Personalization. Out-of-the-box integration with other Oracle WebCenter services (Oracle Activity Graph, CMIS, People Connections) is supported as well as an extensibility model for customers. Oracle WebCenter Personalization Server provides users with the ability to define user and applications property definitions and values and implement structured scenarios. These can be used to personalize any kind of application or application logic for a user, group, or other scope.

Oracle WebCenter Spaces and WebCenter Portal applications can also integrate with the following Oracle WebCenter services:

Announcements - Provides a means of posting announcements about important activities and events to members of a given group space.

Discussions - Provides a means of creating threaded discussions, posting and responding to questions, and searching for answers-all within the context of a group space. Also provides an effective group communication mechanism for important activities and events.

Events - Available in Oracle WebCenter Spaces, the Events service provides a means of creating and maintaining a schedule of events relevant to a wider group of users. Events are published to all members of a group space.

Instant Messaging and Presence (IMP) - Provides a means of observing the status of other authenticated users (whether online, offline, busy, or idle) and contacting them instantly

Links - Provides viewing, accessing, and associating related information; for example you can link to a solution document from a discussion thread.

Lists - Available in Oracle WebCenter Spaces, the Lists service provides the ability to create, publish, and manage lists and tables. Users can create lists from prebuilt structures or create their own custom lists.

Notes - Available in Oracle WebCenter Spaces, the Notes service provides the ability to "jot down" and retain quick bits of personally relevant information.

People Connections - Provides a means of connecting, interacting, and keeping track of other users through social networking applications, such as Message Board, Feedback, Profile, and Activity Stream.

6.3.1.3 Oracle WebCenter State and Configuration Persistence

Oracle WebCenter Spaces runs as a Java EE application. Application state and configuration persist to the MDS repository. Local memory holds user session information. In a cluster environment, this state replicates to other cluster members.

All out-of-the-box Oracle WebCenter Portlets use the database to persist customizations. Oracle WebCenter Discussion Server uses its own database persistence mechanisms for data and metadata.

Oracle Webcenter Analytics has two components: the Oracle Webcenter Analytics service, which consists of task flows and data controls, and the Oracle WebCenter Analytics Collector Service.

Events sent to the Oracle WebCenter Collector are placed into a configurable queue of incoming events for persisting to the database. If an Oracle WebCenter Collector Service fails, these events are lost. Typically, however, the queue doesn't contain many events. The Oracle WebCenter Analytics Collector supports clustering and failover, but collectors do not share the queue.

The Oracle WebCenter Analytics service task flows, which provide the reporting UI for WebCenter are stateless. These features are built on the standard Composer/ADF/MDS framework, which manages the state if a failure occurs.

The engine runs a batch data analysis process to update database tables. It does not support clustering or failover but can recover from failure. The Oracle WebCenter Activity Graph service does not maintain any in-memory state; it is primarily a read-only process.

The Oracle WebCenter Spaces task flows query the Oracle WebCenter Activities database and produce a list of recommendations. State updates in these areas:

Personalization settings

Task flow configuration parameters

"Not-interested" feature

The first two areas are built on the standard Composer/ADF/MDS framework, which manages the state. With the "not interested" feature, you can indicate that you are not interested in a recommendation. This input persists synchronously in the Oracle WebCenter Activity Graph database.

Use the Oracle WebCenter Activity Graph user interface to set up and monitor the nightly schedule. The schedule persists to the database. If the managed server fails, the job continues when the server restarts. Some recommendations may be stale if you do not run the backend daily.

Oracle WebCenter Activity Graph uses JDBC to mine statistical data in the Activities schema, and the client (Oracle WebCenter Spaces) also uses JDBC to connect to the Activities schema to present the data to users.

Oracle WebCenter Spaces uses the Java Client API that is based on Open Usage, and which uses multicasting over UDP, to communicate with Oracle WebCenter Analytics.

Table 6-2 shows the access type for each of the Oracle WebCenter components and services. The Configuration column lists the type of information provided to Oracle WebCenter to configure or initialize the connection. The Access column lists the protocol used in runtime access of the service.

The Oracle Discussion Server is a service provider to Oracle WebCenter Spaces. Oracle Portlets also exposes Portlet Producers as services.

Unavailability of these services does not prevent Oracle WebCenter Spaces application from starting, although application errors may be seen when running. Oracle WebCenter Spaces requires the MDS and WebCenter schemas.

Table 6-2 Oracle WebCenter Access Types

External Server/ Service

Configuration

Access

Oracle Discussion server

HTTP access to discussions server administration

SOAP/HTTP

Oracle Content Server (Documents)

Socket connection to the Administration Server. HTTP access is required only if the content server must be accessed outside Oracle WebCenter.

JCR 1.0 over socket or HTTP

Instant Messaging and Presence server

HTTP access to instant messaging and presence server administration

SOAP/HTTP

Mail server

IMAP/SMTP server

IMAP/SMTP

Personal events server

HTTP access to calendar services

SOAP/HTTP

Portlets

HTTP location of provider WSDLs

SOAP/HTTP

Search server

HTTP access to search server

HTTP

Worklist

HTTP access to BPEL server

SOAP/HTTP

MDS repository and schemas

JDBC

JDBC

Oracle WebCenter Analytics

UDP access to Oracle WebCenter Analytics Collectors

UDP

Oracle WebCenter Activity Graph

HTTP access to Oracle WebCenter Activity Graph

HTTP

Oracle WebCenter Personalization Server

JDBC access to Oracle WebCenter Personalization Server

JDBC, REST

Configure each of the external services independently for high availability. Oracle WebCenter's framework provides a single point of access for external services.

For HTTP services, for example, direct the access URL to a load balancer which provides access to multiple service providers on the back-end.

6.3.1.5 Oracle WebCenter Configuration Considerations

The main configuration files for Oracle WebCenter applications are listed and described in Table 6-3. Both these files are supplied within the Oracle WebCenter application deployment .EAR file.

Table 6-3 Oracle WebCenter Configuration Files

Artifact

Purpose

adf-config.xml

Stores basic configuration for Oracle Application Development Framework (ADF) and Oracle WebCenter application settings, such as which discussions server or mail server the WebCenter application is currently using.

connections.xml

Stores basic configuration for connections to external services.

Oracle WebCenter applications and Portlet Producers both use the Oracle Metadata Services (MDS) repository to store their configuration data; both access the MDS repository as a JDBC data source within the Oracle WebLogic framework.

The MDS repository stores post deployment configuration changes for Oracle WebCenter applications and Portlet Producers as customizations. MDS uses the original deployed versions of adf-config.xml and connection.xml as base documents and stores all subsequent customizations separately into MDS using a single customization layer.

When an Oracle WebCenter application starts up, customizations stored in MDS are applied to the appropriate base documents and the Oracle WebCenter application uses the merged documents (base documents with customizations) as the final set of configuration properties.

For Oracle WebCenter applications that are deployed to a server cluster, all members of a cluster read from the same location in the MDS repository.

Typically, there is no need for administrators to examine or manually change the content of base documents (or MDS customization data) for files such as adf-config.xml and connection.xml as Oracle provides several administration tools for post deployment configuration. For details, see the section "Oracle WebCenter Administration Tools" in the Oracle Fusion Middleware Administrator's Guide for Oracle WebCenter.

Note:

Oracle does not recommend that you edit adf-config.xml or connection.xml by hand (unless specifically instructed to do so) as this can lead to misconfiguration.

Oracle WebCenter Analytics uses WLST commands for the clients. JVM parameters can also be specified and will override the WLST values. If you specified JVM parameters in setDomainEnv.sh in previous releases of Oracle WebCenter, you should remove them if you plan to use Oracle WebCenter Analytics.

The Oracle WebCenter Activity Graph engine writes temporary files to disk, which are recreated if they are lost.

Oracle WebCenter Personalization Server has JMX beans for its provider connections, which get bootstrapped from provider-connections.xml in the classpath. WLST command scripts support configuration of Oracle WebCenter Personalization provider connections via JMX. These beans can also be managed via Oracle Enterprise Manager or JConsole.

6.3.1.6 Oracle WebCenter Log File Locations

Operations performed by Oracle WebCenter applications, Portlet Producers, and discussion servers, and so on, are logged directly to the WebLogic Managed Server where the application is running:

WLS_DOMAIN_HOME/servers/WLS_SERVER_NAME/logs/WLS_SERVER_NAME.log

You can view the log files for each WebLogic managed server from the Oracle WebLogic Server Administration Console. To view the logs, access the Oracle WebLogic Server Administration Console http://<admin_server_host>:<port>/console and click Diagnostics-Log Files.

Oracle WebCenter Analytics integrates with the current WebLogic Server logging framework by using Apache Commons logging in the Collector and ODL configuration files in the domain template used to install the Collector. The ODL logging configuration file is merged with the WebLogic Server default logging configuration file when the Collector is installed using the Configuration Wizard. When the ODL logging configuration file is merged with the WebLogic Server default logging configuration file, if you open the logging.xml file generated in the DOMAIN_HOME/config/fmwconfig/servers/WLS_Utilities folder, you will find logging settings for a log handler named collector-log-handler.

In addition, a diagnostic log is produced in the log directory for the managed server. This log's granularity and logging properties can be changed through the DOMAIN_HOME/config/fmwconfig/servers/SERVER_NAME/logging.xml file.

An Oracle WebLogic cluster provides high availability for applications. When one member of the cluster is unavailable, another member of the cluster handles the request.

Each of the managed servers in an Oracle WebCenter deployment can be deployed as a cluster, with different cluster members either on the same node, or on different nodes. In Figure 6-5, all requests sent to the cluster can be served equally by either member of the cluster.

The startup order is also the order of dependency. If a dependent component does not deploy successfully, a later component may not function correctly. Oracle WebCenter application startup is not dependent of the availability of external services such as the Oracle Discussions server, or other back-end servers.

6.3.2.3 Deploying Oracle WebCenter Application on a Cluster

For an Oracle WebCenter cluster deployment, such as the one shown in Figure 6-5, follow these rules for the targeting of applications, libraries, and system resources:

Target applications and libraries to the cluster target. For example, target the WebCenter application to the Spaces cluster.

JDBC resources to the cluster target.

Oracle WebCenter applications and the Oracle WebCenter Discussions server are deployed as Oracle WebLogic stage applications. During the initial deployment of each application, the deployment files are received from the Administration Server and deployed locally.

6.3.2.4 Oracle WebCenter Analytics Communications

This section describes how an Oracle Analytics Collector cluster works.

In a clustered environment, producers (clients of the collector) and collectors are configured with a cluster-specific channel name. Each collector periodically broadcasts a heartbeat with its location to the cluster-specific channel. The producer listens to the channel for these collector heartbeats, and when it hears one, adds the collector to its list of known collectors. When the producer needs to send an event, it uses a round robin algorithm to select a collector from its list and sends the event to that collector. If a collector stops (either being stopped purposefully or failing), it stops broadcasting a heartbeat. When the producer stops hearing the heartbeat it removes the collector from its list and stops sending events to that collector. If the producer does not hear any collector heartbeats, it does not send any events.

Oracle WebCenter Analytics uses UDP and multicast on a configured set of ports to communicate with its clients. For a single node setup, the client is configured with a WLST command that has the server host/port location and simply transmits all events to this location via UDP. For a multiple node setup, the server is configured to broadcast (via UDP multicast) the location(s) of the various servers running on the cluster; the client is configured with the same WLST command, so it will receive the locations of these servers and keep the list of available servers in a list (which is persisted in memory.) Additionally, if a client does not receive a periodic heartbeat from a server to indicate it is functioning, it will remove that server from its list of known servers, and will cease sending events to it.

Oracle recommends that Collectors be configured as unicast, forming a 1-1 relationship with the Oracle WebCenter Analytics client.

6.3.2.5 Oracle WebCenter State Replication

Oracle WebCenter relies on Oracle ADF, which has several stateful components. Oracle WebCenter itself is also a stateful application. Therefore, you must configure state replication in cluster scenarios.

In-memory replication - Using in-memory replication, Oracle WebLogic Server copies a session state from one server instance to another. The primary server creates a primary session state on the server to which the client first connects, and a secondary replica on another WebLogic Server instance in the cluster. The replica is kept up to date so that it may be used if the server that hosts the servlet fails.

JDBC-based persistence - In JDBC-based persistence, Oracle WebLogic Server maintains the HTTP session state of a servlet or JSP using file-based or database-based persistence.

6.3.2.6 Understanding the Distributed Java Object Cache

Oracle WebCenter applications use a distributed Java Object Cache for greater performance. Configure this cache across all Oracle WebCenter clusters, with one distributed cache per cluster.

Table 6-6 lists some examples of the type of objects which Oracle WebCenter places in the Java Object Cache.

Table 6-6 Oracle WebCenter Object Types for Java Object Cache

Oracle WebCenter Component

Object Cached

Discussions service

Topics and forums

Announcements service

Announcements

Instant Messaging and Presence service

Presence subscription lists

User's presence/subscription status

Worklist service

Called Worklist items, such that cached data is used until refresh is called.

Content Integration (Oracle Portal)

JCR: type information and metadata obtained from the repository.

Service Framework

User profile. Queried usernames.

Recent Activity

Recent activity results per user.

WebCenter Spaces

Global list of group spaces and group space templates in the application.

List of group spaces and group space templates that a user can access.

Page Service

List of pages in a scope.

WSRP Server

Preference store values for WSRP producers.

Documents service

Provisioning and configuration checks for the Documents service configured for an Oracle WebCenter Spaces application

Profile management

Lightweight user profile objects

Navigation service

List of active Navigation Model objects

Portlet Consumer

Portlet markup

Collaboration

Collaboration services cache objects in the Java Object Cache on a per user session basis. These cached user sessions are destroyed when the HTTP session is destroyed.

Worklist

Worklist caches the called items so that unless the refresh button is clicked, or the every fifteen minute refresh poll is triggered, the same items are displayed as the user changes the sort and group by settings. The display is also cached by group and sort order settings, updating when a change occurs. This is read only data which, if not present, is fetched and stored on the cache.

Oracle WebCenter Spaces

The list of all templates and public spaces are maintained in the Java Object Cache. This is shared by all users, in addition to the list of group spaces and templates that a particular user can access. These are cached per user. A template created on one JVM does not show up if the template cache has been initialized in another JVM, unless JOC distributes the data, or an administrator in the second JVM does an explicit refresh where the cache is rebuilt.

Documents Service

The Documents service uses Java Object Cache to cache provisioning and configuration checks for document services, as applies to Oracle WebCenter Spaces application configuration. For provisioning, cached objects are flagged as distributed. They are replicated by a correctly configured Java Object Cache in a high availability environment, the configuration cached state is kept locally. All cached objects are flagged to expire after one minute. Caching reduces the number of times UCM calls are made to check the state of the UCM Server, as Oracle WebCenter Spaces repeatedly checks provisioning and configuration to control service rendering in the UI.

Portlet Consumer

The portlet consumer caches portlet mark-up in the Java Object Cache according to the cache headers in the portlet response. The cache can be expires based or validation based.

Profile Management caches the lightweight user profile objects in Java Object Cache. If particular profile data is not found, it will queried from the backend and the cache would be populated. The number of objects stored have an upper bound of 1000 and are in the Java Object Cache for one hour.

Navigation caches the Navigation Model objects in the Java Object Cache on a per user session basis. These cached objects are destroyed when the HTTP session is destroyed.

Recent Activity

The list of recent activity results are cached for each user to prevent a requery of results each time the recent activity task flow or RSS feed is viewed. The cache is automatically refreshed every fifteen minutes, or can be manually refreshed using the refresh icon in the recent activity task flow.

If you are using Oracle HTTP Server, the server configuration is configured with the WebLogicCluster directive to balance among all available application instances.

If you are using a hardware load balancer, the load balancer is:

Routing traffic to all available instances

Configured correctly with a health monitor to mark unavailable instances

Configured to support persistence of session state

6.3.2.8 Expected Behavior for Application Failover

If the environment has been configured correctly, application users do not notice when an application instance in a cluster becomes unavailable. The sequence of events in an application failover is, for example, as follows:

A user makes a request and is routed by a hardware load balancer to instance A of the application.

Instance A of the application becomes unavailable because of node failure, process failure, or network failure.

The hardware load balancer marks instance A as unavailable.

The user makes a subsequent request. The request is routed to instance B.

Instance B is configured as a replication partner of Instance A and has the user's session state.

The application resumes using the session state on Instance B and the user continues working without interruption.

Exceptions to Expected Behavior

For Oracle WebCenter, the known exceptions are as follows:

Oracle ADF Pop-ups - Open pop-ups close on failover, affecting the following components which otherwise have no exceptions:

When failover occurs, you must repeat the action that leads to the pop-up to make it reappear. The specific ways in which this appears in WebCenter Spaces and suggested remedies are listed in Table 6-7.

Table 6-7 Oracle WebCenter Troubleshooting Scenarios

Action Before Failover

After Failover

Suggested Remedy

Go to any page and click Create Page.

Type in a name, select a theme, and click OK. When you select the theme, the page creation pop-up closes.

Repeat the operation.

Launch Manage Pages.

Perform any operation within the pop-up, except for closing the pop-up, for example, click Page Actions. When you perform any operation, the Manage Pages pop-up is closed.

Repeat the operation.

Launch Manage Pages, click Page Actions against a page, and then the Delete option in the menu.

Click OK on the confirmation pop-up. Clicking OK closes the confirmation pop-up and the Manage Pages pop-up. The result of the deletion (which may have gone through) is not visible among the tabs.

Relaunch the Manage Pages pop-up to see if the page is deleted. If not, try deleting once again.

Launch Personalize Applications pop-up. Perform any operation that sends a request, other than clicking OK, for example, expand the Applications node.

Perform any operation that sends a request, other than clicking OK, for example, expand the Applications node. When you perform any operation that sends a request to the server, the Personalize Applications pop-up is closed.

Repeat the operation

Launch Preferences.

Switch between the Preferences tabs (General, Accounts, Messaging, Search). When you switch between the Preference tabs, the Preferences pop-up is closed.

Repeat the operation

Launch Manage Favorites. Stop the server, perform any operation other than closing the pop-up. For example, expand a folder, click Edit favorite information.

Perform any operation other than closing the pop-up, for example, expand a folder, and click Edit favorite information. When you perform any operation, the Manage Favorites pop-up is closed.

Relaunch the Manage Favorites pop-up to see if the operation was successful. If not, retry the operation.

Choose to edit applications and create a new folder

An MDS exception opens.

Retry the operation

Work in Oracle Composer Customization Manager.

Oracle Composer loses the state associated with the Customization Manager

Relaunch Customization Manager.

Select a component while in Oracle Composer Source View

Oracle Composer loses track of the component.

Repeat the operation

Component Specific Issues - Other issues which are specific to different components in Oracle WebCenter are listed in Table 6-8.

Table 6-8 Oracle WebCenter Exceptions to Expected Behavior

Oracle WebCenter Component

Exceptions to Expected Behavior

Group Space Events

When changing certain fields (start or end date/hour/minute), the event form is closed when failover occurs.

Portlets

Failures are fully transparent.

Lists

Failures are fully transparent.

Links

Failures are fully transparent.

Search

In the midst of a long-running query. The results that come back are not guaranteed (note there is no "write" operation here), the user can just reissue the query.

Tagging

Failures are fully transparent.

Recent Activities

The open/close state of the tree nodes is not replicated. After after failover, the tree of results closes all of the nodes. The nodes need to be reopened.

Worklist

Failures are fully transparent.

Document Manager

When a user uploads a document and a document with the same name already exists, the user is taken to the Confirm new version screen. While on that screen, the file is stored in a temporary local location. If failover occurs at that moment, the uploaded file is lost and you must restart the upload process.

6.3.2.9 Monitoring Logging of Application Deployments

Use Oracle WebLogic Server Administration Console to check the status of the application deployments. You can also use Oracle WebLogic Server infrastructure and Enterprise Manager Fusion Middleware Control for starting stopping, and monitoring Oracle WebCenter processes.

6.3.2.10 Oracle WebCenter Cluster-wide Configuration Changes

For Oracle WebCenter Spaces and Portlet Producers, all configuration data is stored in the MDS repository and Portlet Producers. Additional cluster deployments automatically read the latest configuration upon application startup.

For Oracle Discussion Server, the configuration information should be moved over from an existing cluster server. This is done automatically by the pack/unpack utility of Oracle WebLogic Server. Oracle recommends this procedure.

6.3.2.11 Maintaining Configuration in a Clustered Environment

For Oracle WebCenter Spaces and Portlet Producers, all configuration data is stored in the MDS repository and Portlet Producers. Any changes made to the configuration of one server in the cluster are immediately visible to all other members.

Changes to Oracle WebCenter Discussion Server are not frequent, however, when they do occur, the changes must be reapplied to other members of the cluster. You can do this by connecting directly to each discussions server, instead of using the load balancer, and making the necessary administration changes.

6.4 Configuring High Availability for Oracle WebCenter

Figure 6-5 illustrates a two-node Oracle WebCenter cluster running on two Oracle WebLogic Servers in one WebLogic Server domain. Oracle WebLogic Servers are front-ended by Oracle HTTP Server which load balances incoming requests to them. An Oracle RAC database is used for storing metadata and schemas. VIPs are used for the Administration Server (for manual failover).

Note:

Oracle strongly recommends reading the release notes for any additional installation and deployment considerations prior to starting the setup process.

Note:

All command examples are for k or bash shell. No C shell examples are provided.

6.4.1 Preparing the Environment: Prerequisite Steps Before Setting up an Oracle WebCenter High Availability Configuration

The following sections provide prerequisite steps before setting up an Oracle WebCenter high availability configuration.

Ensure that the following initialization parameter is set to the required value. It is checked by Repository Creation Assistant.

Table 6-9 Required Initialization Parameters

Parameter

Required Value

Parameter Class

PROCESSES

300 or greater

Static

To check the value of the initialization parameter using SQL*Plus, you can use the SHOW PARAMETER command.

As the SYS user, issue the SHOW PARAMETER command as follows:

SQL> SHOW PARAMETER processes

Set the initialization parameter using the following command:

SQL> ALTER SYSTEM SET processes=300 SCOPE=SPFILE

Restart the database.

Note:

The method that you use to change a parameter's value depends on whether the parameter is static or dynamic, and on whether your database uses a parameter file or a server parameter file. See the Oracle Database Administrator's Guide for details on parameter files, server parameter files, and how to change parameter values.

ORACLE_HOME: This environment variable and related directory path refers to the location where Oracle WebCenter is installed.

ORACLE_COMMON_HOME: This environment variable and related directory path refers to the Oracle home that contains the binary and library files required for the Oracle Enterprise Manager Fusion Middleware Control and Java Required Files (JRF).

DOMAIN_HOME: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored.

The values used and recommended for consistency for this directories are:

Host Name: Enter the name of the node that is running the database. For an Oracle RAC database, specify the VIP name, or one of the node names as the host name: WCDBHOST1VIRTUAL.

Port: The port number for the database: 1521

Service Name: Enter the service name of the database: wcha.mycompany.com

Username: SYS

Password: Enter the password for the SYS user.

Role: SYSDBA

Click Next.

If you receive the following warning message, click Ignore or Stop:

The database you are connecting is with non-UTF8 charset, if you are going to use this DB for multilingual support, you may have data loss. If you are not using for multilingual support you can continue, otherwise we strongly recommend using UTF-8 database.

In the Select Components screen, select Create a New Prefix, and enter a prefix to use for the database schemas, for example, WCHA

Write down the schema names so they are available in later procedures.

Select the following schemas:

AS Common schemas:

Metadata Services

WebCenter Infrastructure:

WebCenter Suite

Click Next.

In the Schema Passwords screen, enter passwords for the main and additional (auxiliary) schema users, and click Next.

In the Map Tablespaces screen, choose the tablespaces for the selected components, and click Next.

This example uses port 7777. If you choose port 7777, ensure that it is not used by any service on the nodes. You can verify this by running the following command:

Unix:

netstat -an | grep LISTEN | grep ":7777"

Windows:

netstat -an | findstr "LISTEN" | findstr ":7777"

If port 7777 is in use, choose another port, or make it available.

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct, and that you have write permissions for that directory.

To install Oracle WebLogic Server on all nodes in the application tier:

On UNIX platforms, if the /etc/oraInst.loc or /var/opt/oracle/oraInst.loc file exists, check that its contents are correct. Specifically, check that the inventory directory is correct and that you have write permissions for that directory.

Run Oracle Fusion Middleware Configuration Wizard from the WebCenter directory in the Middleware home to create a domain containing the Administration Server and Oracle WebCenter components. Ensure that the database where you installed the repository is running. For Oracle RAC databases, Oracle recommends having all the instances running.

Start Oracle Fusion Middleware Configuration Wizard from the ORACLE_HOME/common/bin directory using the following command:

APPHOST1> ./config.sh

In the Welcome screen, select Create a New WebLogic Domain, and click Next.

In the Select Domain Source screen, select Generate a domain configured automatically to support the following products, and select the following products:

The Repository Creation Utility creates the necessary schemas in the Oracle database. You provide a custom prefix for these schemas. Table 6-10 lists the data sources, the schemas used and the managed servers to which they are assigned.

Ensure that the following data source appears on the screen.

Table 6-10 Oracle WebCenter Data Sources

Data Source

Schema Name

ActivitiesDS

wcedg_activities

DiscussionsDS

wcedg_discussions

PersonalizationDS

wcedg_webcenter

PortletDS

wcedg_portlet

WebCenterDS

wcedg_webcenter

WebCenterMDS

wcedg_mds

Select Configure all data sources as RAC multi data sources in the next panel.

Update each multi data source schema by selecting one data source at a time and adding the appropriate details.

Click Next.

In the Test JDBC Data Sources screen, the connections are tested automatically. The Status column shows the results. Ensure that all connections were successful. If not, click Previous to return to the previous screen and correct your entries.

Click Next when all the connections are successful.

In the Select Optional Configuration screen, select the following:

Administration Server

Managed Servers, Clusters and Machines

In the Customize Server and Cluster Configuration screen, select Yes, and click Next.

In the Configure the Administration Server screen, enter the following values:

6.4.6 Creating boot.properties for the Administration Server and for Managed Servers on APPHOST1

This is an optional step for enabling the Administration Server to start without prompting you for the administrator username and password. Create a boot.properties file for the Administration Server and for the managed servers on APPHOST1.

6.4.7.2 Validating the Administration Server

Verify that all the managed servers (WC_Spaces1, WC_Spaces2, and so on) are listed.

Verify that all Clusters are listed.

Verify that you can access Enterprise Manager at http://VIP1:7001/em.

6.4.7.3 Disabling Host Name Verification for the Administration Server and the Managed Servers for APPHOST1 and APPHOST2

This step is required if you have not set up SSL communication between the Administration Server and the Node Manager. If SSL is not set up, you receive an error message unless you disable host name verification.

You can re-enable host name verification when you have set up SSL communication between the Administration Server and the Node Manager.

6.4.7.4 Starting Node Manager on APPHOST1

Run the setNMProps.sh script, which is located in the ORACLE_COMMON_HOME/common/bin directory, to set the StartScriptEnabled property to true before starting Node Manager:

APPHOST1> cd ORACLE_COMMON_HOME/common/bin
APPHOST1> ./setNMProps.sh

Note:

You must use the StartScriptEnabled property to avoid class loading failures and other problems.

Start Node Manager:

APPHOST1> cd WL_HOME/server/bin
APPHOST1> ./startNodeManager.sh

6.4.8 Install WebLogic Server and Oracle WebCenter on APPHOST2

Repeat the procedures for installing WebLogic Server and Oracle WebCenter for APPHOST2, start with Section 6.4.3.1, "Installing Oracle WebLogic Server". The directory paths for binary files and domains used when installing new nodes must be exactly the same as those used for first node. If these paths and domains are not exactly the same as those used for the first node, failover is does not occur.

6.4.9 Propagating the Domain Configuration to APPHOST2 with pack/unpack Utilities

Follow these steps to propagate the domain configuration to APPHOST2 using Pack/Unpack utilities:

This entry grabs all context roots that are not defined explicitly. Also, Sharepoint and Pagelet Producer require / mapping, which isn't possible to do in a single OHS. For this reason, you need a virtual host configuration. A virtual host configuration is a single system that runs more than one web site, such as www.company1.com and www.company2.com. Virtual hosts are IP-based (a unique IP address for each web site) or name-based (multiple names run on each IP address). It is not obvious to end users that the web sites run on the same physical server. For additional information on virtual hosts, please see Apache HTTP Server documentation.

Oracle HTTP Server Configuration

The following entry adds two name-based hosts: one for Sharepoint Root and one for Pagelet Producer Root.

Ensure that the new virtual hosts webtier-pageletproducer and webtier-spaces have been added to DNS.

6.4.11.1.2 Additional Configuration

In case of Virtual Host setup, you must configure additional properties to use applications routed via virtual host.

Sharepoint

For single sign-on setups, including integration with Oracle Access Manager 10g or Oracle Access Manager 11g, see the Oracle® Fusion Middleware Administrator's Guide for Oracle WebCenter.

Oracle Pagelet Producer

To access Oracle Pagelet Producer, use webtier-pageletproducer.example.com. For example, when you register Oracle Pagelet Producer in WebCenter Spaces or a custom application, it should use the virtual host for Oracle Pagelet Producer.

Similarly, access to Oracle Pagelet Producer and any Pagelet Producer resources should be by means of Virtual Host. Please refer to Oracle Pagelet Producer documentation for additional information.

6.4.11.2 Validating Access through Oracle HTTP Server

Verify the URLS to ensure that appropriate routing and failover is working from the HTTP Server to Oracle WebCenter cluster.

6.4.15 Configuring Oracle WebCenter for Replication

Use the procedures in this section to configure Oracle WebCenter for replication.

Clustering Requirement

The application must be deployed to an Oracle WebLogic Cluster. This automatically establishes a replication channel for the multiple instances of the application.

Note:

In a Unicast cluster, the default replication channel is configured using the Listen address of each managed server. Therefore, the Listen address should be configured to be a specific IP address or host name, instead of being configured to listen on Any.

Oracle ADF Replication

Oracle WebCenter relies on Oracle ADF components, therefore, essential that Oracle ADF is configured properly. The following tag should be present in the adf-config.xml file, one of the Application Resources, for a stateful application:

Ensure that any Oracle WebCenter Portal Application is configured for in-memory replication.

6.4.16 Configuring the Analytics Collectors

The Analytics Collectors must be configured to communicate with WebCenter Spaces. Each Collector is configured to only communicate with the local WebCenter Spaces in a 1-1 relationship. Because the Collectors are configured by default, the only task you must complete is to configure WebCenter Spaces.

6.4.18 Configuring Clustering for Discussion Server

Ensure that all members of the Discussion Server cluster can communicate with each other using the Discussion Server Administration Console:

Log into each member of the cluster at:

http://<host>:<port>/owc_discussions/admin

Go to Cache Settings.

At the bottom of the page, in the Cache Features section, ensure that Clustering is set to Enabled.

The top of the page should now list all members of the cluster.

Again, towards the end of the page, under the Cache Tools section, do Cluster wide cache reset and the Cache warm up Task. Repeat the Cache warm up task on all members of the cluster.

6.4.19 Scaling the Topology

You can scale out and scale up an Oracle WebCenter topology. When you "scale up" the topology, you add new managed servers to nodes that are already running one or more managed servers. When you "scale out" the topology, you add new managed servers to new nodes.

In this case, you already have a node that runs a managed server configured with WebCenter components. The node contains a Middleware home and a WebCenter directory in shared storage.

You can use the existing installations (Middleware home, and domain directories) for creating new Oracle WebCenter managed servers. There is no need to install WebCenter binaries in a new location, or to run pack and unpack. Running multiple managed servers on one node is only supported for the WC_Spaces servers and the WC_Portlet servers.

Follow these steps for scaling up the topology:

Using the Administration Console, clone WC_Spaces1 or WC_Portlet1 into a new managed server. The source managed server to clone should be one that already exists on the node where you want to run the new managed server.

To clone a managed server:

Select Environment -> Servers from the Administration Console.

Select the managed server that you want to clone (for example, WC_Spaces1 or WC_Portlet1).

Select Clone.

Name the new managed server SERVER_NAMEn, where n is a number to identify the new managed server.

For the listen address, assign the host name or IP to use for this new managed server.

Ensure the port number for this managed server is available on this node.

The new node can access the existing home directories for WebLogic Server and Oracle WebCenter. You use the existing installations in shared storage for creating a new managed server. There is no need to install WebLogic Server or WebCenter binaries in a new location, although you must run pack and unpack to create a managed server domain.

Follow these steps for scaling out the topology:

On the new node, mount the existing Middleware home, which should include the WebCenter installation and the domain directory, and ensure that the new node has access to this directory, just like the rest of the nodes in the domain.

To attach ORACLE_HOME in shared storage to the local Oracle Inventory, execute the following commands:

To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.

Log into the Oracle WebLogic Administration Console.

Create a new machine for the new node that will be used, and add the machine to the domain.

Update the machine's Node Manager's address to map the IP of the node that is being used for scale out.

Use the Oracle WebLogic Server Administration Console to clone either WC_Spaces1 or WC_Portlet1 or WC_Collaboration1 or WC_Utilities1 into a new managed server. Name it WLS_XXXn, where n is a number and assign it to the new machine.

For the listen address, assign the host name or IP to use for the new managed server. Perform these steps to set the managed server listen address:

Log into the Oracle WebLogic Server Administration Console.

In the Change Center, click Lock & Edit.

Expand the Environment node in the Domain Structure window.

Click Servers. The Summary of Servers page is displayed.

Select the managed server whose listen address you want to update in the Names column of the table. The Setting page for that managed server is displayed.

Set the Listen Address to WCHOSTn where WCHOSTn is the DNS name of your new machine.

Click Save.

Save and activate the changes.

The changes will not take effect until the managed server is restarted.

Start the Node Manager on the new node. To start the Node Manager, use the installation in shared storage from the existing nodes, and start Node Manager by passing the host name of the new node as a parameter as follows:

6.4.20 Troubleshooting Oracle WebCenter High Availability

6.4.20.1 Troubleshooting Oracle WebCenter Deployment Issues

WebCenter Applications are deployed when the managed server is first started. Use Oracle WebLogic Server Administration Console first to check that all application deployments were successful:

Click Deployments in the left hand pane. The right hand pane shows the application deployments and their status. The state of all applications, assuming all the servers are running, should be ACTIVE.

If an application deployment has failed, the server logs may provide some indication of why the application was not deployed successfully. The server logs are located in the DOMAIN_HOME/servers/SERVER_NAME/logs directory. Common issues include:

Unavailability of external resources, such as database resources. Examine the error, fix it, and attempt to redeploy the application.

The appropriate applications or libraries are not targeted correctly to the right managed server or Cluster.

Replication occurs within the context of a cluster. For failover to be successful, there must be at least one other healthy member of the cluster available. You can check luster status in one of two ways:

Check the cluster status using Oracle WebLogic Server Administration Console - In the Left-hand pane, click on Servers. Verify the state of all servers in the cluster.

Check the cluster status using weblogic.Admin utility The weblogic.Admin command can be used to query the state of all servers in a specific cluster. For example:

There are 2 server(s) in cluster: Spaces_Cluster
The alive servers and their respective states are listed below:
WC_Spaces---RUNNING
WC_Spaces2---RUNNING

Check cluster communications.

Although Cluster members may all be running, there may be communication issues which prevent them from communicating replication information to each other. There are two types of cluster communication configurations. Troubleshooting depends on the cluster type:

Checking Unicast cluster communications - For Unicast clusters, managed servers must be able to access each other's hosts and each other's default listening port.

Ensure that all individual managed servers have their Listen Address set correctly. You can find this setting by selecting Configuration, General for each managed server.

Checking Multicast cluster communications - For multicast clusters, servers must be able to intercept the same multicast traffic. Ensure that multicast is configured correctly by running the WebLogic utility utils.MulticastTest on each machine. For example:

$ java utils.MulticastTest -H

Confirm application configuration.

WebCenter applications are configured correctly by default. For user applications, in-memory replication take place only with the proper configuration in weblogic.xml:

Check the server logs for any unusual messages on managed server startup. In particular, if the managed server is unable to locate other members of the cluster. The server logs are located in the DOMAIN_HOME/servers/SERVER_NAME/logs directory.

For further debugging, enable the flags DebugCluster, DebugClusterAnnouncements, DebugFailOver, DebugReplication, and DebugReplicationDetails. Each flag can be enabled with the weblogic.Admin utility:

6.4.20.3 Troubleshooting Lost Changes to Policies

Policies are refreshed at an interval defined in jps-config.xml by the variable oracle.security.jps.ldap.policystore.refresh.interval. By default, this interval is ten minutes. If a server is lost, requests are routed to another server in the cluster where the original policies are cached. Recent policy changes that occurred since the last refresh may appear to be lost. For example, roles created immediately before the failover may be seen as unavailable.

The policies are still in the back-end policy store. Refresh the cache to restore policy information. Re-try after a few minutes, or log in and log out of the application to force a policy cache refresh.

6.4.20.4 Troubleshooting JOC Configuration

If attempts to start a Managed Server that uses the Java Object Cache, such as OWSM or WebCenter Spaces Managed Servers, fail, and the following errors appear in the logs:

Another process is using the same port that JOC is attempting to obtain. To solve this problem, either stop that process, or reconfigure JOC for this cluster to use another port in the recommended port range.

To use a port other than 8088 for WebCenter Coherence communications, specify a different port number using the -Dtangosol.coherence.wka1.port and -Dtangosol.coherence.wka2.port arguments in the example above.

Oracle WebCenter Portal Applications are deployed on servers and clusters which have been provisioned with the necessary libraries. This provisioning occurs by applying a domain template to an existing WebCenter domain.

The extension can be done only once per domain. This means that any Oracle WebCenter Portal Application Servers must be either created at one time or as clones of existing Oracle WebCenter Portal Application Servers.

6.5.1 Configuring a Cluster for Oracle WebCenter Portal Applications

The following are the necessary steps for configuring a cluster for deploying Oracle WebCenter Portal Applications:

Start Oracle Fusion Middleware Configuration Wizard from the ORACLE_HOME/common/bin directory using the following command:

APPHOST1> ./config.sh

In the Welcome screen, select Extend an Existing WebLogic Domain, and click Next.

In the next screen, select your Weblogic Domain Directory.

In the next screen, choose Extend my Domain using an existing Extension template.

Choose either of these two templates, depending on whether you are creating Custom Applications or Custom Producers:

A managed server named WC_CustomPortal should appear. Rename this server to WC_CustomPortal1.

Click Add to create a new managed server and name it WC_CustomPortal2. Set the Listen Port to be the same as that of WC_CustomPortal1.

In the Configure Clusters screen, add a Cluster named WC_CustomPortalCluster.

In the Assign Servers to Clusters screen, add the two newly created servers to the Cluster.

In the Assign Server to Machines screen, place each of the two servers on separate machines.

Click Extend and then Done to Extend the Domain.

When you complete these steps, you can start the managed server using Oracle WebLogic Server Administration Console, and you can then deploy Oracle WebCenter Portal Applications. Deploy these Oracle WebCenter Portal Applications to the Cluster target, not the managed server target.

6.5.2 Adding More Oracle WebCenter Portal Applications Servers

New servers can only be added to a domain that has already been extended by cloning an existing server: