28 Tuning Integration Flows

To maximize the performance of your integration flows, you must monitor, analyze, and tune all the components. This chapter describes the tools that you can use to monitor performance and the techniques for optimizing the performance of the integration flows as well as the underlying Oracle Fusion Middleware components.

28.1 Introduction to Tuning

Performance tuning usually involves a series of trade-offs. After you have determined what is causing the bottlenecks, you may have to modify performance in some other areas to achieve the desired results. However, if you have a clearly defined plan for achieving your performance objectives, the decision on what to trade for higher performance is easier because you have identified the most important areas.

28.1.1 How to Use Baselines

The most effective way to tune is to have an established performance baseline that can be used for comparison if a performance issue arises.

It is important to identify the peak periods at the site and install a monitoring tool that gathers performance data for those high-load times. Optimally, data gathering should be configured when the application is in its initial trial phase during the QA cycle. Otherwise, this should be configured when the system is first in production.

28.1.2 How to Handle Resource Saturation

If any of the hardware resources are saturated (consistently at or near 100% utilization), one or more of the following conditions may exist:

The hardware resources are insufficient to run the application

The system is not properly configured.

The application or database must be tuned.

There could be some serious problem/bottleneck with in the code/service/process which is consuming more resources or not releasing the resources for a long time.

There could be a problem with echo systems not responding within the expected time and causing a choking situation.

Any bottlenecks identified in production should be fixed in the code and a performance improvement patch must be applied.

For a consistently saturated resource, the solutions are to reduce load or increase resources. For peak traffic periods when the increased response time is not acceptable, consider increasing resources or determine if there is traffic that can be rescheduled to reduce the peak load, such as scheduling batch or background operations during slower periods.

28.1.3 How to Use Proactive Monitoring

Proactive monitoring usually occurs on a regularly scheduled interval, where a number of performance statistics are examined to identify whether the system behavior and resource usage has changed. Proactive monitoring can also be considered as proactive tuning.

Usually, monitoring does not result in configuration changes to the system, unless the monitoring exposes a serious problem that is developing. In some situations, experienced performance engineers can identify potential problems through statistics alone, although accompanying performance degradation is usual.

Experimenting with or tweaking a system when there is no apparent performance degradation as a proactive action can be a dangerous activity, resulting in unnecessary performance drops. Tweaking a system should be considered reactive tuning, and the steps for reactive tuning should be followed.

Oracle Fusion Middleware provides a variety of technologies and tools that can be used to monitor Server and Application performance. Monitoring is an important step in performance tuning and enables you to evaluate server activity, watch trends, diagnose system bottlenecks, debug applications with performance problems and gather data that can assist you in tuning the system.

28.1.4 How to Eliminate Bottlenecks

Tuning usually implies fixing a performance problem. However, tuning should be part of the life cycle of an application-through the analysis, design, coding, production, and maintenance stages. Often the tuning phase is left until the system is in production. At this time, tuning becomes a reactive fire-fighting exercise, where the most important bottleneck is identified and fixed.

System test should be done rigorously against the baseline performance requirements identified during the requirements, analysis, and design phases. System test should be planned with robust test cases considering all the possible scenarios with expected peak loads in each use case which helps in eliminating most of the serious bottlenecks. This comes under the preventive tuning category because the problem would be identified before production. Any bottleneck or performance issue identified during production would fall under the reactive tuning category.

The only constraint with the proactive or preventive tuning is that the production-like environment or the deployment topology cannot be predicted during the QA phase though there are some baselines identified during the requirement and analysis phase. Using an appropriate hardware and extrapolation logic helps in addressing this constraint.

During the application's initial trial run, or while entering QA phase, full-fledged efforts should be taken to tune the application and the environment as part of baseline data collection. Rather than deploying all parts of the application and testing the application exactly the way a production environment will eventually turn out to be, it would be better to take a bottom-up approach. It is highly recommended to take a single end-to-end flow and optimize it fully before introducing multiple flows into the mix. Even for a single flow, it is better to test it with a single user running the five to ten iterations. You must keep tuning the flow until the average response time and throughput for a specific flow is in adherence to the KPI. Look at the metrics for all of the components participating in the flow; identify the areas where most of the time has been spent and find ways to fix them.

Repeat this process with multiple concurrent users executing the same flow. Start with a small number of concurrent users and gradually increase the number. During this testing phase, in addition to looking at metrics generated by the services, look at how garbage collection is occurring by looking at the garbage collection log. This key piece of information sheds light on how the JVM environment can be configured for better performance and throughput. In addition, review the AWR report to check how the database is behaving.

Once all the end-to-end flows have been individually optimized to perform according to KPI for concurrent number of users, multiple disparate flows can be tested simultaneously to check whether a specific flow's characteristics has any kind of impact on other flows.

28.1.5Top Performance Areas

Even though this chapter can be considered a 'Quick Start' guide for fine tuning integration flows, it is not intended to detail a complete list of areas to tune or the techniques on how to identify and fix the issues.

28.2.2 Introducing Automatic Workload Repository

The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and self-tuning purposes. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views.

Oracle generates many types of cumulative statistics for the system, sessions, and individual SQL statements. Oracle also tracks cumulative statistics on segments and services. When analyzing a performance problem in any of these scopes, you typically look at the change in statistics (delta value) over the period you are interested in. Specifically, you look at the difference between the cumulative value of a statistic at the start of the period and the cumulative value at the end.

Cumulative values for statistics are generally available through dynamic performance views, such as the V$SESSTAT and V$SYSSTAT views. Note that the cumulative values in dynamic views are reset when the database instance is shutdown.

The AWR automatically persists the cumulative and delta values for most of the statistics at all levels except the session level. This process is repeated on a regular time period and the result is called an AWR snapshot. The delta values captured by the snapshot represent the changes for each statistic over the time period.

AWR supports the capture of baseline data by enabling you to specify and preserve a pair or range of AWR snapshots as a baseline. Carefully consider the time period you choose as a baseline; the baseline should be a good representation of the peak load on the system. In the future, you can compare these baselines with snapshots captured during periods of poor performance.

Oracle Enterprise Manager is the recommended tool for viewing both real time data in the dynamic performance views and historical data from the AWR history tables. Enterprise manager can also be used to capture operating system and network statistical data that can be correlated with AWR data. For more information, see Oracle Database 2 Day + Performance Tuning Guide.

The statistics collected and processed by AWR include:

Object statistics that determine both access and usage statistics of database segments

Time model statistics based on time usage for activities, displayed in the V$SYS_TIME_MODEL and V$SESS_TIME_MODEL views

Some of the system and session statistics collected in the V$SYSSTAT and V$SESSTAT views

SQL statements that are producing the highest load on the system, based on criteria such as elapsed time and CPU time

Active Session History (ASH) statistics, representing the history of recent sessions activity

Below are the minimum basic configurations to be set for the dehydration store.

Tip:

This SHOULD be implemented in the staging and production environment.

Table 28-1 provides common init.ora parameters and their descriptions. Consider following these guidelines to set the database parameters. Ultimately, however, the DBA should monitor the database health and tune parameters based on the need.

The values should be considered as starting values. Again these values can vary depending upon the target environment's hardware and software topologies. Additional applications or processes, if any, that might run on the target environment could significantly have an impact on these properties. Further, these values must be adjusted based on the inferences made using the data collected through monitoring tools.

Table 28-1 Common init.ora Parameters

Dehydration Store Specific Parameter

Recommended Setting for Staging and Production

Description

shared_pool_size

800M

Applicable when SGA Auto Tuning using sga_target and sga_maxsize is not being used.

The SGA_MAX_SIZE initialization parameter specifies the maximum size of the System Global Area for the lifetime of the instance. You can dynamically alter the initialization parameters affecting the size of the buffer caches, shared pool, and large pool, but only to the extent that the sum of these sizes and the sizes of the other components of the SGA (fixed SGA, variable SGA, and redo log buffers) does not exceed the value specified by SGA_MAX_SIZE.

Ensure that you regularly monitor the buffer cache hit ratio and size the SGA so that the buffer cache has an adequate number of frames for the workload. The buffer cache hit ratio may be calculated from data in the view V$SYSSTAT. Also the view V$DB_CACHE_ADVICE provides data that can be used to tune the buffer cache.

Specifies the target aggregate PGA memory available to all server processes attached to the instance. Starting from 11g, set MEMORY_TARGET instead of setting SGA andthe PGA separately.

processes

800

Set to 800

log_buffer (applicable only for 10g database)

30523392

LOG_BUFFER specifies the amount of memory (in bytes) that Oracle uses when buffering redo entries to a redo log file. Redo log entries contain a record of the changes that have been made to the database block buffers. The LGWR process writes redo log entries from the log buffer to a redo log file

The goal in sizing log_buffer is to set a value that results in the least overall amount of log-related wait events. Common wait events related to a too-small log_buffer size include high "redo log space requests" and a too-large log_buffer may result in high "log file sync" waits.

db_block_size

8192

DB_BLOCK_SIZE specifies (in bytes) the size of Oracle database blocks. The default block size of 8K is optimal for most systems. Set this parameter at the time of database creation.

job_queue_processes

10

JOB_QUEUE_PROCESSES specifies the maximum number of processes that can be created for the execution of jobs. It specifies the number of job queue processes per instance

UNDO_MANAGEMENT

AUTO

UNDO_MANAGEMENT specifies which undo space management mode the system should use. When set to AUTO, the instance starts in automatic undo management mode. In manual undo management mode, undo space is allocated externally as rollback segments. Starting with Oracle Database 11g Release 1 (11.1), the default value of the UNDO_MANAGEMENT parameter is AUTO so that automatic undo management is enabled by default. You must set the parameter to MANUAL to turn off automatic undo management, if required.

open_cursors

1000

OPEN_CURSORS specifies the maximum number of open cursors (handles to private SQL areas) a session can have at once. You can use this parameter to prevent a session from opening an excessive number of cursors. It is important to set the value of OPEN_CURSORS high enough to prevent your application from running out of open cursors. The number varies from one application to another. Assuming that a session does not open the number of cursors specified by OPEN_CURSORS, there is no added performance impact to setting this value higher than actually needed. A value of 1000 for open_cursors is a reasonable number to start with.

Sga_target

1504M

Setting this parameter to a nonzero value enables Automatic Shared Memory Management. Consider using automatic memory management, both to simplify configuration and to improve performance. Consider setting MEMORY_TARGET instead of setting SGA and the PGA separately.

MEMORY_TARGET

2500M

MEMORY_TARGET specifies the Oracle systemwide usable memory. The database tunes memory to the MEMORY_TARGET value, reducing or enlarging the SGA and PGA as needed.

MEMORY_MAX_TARGET

3000M

MEMORY_MAX_TARGET specifies the maximum value to which a DBA can set the MEMORY_TARGET initialization parameter.

Session_cached_cursors

200

SESSION_CACHED_CURSORS specifies the number of session cursors to cache. Repeated parse calls of the same SQL statement cause the session cursor for that statement to be moved into the session cursor cache. Subsequent parse calls find the cursor in the cache and do not reopen the cursor. Oracle uses a least recently used algorithm to remove entries in the session cursor cache to make room for new entries when needed. This parameter also constrains the size of the PL/SQL cursor cache which PL/SQL uses to avoid having to re-parse as statements are re-executed by a user. A starting value of 200 is reasonable for a typical AIA system.

TRACE_ENABLED

FALSE

TRACE_ENABLED controls tracing of the execution history, or code path, of Oracle. Oracle Support Services uses this information for debugging. Although the performance impact incurred from processing is not excessive, you may improve performance by setting TRACE_ENABLED to FALSE.

28.2.4 Tuning Redo Logs Location and Sizing

Managing the database I/O load balancing is a non-trivial task. However, tuning the redo log options can provide performance improvement for applications running in an Oracle Fusion Middleware environment, and in some cases, you can significantly improve I/O throughput by moving the redo logs to a separate disk.

The size of the redo log files can also influence performance, because the behavior of the database writer and archiver processes depend on the redo log sizes. Generally, larger redo log files provide better performance by reducing checkpoint activity. It is not possible to provide a specific size recommendation for redo log files, but redo log files in the range of a hundred megabytes to a few gigabytes are considered reasonable. Size your online redo log files according to the amount of redo your system generates. A rough guide is to switch logs at most once every twenty minutes. Set the initialization parameter LOG_CHECKPOINTS_TO_ALERT = TRUE to have\checkpoint times written to the alert file.\The complete set of required redo log files can be created during database creation. After they are created, the size of a redo log size cannot be changed. New, larger files can be added later, however, and the original (smaller) ones can be dropped.

For more information, see Oracle Fusion Middleware Performance and Tuning Guide.

28.2.5 Automatic Segment-Space Management (ASSM)

For permanent tablespaces, consider using automatic segment-space management. Such tablespaces, often referred to as bitmap tablespaces, are locally managed tablespaces with bitmap segment space management.

For backward compatibility, the default local tablespace segment-space management mode is MANUAL.

28.2.6 Changing the Driver Name to Support XA Drivers

If your data sources require support for XA drivers, you must change the driver name on Oracle WebLogic Server. This is particularly true for environments in which BPEL processes assume XA is present when calling database adapters and JMS adapters.

By properly configuring the connection pool attributes in JDBC data sources in your WebLogic Server domain, you can improve application and system performance. The following sections include information about tuning options for the connection pool in a JDBC data source:

28.2.7.1JDBC Datasource Connection Pool Settings

Statement Cache Type

The Statement Cache Type (or algorithm) determines which prepared and callable statements to store in the cache for each connection in a data source.AIA recommends the property to set to 'LRU'

Statement Cache Size

The Statement Cache Size attribute determines the total number of prepared and callable statements to cache for each connection in each instance of the data source. By caching statements, you can increase your system performance. However, you must consider how your DBMS handles open prepared and callable statements. In many cases, the DBMS maintains a cursor for each open statement. This applies to prepared and callable statements in the statement cache.

AIA recommends keeping the default value of 10. If you cache too many statements, you may exceed the limit of open cursors on your database server. For example, if you have a data source with 50 connections deployed on 2 servers, if you set the Statement Cache Size to 10 (the default), you may open 1000 (50 x 2 x 10) cursors on your database server for the cached statements.

Initial Capacity and Maximum Capacity

For applications that use a database, performance can improve when the connection pool associated with a data source limits the number of connections. You can use the Maximum Capacity to limit the database requests from Oracle Application Server so that incoming requests do not saturate the database, or to limit the database requests so that the database access does not overload the Oracle Application Server-tier resource.

The connection pool MaxCapacity attribute specifies the maximum number of connections that a connection pool allows. By default, the value of MaxCapacity is set to 15. For best performance, you should specify a value for MaxCapacity that matches the number appropriate to your database performance characteristics.

Limiting the total number of open database connections to a number your database can handle is an important tuning consideration. You should check to ensure that your database is configured to allow at least as large a number of open connections as the total of the values specified for all the data sources MaxCapacity option, as specified in all the applications that access the database.

AIA recommends setting Initial and Maximum Capacity to 50 to start with; and make the necessary adjustments based on the database performance.

28.2.7.2 Getting the Right Mix of Performance and Fault Tolerance

These four options can be used to get the right mix of performance and fault tolerance for your system.

Test Frequency

Enable periodic background connection testing by entering the number of seconds between periodic tests. . If the request is made within the time specified for Seconds to Trust an Idle Pool Connection, since the connection was tested or successfully used by an application, WebLogic Server skips the connection test.

Test Reserved Connections

If Test Reserved Connections is enabled on your data source, when an application requests a database connection, WebLogic Server tests the database connection before giving it to the application. If the request is made within the time specified for Seconds to Trust an Idle Pool Connection, since the connection was tested or successfully used by an application, WebLogic Server skips the connection test before delivering it to an application.

Seconds to Trust an Idle Pool Connection

Seconds to Trust an Idle Pool Connection is a tuning feature that can improve application performance by minimizing the delay caused by database connection testing, especially during heavy traffic. However, it can reduce the effectiveness of connection testing, especially if the value is set too high. The appropriate value depends on your environment and the likelihood that a connection will become defunct.

Remove Infected Connections Enabled

Specifies whether a connection will be removed from the connection pool after the application uses the underlying vendor connection object. If you disable removing infected connections, you must ensure that the database connection is suitable for reuse by other applications.

When set to true (the default), the physical connection is not returned to the connection pool after the application closes the logical connection. Instead, the physical connection is closed and re-created.

When set to false, when the application closes the logical connection, the physical connection is returned to the connection pool and can be reused by the application or by another application.

28.2.8Oracle Metadata Service (MDS) Performance Tuning

For optimal performance of MDS APIs, the database schema for the MDS repository must be monitored and tuned by the database administrator. This section lists some recommended actions to tune the database repository:

Collect schema statistics

Increase redo log size

Reclaim disk space

Monitor the database performance

28.2.8.1 Using Database Polling Interval for Change Detection

MDS employs a polling thread which queries the database to gauge if the data in the MDS in-memory cache is out of sync with data in the database. This can happen when metadata is updated in another JVM. If it is out of sync, MDS clears any out of date-cached data so subsequent operations see the latest versions of the metadata.

MDS invalidates the document cache and the MDS cache, so subsequent operations have the latest version of the metadata.

The polling interval can be configured or changed post deployment through MBeans. The element maps to the ExternalChangeDetection and ExternalChangeDetectionInterval attributes of the MDSAppConfig MBean.

Prior to packaging the Enterprise ARchive (EAR) file, you can configure the polling interval by adding this entry in adf-config.xml:

In Example 28-2, 'T' specifies the polling interval in seconds. The minimum value is 1. Lower values cause metadata updates that are made in other JVMs, to be seen more quickly. It is important to note, however, that a lower value can also create increased middle tier and database CPU consumption due to the frequent queries. By default, polling is enabled ('true') and the default value of 30 seconds should be suitable for most purposes. Consider increasing the value to a higher number if the number of updates to MDS are few and far between.

28.2.8.2Tuning Cache Configuration

MDS uses a cache to store metadata objects and related objects (such as XML content) in memory. MDS Cache is a shared cache that is accessible to all users of the application (on the same JVM). If a metadata object is requested repeatedly, with the same customizations, that object may be retrieved more quickly from the cache (a "warm" read). If the metadata object is not found in the cache (a "cold" read), then MDS may cache that object to facilitate subsequent read operations depending on the cache configuration, the type of metadata object and the frequency of access.

Cache can be configured or changed post deployment through MBeans. This element maps to the MaximumCacheSize attribute of the MDSAppConfig mbean.

Having a correctly sized cache can significantly improve throughput for repeated reading of metadata objects. The optimal cache size depends on the number of metadata objects used and the individual sizes of these objects. Prior to packaging the Enterprise ARchive (EAR) file, you can manually update the cache-config in adf-config.xml, by adding the entry shown in Example 28-3.

For more information about tuning the MDS and the database, see "Optimizing Instance Performance" in Oracle Database Performance Tuning Guide, and "Oracle Metadata Service (MDS) Performance Tuning" in Oracle Fusion Middleware Performance and Tuning Guide.

28.3 Configuring the Common SOA Infrastructure

This section discusses properties that impact the entire SOA infrastructure.

28.3.1Configuring SOA Infrastructure Properties

These settings apply to all SOA Composite applications running in the SOA infrastructure. The properties set at this level impact all deployed SOA composite applications, except those composites for which you explicitly set different audit level values at the composite application or service engine levels. AIA recommends configuring at least the following properties.

auditLevel

This property sets the audit trail logging level. This property controls the amount of audit events that are logged by a process.

AIA recommends the following value to be set to this property:

Production: The BPEL service engine does not capture the payload. The payload details are not available in the flow audit trails. Payload details for other BPEL activities are collected, except for assign activities. This level is optimal for most normal operations and testing.

captureCompositeInstanceState

Enabling this option may result in additional run time overhead during instance processing. This option provides for separate tracking of the running instances. All instances are captured as either running or not running. This information displays later in the State column of the composite instances tables for the SOA Infrastructure and SOA composite application, where it shows the counts of running instances versus total instances. You can also limit the view to running instances only

Valid states are Running, Completed, Faulted, Recovery Needed, Stale, Terminated, Suspended, and State Not Available.

The Running and Completed states are captured only if this check box is selected. Otherwise, the state is set to Unknown. The conditional capturing of these states is done mainly to reduce the performance overhead on SOA Infrastructure run time.

Note: If this property is disabled and you create a new instance of a SOA composite application, a new instance is created, but the instance does not display as running, faulted, stale, suspended, terminated, completed, or requiring recovery in the table of the Dashboard page of the composite application. This is because capturing the composite state of instances is a performance-intensive process.

AIA recommends disabling this property.

PayloadValidation

This property validates incoming and outgoing XML documents. If set to True, the SOA Infrastructure applies schema validation for incoming and outgoing XML documents.

This property is applicable to both durable and transient processes. The default value is False. This value can be overridden to turn on schema validation based on business needs. Turning on schema validation both at SOA Infrastructure and at the BPEL Process Service engine causes the schema validation to be done twice.

28.3.2Disabling HTTP Logging

To disable HTTP logging from Weblogic Console for Admin, SOA and BAM servers

28.4.1 Configuring BPEL Process Service Engine Properties

The basic BPEL Process Manager performance tuning properties can be configured using WLST or Enterprise Manager. These properties must be configured:

auditLevel

This property sets the audit trail logging level. This property controls the amount of audit events that are logged by a process.

AIA recommends the following value to be set to this property.

Production: The BPEL service engine does not capture the payload. The payload details are not available in the flow audit trails. Payload details for other BPEL activities are collected, except for assign activities. This level is optimal for most normal operations and testing.

instanceKeyBlockSize

This property controls the instance ID range size. Oracle BPEL Server creates instance keys (a range of process instance IDs) in batches using the value specified. After creating this range of in-memory IDs, the next range is updated and saved in the ci_id_range table. For example, if instanceKeyBlockSize is set to 100, Oracle BPEL Server creates a range of instance keys in-memory (100 keys, which are later inserted into the cube_ instance table as cikey). To maintain optimal performance, ensure that the block size is larger than the number of updates to the ci_id_range table.

The default value is 10000. AIA recommends the default value to be used as is.

auditDetailThreshold

This property sets the maximum size (in kilobytes) of an audit trail details string before it is stored separately from the audit trail. If an audit trail details string is larger than the threshold setting, it is not immediately loaded when the audit trail is initially retrieved; a link is displayed with the size of the details string. Strings larger than the threshold setting are stored in the audit_details table, instead of the audit_trail table. The details string typically contains the contents of a BPEL variable. In cases where the variable is very large, performance can be severely impacted by logging it to the audit trail.

This property sets the large XML document persistence threshold. This is the maximum size (in kilobytes) of a BPEL variable before it is stored in a separate location from the rest of the instance scope data. This property is applicable to both durable and transient processes. Large XML documents impact the performance of the entire Oracle BPEL Server if they are constantly read in and written out whenever processing on an instance must be performed.

This property validates incoming and outgoing XML documents. If set to True, the Oracle BPEL Process Manager applies schema validation for incoming and outgoing XML documents.

This property is applicable to both durable and transient processes. The default value is False. This value can be overridden to turn on schema validation based on business needs. Turning on schema validation both at SOA Infrastructure and at the BPEL Process Service engine causes the schema validation to be done twice.

DispatcherInvokeThreads (dspInvokeThreads in 10g)

This property specifies the total number of threads allocated to process invocation dispatcher messages. Invocation dispatcher messages are generated for each payload received and are meant to instantiate a new instance. If the majority of requests processed by the engine are instance invocations (as opposed to instance callbacks), greater performance may be achieved by increasing the number of invocation threads. Higher thread counts may cause greater CPU utilization due to higher context switching costs.

The default value is 20. AIA recommends this value as the starting point. This parameter can be used throttle the requests to avoid overloading the mid-tier, database tier, and participating application systems and tune for best performance.

DispatcherEngineThreads (dspEngineThreads in 10g)

This property specifies the total number of threads allocated to process engine dispatcher messages. Engine dispatcher messages are generated whenever an activity must be processed asynchronously. If the majority of processes deployed are durable with a large number of dehydration points (mid-process receive, onMessage, onAlarm, and wait activities), greater performance may be achieved by increasing the number of engine threads. Note that higher thread counts can cause greater CPU utilization due to higher context switching costs.

The default value is 30. Since the majority of AIA flows do not have dehydration points, AIA recommends the value to be set to 20. This parameter can be used to throttle the requests to avoid overloading the mid-tier, database tier, and participating application systems and tune for best performance.

DispatcherSystemThreads (dspSystemThreads in 10g)

This property specifies the total number of threads allocated to process system dispatcher messages. System dispatcher messages are general clean-up tasks that are typically processed quickly by the server (for example, releasing stateful message beans back to the pool). Typically, only a small number of threads are required to handle the number of system dispatch messages generated during run time.

The default value is 2. AIA recommends the default value is kept as is.

Disable BPEL Monitors and Sensors

Select this check box to disable all BPEL monitors and sensors defined for all BPEL components across all deployed SOA composite applications.

syncMaxWaittime

This property sets the maximum time the process result receiver waits for a result before returning. Results from asynchronous BPEL processes are retrieved synchronously by a receiver that waits for a result from Oracle BPEL Server. This property is applicable to transient processes.

The default value is 45. Value of syncMaxWaittime parameter mainly depends on the scenario and the number of concurrent processes.

StatsLastN

This property sets the size of the most-recently processed request list. After each request is finished, statistics for the request are kept in a list.

A value less than or equal to 0 disables statistics gathering. This property is applicable to both durable and transient processes.

28.4.2Configuring BPEL Properties Inside a Composite

This section lists the config properties of some sections of the deployment descriptor.

For each configuration property parameter, a description is given, and the expected behavior of the engine when it is changed. All the properties set in this section affect the behavior of the component containing the BPEL composite only. Each BPEL process can be created as a component of a composite. These properties are modified through WLST.

inMemoryOptimization

This property indicates to Oracle BPEL Server that this process is a transient process and dehydration of the instance is not required. When set to True, Oracle BPEL Server keeps the instances of this process in memory only during execution. This property can only be set to True for transient processes or processes that do not contain any dehydration points such as receive, wait, onMessage and onAlarm activities. This property has the following values:

False (default): instances are persisted completely and recorded in the dehydration store database.

True: Oracle BPEL Process Manager keeps instances in memory only.

AIA recommends setting the value to True for transient BPEL processes. BPEL processes that have request-response pattern with no dehydration points should have this property set to True.

completionPersistPolicy

This property configures how much of instance data should be saved. It can only be set at the BPEL component level.

AIA recommends setting the property to Off for transient services. In that situation, no instances of process are saved. It persists only the faulted instances.

PayloadValidation

This property is set at the partnerlink level. This property validates incoming and outgoing XML documents.

If set to True, the Oracle BPEL Process Manager applies schema validation for incoming and outgoing XML documents. When set to True the engine validates the XML message against the XML schema during <receive> and <invoke> for this partnerLink. If the XML message is invalid then bpelx:invalidVariables run time BPEL Fault is thrown. This overrides the domain level validateXML property.

The default value is False. This value can be overridden to turn on schema validation based on business needs.

28.4.3 How to Monitor the BPEL Service Engine

The request and thread statistics for all BPEL process service components running in the service engine must be monitored regularly to check whether the above properties must be tweaked. The statistics page for the BPEL engine provides valuable information. These details help determine how the BPEL engine is performing:

Details on Pending requests in the service engine

Active requests in the service engine

Thread statistics for the service engine

The statistics page also provides information about the count, minimum, average, and maximum processing times for each of the tasks. This information helps the developer and the administrator identify the hot spots.

28.5Oracle Mediator: General Performance Recommendations

28.5.1Configuring Mediator Service Engine Properties

You can configure the properties of Mediator Service Engine by setting the parameters mentioned in this section. These parameters can be set by setting the values of the parameters in the Mediator Service Engine Properties page. These properties must be configured:

auditLevel

This property sets the audit trail logging level. This property controls the amount of audit events that are logged by a process.

AIA recommends the following value to be set to this property.

Production: All events are logged. All audit details, except the details of assign activities, are logged. Instance tracking information is collected, but payload details are not captured and these details are not available in the flow audit trails. This level is optimal for most typical operations and testing.

metricsLevel

Administrator can set the Mediator-specific flag metricsLevel for configuring the Dynamic Monitoring Service (DMS) metrics level. DMS metrics are used to measure the performance of application components. The possible values of this flag are:

Enabled - Enables DMS metrics tracking

Disabled - Disables DMS metrics tracking

AIA recommends disabling the DMS metrics tracking in production. Consider turning this on when there is a need for tracking DMS metrics.

DeferredWorkerThreadCount

Specifies the number of deferred dispatchers for processing messages in parallel. For higher loads consider increasing this parameter to have more outbound threads for deferred processing as each parallel rule is processed by one of the DeferredWorkerThreads.

Default value is 4 threads. AIA recommends changing the value to a higher number only when there are a significant number of mediator services having parallel routing rules.

DeferredMaxRowsRetrieved

When the Mediator routing rule type is set to 'Parallel', DeferredMaxRowsRetrieved sets the number of maximum rows (maximum number of messages for parallel routing rule processing) that are retrieved from Mediator store table (that stores messages for parallel routing rule) for processing.

Note that each message retrieved in this batch is processed by one worker thread at a time. The default value is 20 threads. AIA recommends altering the value only when there are a significant number of mediator services having parallel routing rules.

DeferredLockerThreadSleep

For processing parallel routing rules, Oracle Mediator has a daemon locker thread that retrieves and locks messages from Mediator store database. The thread polls the database to retrieve messages for parallel processing. When no messages are available, the locker thread "sleeps" for the amount of time specified in the DeferredLockerThreadSleep and prevents round trips to database.

Default value is 2 seconds. Consider increasing the value to a higher number, say 120 seconds, for integrations having no mediator components with parallel routing rules.

During the specified time, no messages are available for parallel routing in either of the following cases:

There are no Mediator components with parallel routing rules deployed.

Mediator component(s) with parallel routing rule is deployed, but there are no continuous incoming messages for such components.

SyncMaxWaitTime

The maximum time a request and response operation takes before timing out.

Default value is 45 seconds.

28.5.2 How to Monitor the Mediator Service Engine

The efficiency level of the Mediator service engine can be assessed by monitoring the following:

Routing statistics

Request breakdown statistics.

Instance statistics

The request breakdown statistics provide information about the count and the time taken for processing each of the following actions:

Invoke one-way

Enqueues

Transformation

Invoke request-response

Publish

Condition evaluation

Message validation

Keeping the number of time consuming actions such as enqueues, message validations, transformations as minimal as possible is critical for improving the performance and scalability.

This section describes how to tune Oracle Adapters for optimal performance.

This section focuses only on the tuning aspect for technology adapters that are used by AIA to integrate with applications.

For more information about Oracle Adapters, see Oracle Fusion Middleware User's Guide for Technology Adapters.

28.6.1 How to Tune JMS Adapters

This section describes some of the properties that can be set for the Oracle SOA JMS Adapter to optimize performance.

For more information, see "Introduction to the Oracle JMS Adapter" in Oracle Fusion Middleware User's Guide for Technology Adapters.

If during stress it is taking time to read the messages off the queue, then the JMS adapter thread shown in Example 28-4 can be increased to get the desired output. The following parameter must be set in the bpel.xml property file to achieve the above purpose:

To improve performance, the adapter.jms.receive.threads property can be tuned for an adapter service. The default value is 1, but multiple inbound threads can be used to improve performance. When specified, the value of adapter.jms.receive.threads is used to spawn multiple inbound poller threads.

Be very cautious about increasing the value for this property. Increasing the value increases the load for the Fusion Middleware Server and puts additional stress on the downstream systems such as application systems and databases. Consider increasing the value only when the CPU is underutilized.

Example 28-4 shows the parameter that must be set in the composite.xml property file to achieve the above purpose.

28.6.2 How to Tune AQ Adapters

If during stress it is taking time to read the messages off the queue, then the AQ adapter thread shown in Example 28-5 can be increased to get the desired output. The following parameter must be set in the bpel.xml property file to achieve the above purpose:

To improve performance, the adapter.aq.receive.threads property can be tuned for an adapter service. The default value is 1, but multiple inbound threads can be used to improve performance. When specified, the value of adapter.aq.receive.threads is used to spawn multiple inbound poller threads.

Be very cautious about increasing the value for this property. Increasing the value increases the load for the Fusion Middleware Server and puts additional stress on the downstream systems such as application systems and databases. Consider increasing the value only when the CPU is underutilized.

Example 28-5 shows the parameter that must be set in the composite.xml property file to achieve the above purpose.

28.6.3 How to Tune Database Adapters

This property controls the number of records processed per transaction by each thread. This should be set to a reasonable value, such as 100.

NumberOfThreads

This property controls how many threads the DB adapter uses to process records. The default is 1.

Reducing the numProcessorThread (setting numProcessorThread=4) may improve performance in Core2 Duo boxes but not for Netburst boxes.

UseBatchDestroy

This property controls how the processed records are updated (for example, Deleted for DeletePollingStrategy, MarkedProcessed for LogicalDeleteStrategy). If set, only one update or delete is executed for all the rows part of that transaction. The number of rows in a transaction is controlled by the MaxTransactionSize option.

MaxTransactionSize

If set to a large value such as 1000, turning on the UseBatchDestroy option could have a negative impact on performance.

Setting a large MaxTransactionSize and a small MaxRaiseSize could have negative impact on performance.

Maintaining 10:1 ratio could give better performance.

28.6.4Throttling Inbound Message Flows

The setting in 11g should be configured as a binding property in the composite.xml for the corresponding (inbound) <service> as shown in Example 28-6.

This setting ensures that there is at least 1000 milli seconds delay between two consecutive messages being posted to the BPEL process.

Note:

This setting pertains only to one adapter polling thread. If multiple adapter polling threads (for example, multiple JMS dequeuer threads) have been configured, this setting controls the speed of each thread, not the combined speed.

28.7Purging the Completed Composite Instances

Data growth from auditing and dehydration can have a significant impact on database performance and throughput. See the sections on auditLevel for audit configuration and inMemoryOptimization for dehydration configuration.

These tables are impacted by instance data growth:

Audit_trail

Audit_details

Cube_instance

Cube_scope

Dlv_message

Dlv_subscription

Document_ci_ref

Document_dlv_message_ref

Schema_md

Task

Work_item

Xml_document

Header_properties

The database administrator must continuously monitor the growth of these tables and make sure these tables are configured properly to give the optimal performance.

In addition, AIA highly recommends having a process in place to archive and clean up the completed composite instances after a specified period.

Synchronous request -response based completed composite instances should be purged from SOA dehydration data store after a month.

Composite Business Processes should be purged 1 - 2 years after their completion.

Even though this duration is subject to change based on your company's policies, attempts should be made to do the purging at the earliest possible time. Purging must be preceded by the archiving of the instance data.

28.8Tuning Java Virtual Machines (JVMs)

This section discusses how to optimize the JVM heap.

28.8.1 How to Optimize the JVM Heap - Specifying Heap Size Values

The goal of tuning your heap size is to minimize the time that your JVM spends doing garbage collection while maximizing the number of clients that the Fusion Middleware stack can handle at a given time.

Specifically the Java heap is where the objects of a Java program live. It is a repository for live objects, dead objects, and free memory. When an object can no longer be reached from any pointer in the running program, it is considered "garbage" and ready for collection. A best practice is to tune the time spent doing garbage collection to within 5% of execution time.

The JVM heap size determines how often and how long the virtual machine spends collecting garbage. An acceptable rate for garbage collection is application-specific and should be adjusted after analyzing the actual time and frequency of garbage collections. If you set a large heap size, full garbage collection is slower, but it occurs less frequently. If you set your heap size in accordance with your memory needs, full garbage collection is faster, but occurs more frequently.

In production environments, set the minimum heap size and the maximum heap size to the same value to prevent wasting virtual machine resources used to constantly grow and shrink the heap. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM. If this value is exceeded, the Operating System starts paging and performance degrades significantly. The virtual machine always uses more memory than the heap size. The memory required for internal virtual machine functionality, native libraries outside of the virtual machine, and permanent generation memory (memory required to store classes and methods) is allocated in addition to the heap size settings.

To tune the JVM garbage collection options you must analyze garbage collection data and check for the frequency and type of garbage collections, the size of the memory pools, and the time spent on garbage collection.

Before you configure JVM garbage collection, analyze the following data points:

How often is garbage collection taking place? Compare the time stamps around the garbage collection.

How long is a full garbage collection taking?

What is the heap size after each full garbage collection? If the heap is always 85% free, for example, you might set the heap size smaller.

Do the young generation heap sizes (Sun) or Nursery size (Jrockit) need tuning?

If you run out of heap memory (not due to a memory leak), increase -Xmx.

If you run out of native memory, you may need to decrease -Xmx.

For Sun JVM, modify -Xmn to tune the size of the heap for the young generation.

For Oracle JRockit, modify -Xns:<nursery size> to tune the size of the nursery.

If your AIA service instance runs on a dedicated host, set the heap size value as high as possible but make sure it does not exceed the available physical RAM.

Example 28-6 shows values suggested for 32-bit Sun JVM running on 32-bit Linux on 32-bit/64-bit hardware. It also assumes that the host machine has sufficient physical RAM installed and that other OS processes are not consuming a significant portion of the available memory.

It is highly recommended that the production environment has 64-bit JVM running on 64-bit hardware with atleast 4 GB of memory. The following configurations are based on assumption that the environment has atleast 3 GB of memory.

Table 28-2 JVM Heap Size Values

JVM Parameters

Recommended Value for Production

Notes

Initial heap size

2048

It is recommended that Initial Heap and Maximum Heap are set to the same value for the proper memory utilization and minimize major garbage collection.

A small heap becomes full quickly and must be garbage collected more often. It is also prone to more fragmentation, making object allocation slower. A large heap introduces a slight overhead in garbage collection times. A heap that is larger than the available physical memory in the system must be paged out to disk, which leads to long access times or even application freezes, especially during garbage collection.

In general, the extra overhead caused by a larger heap is smaller than the gains in garbage collection frequency and allocation speed, as long as the heap does not get paged to disk. Thus a good heap size setting would be a heap that is as large as possible within the available physical memory.

Maximum heap size

2048

Enable J2SE 5.0 Platform Mbeans

Enable

Keep it enabled, so that memory and cpu real time monitoring can be done and further tuning can be done if required.

-XX:PermSize

256

This setting is used to allocate memory outside of heap space to hold java classes.

Note: JRockit JVM does not have this capability.

-XX:MaxPermSize

256

-XX:AppendRatio

3

None

-XX:NewSize

818

Another important heap configuration is the garbage collector's generational settings. The garbage collector optimizes collection by classifying objects by how long they live. Most of the BPEL engine's objects are short lived, thus they live in the "Eden" space. We recommend sizing the "Eden" space to be 40-50% of the total maximum heap size. The following command line starts Java with a Eden sizing that is 60% of the maximum heap size. Also If you are using 2 or more than 2 CPUs and it is a non-Windows machine, it is recommended to use -XX:+ AggressiveOpts jvm flag. The -XX:+ AggressiveOpts option inspects the machine resources (size of memory and number of processors) and attempts to set various parameters to be optimal for long-running, memory allocation-intensive jobs. Note this flag does not affect Windows performance.

In JRockit JVM, young (new) generation is called nursery size - and it is specified using -Xns<size>

-XX:MaxNewSize

818

-XX:SurvivorRatio

6

-XX:+AggressiveOpts

Set in the JVM argument

Example 28-7 shows sample JVM configurations for 64-bit Sun JVM running on 64-bit Linux on 64-bit hardware having 16GB RAM. Note that settings such as heap space (-mx & -ms ), young generation (MaxNewSize & NewSize), permanent size (MaxPermSize), ParallelGCThreads must be adjusted according to the memory and processors available in your environment. Similarly, directory paths for several of the directives listed below must be adjusted accordingly.

28.8.1.1 Java Performance Analysis Tools

A profiler is a performance analysis tool that enables you to reveal hot spots in the application that result in either high CPU utilization or high contention for shared resources. Some common profilers are:

OptimizeIt Java Performance Profiler at http://www.borland.com/optimizeit/optimizeit_profiler/index.html from Borland, a performance debugging tool for Solaris and Windows

JProbe Profiler with Memory Debugger at http://www.quest.com/jprobe/, a family of products that provide the capability to detect performance bottlenecks, perform code coverage and other metrics

VTune Performance Analyzer at http://www.intel.com/software/products/vtune/, a tool to identify and locate performance bottlenecks in your code

PerformaSure at http://www.quest.com/performasure/, a tool to detect, diagnose, and resolve performance problems in multitier J2EE applications

28.9Tuning Weblogic Application Server

This section discusses how to optimize the Weblogic server for high performance.

28.9.1 Domain Startup Mode - Production

The Weblogic server must be created using production mode. This sets many properties like log level to minimum to optimize the Weblogic server performance.

28.9.2Work Manager - default

To handle thread management and perform self-tuning, WebLogic Server implements a default Work Manager. This Work Manager is used by an application when no other Work Managers are specified in the application's deployment descriptors.

In many situations, the default Work Manager WebLogic Server's thread-handling algorithms assign each application its own fair share by default. Applications are given equal priority for threads and are prevented from monopolizing them.

28.9.3 Tuning Network I/O

Download and enable performance pack for the specific operating system where Weblogic server is installed.

Scripting on this page enhances content navigation, but does not change the content in any way.