2 Top Performance Areas

This chapter describes the top tuning areas for Oracle Fusion Middleware. It covers critical Oracle Fusion Middleware performance areas and provides a quick start for tuning Java EE applications in the following sections:

2.1 About Identifying Top Performance Areas

One of the most challenging aspects of performance tuning is knowing where to begin. This chapter serves as a 'quick start' guide to performance tuning your Oracle Fusion Middleware applications.

Table 2-1 provides a list of common performance considerations for Oracle Fusion Middleware. While the list is a useful tool in starting your performance tuning, it is not meant to be comprehensive list of areas to tune. You must monitor and track specific performance issues within your application to understand where tuning can improve performance. See Chapter 4, "Monitoring Oracle Fusion Middleware" for more information.

Logging levels are thresholds that a system administrator sets to control how much information is logged. Performance can be impacted by the amount of information that applications log therefore it is important to set the logging levels appropriately.

2.2 Securing Sufficient Hardware Resources

A key component of managing the performance of Oracle Fusion Middleware applications is to ensure that there are sufficient CPU, memory, and network resources to support the user and application requirements for your installation.

No matter how well you tune your applications, if you do not have the appropriate hardware resources, your applications cannot reach optimal performance levels. Oracle Fusion Middleware has minimum hardware requirements for its applications and database tier. For details on Oracle Fusion Middleware supported configurations, see "System Requirements and Prerequisites" in the Oracle Fusion Middleware Installation Planning Guide for your platform.

Sufficient hardware resources should meet or exceed the acceptable response times and throughputs for applications without becoming saturated. To verify that you have sufficient hardware resources, you should monitor resource utilization over an extended period to determine if (or when) you have occasional peaks of usage or whether a resource is consistently saturated. For more information on monitoring, see Chapter 4, "Monitoring Oracle Fusion Middleware".

Tip:

Your target CPU usage should not reach 100% utilization. You should determine a target CPU utilization based on your application needs, including CPU cycles for peak usage.

If your CPU utilization is optimized at 100% during normal load hours, you have no capacity to handle a peak load. In applications that are latency sensitive and maintaining a fast response time is important, high CPU usage (approaching 100% utilization) can increase response times while throughput stays constant or even decreases. For such applications, a 70% - 80% CPU utilization is recommended. A good target for non-latency sensitive applications is about 90%.

If any of the hardware resources are saturated (consistently at or near 100% utilization), one or more of the following conditions may exist:

The hardware resources are insufficient to run the application.

The system is not properly configured.

The application or database must be tuned.

For a consistently saturated resource, the solutions are to reduce load or increase resources. For peak traffic periods when the increased response time is not acceptable, consider increasing resources or determine if there is traffic that can be rescheduled to reduce the peak load, such as scheduling batch or background operations during slower periods.

Oracle Fusion Middleware provides a variety of mechanisms to help you control resource concurrency; this can limit the impact of bursts of traffic. However, for a consistently saturated system, these mechanisms should be viewed as temporary solutions. For more information see Section 2.9, "Controlling Concurrency".

2.3 Tuning the Operating System

Each operating system has native tools and utilities that can be useful for monitoring and tuning purposes. Native operating system commands enable you to monitor CPU utilization, paging activity, swapping, and other system activity information.

For details on operating system commands, and guidelines for performance tuning of the network or operating system, refer to the documentation provided by the operating system vendor.

2.4 Tuning Java Virtual Machines (JVMs)

How you tune your JVM greatly affects the performance of Oracle Fusion Middleware and your applications.

Note:

To maximize performance from your JVM, be sure that you use only production JVMs on which your applications have been certified and that your operating system patches are up-to-date.

2.4.1 Configuring Garbage Collection

Garbage collection is the JVM process of freeing up unused Java objects in the Java heap. JVM garbage collection can be a resource-intensive operation and may effect application performance. In some cases, inefficient garbage collection can severely degrade application performance. Therefore, it is important to understand how applications create and destroy objects.

An acceptable rate for garbage collection is application-specific and should be adjusted after analyzing the actual time and frequency of garbage collections. If you set a large heap size, full garbage collection is slower, but it occurs less frequently. If you set your heap size in accordance with your memory needs, full garbage collection is faster, but occurs more frequently.

To tune the JVM garbage collection options you must analyze garbage collection data and check for the frequency and type of garbage collections, the size of the memory pools, and the time spent on garbage collection.

Before you configure JVM garbage collection, analyze the following data points:

How often is garbage collection taking place? Compare the time stamps around the garbage collection.

How long is a full garbage collection taking?

What is the heap size after each full garbage collection? If the heap is always 85 percent free, for example, you might set the heap size smaller.

Do the young generation heap sizes (Sun) or Nursery size (Jrockit) need tuning?

2.4.1.1 Specifying Heap Size Values

The goal of tuning your heap size is to minimize the time that your JVM spends doing garbage collection while maximizing the number of clients that the Fusion Middleware stack can handle at a given time.

Specifically the Java heap is where the objects of a Java program live. It is a repository for live objects, dead objects, and free memory. When an object can no longer be reached from any pointer in the running program, it is considered "garbage" and ready for collection. A best practice is to tune the time spent doing garbage collection to within 5% of execution time.

The JVM heap size determines how often and how long the virtual machine spends collecting garbage. An acceptable rate for garbage collection is application-specific and should be adjusted after analyzing the actual time and frequency of garbage collections. If you set a large heap size, full garbage collection is slower, but it occurs less frequently. If you set your heap size in accordance with your memory needs, full garbage collection is faster, but occurs more frequently.

In production environments, set the minimum heap size and the maximum heap size to the same value to prevent wasting virtual machine resources used to constantly grow and shrink the heap. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM. If this value is exceeded, the Operating System starts paging and performance degrades significantly. The virtual machine always uses more memory than the heap size. The memory required for internal virtual machine functionality, native libraries outside of the virtual machine, and permanent generation memory (memory required to store classes and methods) is allocated in addition to the heap size settings.

For example, you can use the following JVM options to tune the heap:

If you run out of heap memory (not due to a memory leak), increase -Xmx.

If you run out of native memory, you may need to decrease -Xmx.

For Oracle JRockit, modify -Xns:<nursery size> to tune the size of the nursery.

For Sun JVM, modify -Xmn to tune the size of the heap for the young generation.

If you receive java.lang.OutOfMemoryError: PermGen space errors, you may also need to increase the permanent generation space.

2.4.1.2 Selecting a Garbage Collection Scheme

Depending on which JVM you are using, you can choose from several garbage collection schemes to manage your system memory. Some garbage collection schemes are more appropriate for a given type of application. Once you have an understanding of the workload of the application and the different garbage collection algorithms utilized by the JVM, you can optimize the configuration of the garbage collection.

Refer to the following links for garbage collection options for your JVM:

2.4.1.3 Disabling Explicit Garbage Collection

The following parameters are used to help diagnose whether explicit garbage collections are occurring. They can also be used to disable the explicit garbage collections if necessary until the code is fixed:

These parameters disable explicit garbage collection. Applications should avoid the use of system.gc() calls. If you suspect an application may be explicitly triggering garbage collection, set this parameter and observe the differences in your garbage collection behavior. If you detect that performance is affected by explicit collections, check the code to determine where explicit garbage collections are used and why, and the impact of disabling the calls. Application developers sometimes use system.gc() calls to trigger finalizers. This is not a recommended practice and can yield indeterminate behavior.

2.4.2 Logging Low Memory Conditions

WebLogic Server enables you to automatically log low memory conditions observed by the server. WebLogic Server detects low memory by sampling the available free memory a set number of times during a time interval. At the end of each interval, an average of the free memory is recorded and compared to the average obtained at the next interval. If the average drops by a user-configured amount after any sample interval, the server logs a low memory warning message in the log file and sets the server health state to "warning."

See Also:

For more information on using WebLogic Server to detect low memory conditions refer to the following:

2.4.3 Monitoring and Profiling the JVM

Monitoring the performance of your JVM is crucial to achieving optimal performance. Depending on your platform, the following tools can be used to monitor and profile your JVM:

Oracle JRockit® Mission Control

Oracle JRockit Mission Control is a suite of tools designed to monitor, manage, profile, and eliminate memory leaks in your Java application without the performance impacts normally associated with these types of tools.

2.5 Tuning the WebLogic Server

2.6 Tuning Database Parameters

To achieve optimal performance for applications that use the Oracle database, the database tables you access must be designed with performance in mind. Monitoring and tuning the database ensures that you get the best performance from your applications.

2.6.1 Tuning init.ora Database Parameters

The following tables provide common init.ora parameters and their descriptions. Consider following these guidelines to set the database parameters. Ultimately, however, the DBA should monitor the database health and tune parameters based on the need. See the following tables for more information:

Consider applying Patch Set Release (PSR) 11.1.0.7 and upgrade the database prior to attempting the following modifications.

2.6.1.1 Initialization Parameters for Oracle 10g

The following table describes several performance-related database initialization parameters for Oracle 10g database. The tuning considerations listed below are applicable to most scenarios. Always set your database parameters based on your own use case scenarios.

Table 2-2 Important init.ora Oracle 10g Database Tuning Parameters

Database Parameter

Description

_b_tree_bitmap_plans

Consider setting this parameter to FALSE to prevent optimizer from attempting bitmap operations as there are no bitmap indexes in Fusion Middleware.

DB_BLOCK_SIZE

DB_BLOCK_SIZE specifies (in bytes) the size of Oracle database blocks. The default block size of 8K is optimal for most systems. Set this parameter at the time of database creation.

NLS_SORT

Consider setting NLS_SORT to BINARY, otherwise sort will do full table scan and performance can be impacted.

OPEN_CURSORS

Consider using a value of 500 open cursors (handles to private SQL areas) a session can have at once.

SESSION_CACHED_CURSORS

Consider using a value of 500 session cursors to cache.

SESSION_MAX_OPEN_FILES

SESSION_MAX_OPEN_FILES specifies the maximum number of BFILEs that can be opened in any session. Once this number is reached, subsequent attempts to open more files in the session by using DBMS_LOB.FILEOPEN() or OCILobFileOpen() may fail. The maximum value for this parameter depends on the equivalent parameter defined for the underlying operating system.

JOB_QUEUE_PROCESSES

JOB_QUEUE_PROCESSES specifies the maximum number of processes that can be created for the execution of jobs. It specifies the number of job queue processes per instance.

LOG_BUFFER

LOG_BUFFER specifies the amount of memory (in bytes) that Oracle uses when buffering redo entries to a redo log file. Redo log entries contain a record of the changes that have been made to the database block buffers. The LGWR process writes redo log entries from the log buffer to a redo log file.

UNDO_MANAGEMENT

UNDO_MANAGEMENT specifies which undo space management mode the system should use. When set to AUTO, the instance starts in automatic undo management mode. In manual undo management mode, undo space is allocated externally as rollback segments.

PL_SQL_CODE_TYPE

Consider setting PL_SQL_CODE_TYPE to NATIVE

PROCESSES

Consider using a value of 5000 operating system processes to be connected to Oracle concurrently.

PGA_AGGREGATE_TARGET

Consider setting PGA_AGGREGATE_TARGET to 1G of PGA memory available to all server processes attached to the instance.

SGA_MAX_SIZE

Consider setting the SGA_MAX_SIZE to 2G initially and then monitor the production database on daily basis and adjust SGA and PGA accordingly.

SGA_TARGET

Consider setting the SGA_TARGET to 2G initially and then monitor the production database on daily basis and adjust SGA and PGA accordingly.

TRACE_ENABLED

TRACE_ENABLED controls tracing of the execution history, or code path, of Oracle. Oracle Support Services uses this information for debugging.

Although the performance impacts incurred from processing is not excessive, you may improve performance by setting TRACE_ENABLED to FALSE.

2.6.1.2 Initialization Parameters for Oracle 11g

The following table provides information on some important performance-related database initialization parameters for Oracle 11g database.

Table 2-3 Important inti.ora Oracle 11g Database Tuning Parameters

Database Parameter

Description

AUDIT_TRAIL

If there is NO policy to audit db activity, consider setting this parameter to NONE. Enabling auditing can impact performance.

MEMORY_MAX_TARGET

MEMORY_MAX_TARGET specifies the maximum value to which a DBA can set the MEMORY_TARGET initialization parameter.

MEMORY_TARGET

Consider setting the MEMORY_TARGET to NONE. Set SGA and PGA separately as setting MEMORY_TARGET does not allocate sufficient memory to SGA and PGA as needed.

PGA_AGGREGATE_TARGET

Consider using a value of 1G for PGA initially and then monitor the production database on daily basis and adjust SGA and PGA accordingly.

If the database server has more memory, consider setting PGA_AGGREGATE_TARGET to a value higher than 1G based on usage needs.

Consider using a value of 2G for SGA is 2G to start with and initially and then monitor the production database on daily basis and adjust SGA and PGA accordingly.

If the database server has more memory, consider setting SGA_TARGET to a value higher than 2G based on usage needs.

2.6.2 Tuning Redo Logs Location and Sizing

Managing the database I/O load balancing is a non-trivial task. However, tuning the redo log options can provide performance improvement for applications running in an Oracle Fusion Middleware environment, and in some cases, you can significantly improve I/O throughput by moving the redo logs to a separate disk.

Consider having at least 3 redo log groups with 2G of size each. Redo log files should be placed on a disk separate from data files to improve I/O performance.

2.6.3 Tuning Automatic Segment-Space Management (ASSM)

For permanent tablespaces, consider using automatic segment-space management. Such tablespaces, often referred to as bitmap tablespaces, are locally managed tablespaces with bitmap segment space management.

For backward compatibility, the default local tablespace segment-space management mode is MANUAL.

2.7 Reusing Database Connections

Creating a database connection is a relatively resource intensive process in any environment. Typically, a connection pool starts with a small number of connections. As client demand for more connections grow, there may not be enough in the pool to satisfy the requests. WebLogic Server creates additional connections and adds them to the pool until the maximum pool size is reached.

One way to avoid connection creation delays is to initialize all connections at server startup, rather than on-demand as clients need them. This may be appropriate if your load is predictable and even. Set the initial number of connections equal to the maximum number of connections in the Connection Pool tab of your data source configuration. Determine the optimal value for the Maximum Capacity as part of your pre-production performance testing.

If your load is uneven, and has a much higher number of connections at peak load than at typical load, consider setting the initial number of connections equal to your typical load. In addition, consider setting the maximum number of connections based on your supported peak load. With these configurations, WebLogic server can free up some connections when they are not used for a period of time.

2.8 Enabling Data Source Statement Caching

When you use a prepared statement or callable statement in an application or EJB, there may be a performance impact associated with the processing of the communication between the application server and the database server and on the database server. To minimize the processing impact, enable the data source to cache prepared and callable statements used in your applications. When an application or EJB calls any of the statements stored in the cache, the server reuses the statement stored in the cache. Reusing prepared and callable statements reduces CPU usage on the database server, improving performance for the current statement and leaving CPU cycles for other tasks.

Each connection in a data source has its own individual cache of prepared and callable statements used on the connection. However, you configure statement cache options per data source. That is, the statement cache for each connection in a data source uses the statement cache options specified for the data source, but each connection caches it's own statements. Statement cache configuration options include:

Statement Cache Type—The algorithm that determines which statements to store in the statement cache.

Statement Cache Size—The number of statements to store in the cache for each connection. The default value is 10. You should analyze your database's statement parse metrics to size the statement cache sufficiently for the number of statements you have in your application.

2.9 Controlling Concurrency

Limiting concurrency, at multiple layers of the system to match specific usage needs, can greatly improve performance. This section discusses a few of the areas within Oracle Fusion Middleware where concurrency can be controlled.

When system capacity is reached, and a web server or application server continues to accept requests, application performance and stability can deteriorate. There are several places within Oracle Fusion Middleware where you can throttle the requests to avoid overloading the mid-tier or database tier systems and tune for best performance.

2.9.1 Setting HTTP Connection Limits

Oracle HTTP Server uses directives in httpd.conf. This configuration file specifies the maximum number of HTTP requests that can be processed simultaneously, logging details, and certain limits and time outs.

You can use the MaxClients and ThreadsPerChild directives to limit incoming requests to WebLogic instances from the Oracle HTTP Server based on your expected client load and system resources. The following sections describe some Oracle HTTP Server tuning parameters related to connection limits that you typically need to tune based on your expected client load. See Chapter 5, "Oracle HTTP Server Performance Tuning" for more information and a more complete list of tunable parameters.

2.9.1.1 MaxClients/ThreadsPerChild

Note:

The MaxClients parameter is applicable only to UNIX platforms and on Microsoft Windows (mpm_winnt), the same is achieved through the ThreadsPerChild and ThreadLimit parameters.

The MaxClients property specifies a limit on the total number of server threads running, that is, a limit on the number of clients who can simultaneously connect. If the number of client connections reaches this limit, then subsequent requests are queued in the TCP/IP system up to the limit specified (in the ListenBackLog directive).

You can configure the MaxClients directive in the httpd.conf file up to a maximum of 8K (the default value is 150). If your system is not resource-saturated and you have a user population of more than 150 concurrent HTTP connections, you can improve your performance by increasing MaxClients to increase server concurrency. Increase MaxClients until your system becomes fully utilized (85% is a good threshold).

When system resources are saturated, increasing MaxClients does not improve performance. In this case, the MaxClients value could be reduced as a throttle on the number of concurrent requests on the server.

If the server handles persistent connections, then it may require sufficient concurrent httpd server processes to handle both active and idle connections. When you specify MaxClients to act as a throttle for system concurrency, you need to consider that persistent idle httpd connections also consume httpd processes. Specifically, the number of connections includes the currently active persistent and non-persistent connections and the idle persistent connections. When there are no httpd server threads available, connection requests are queued in the TCP/IP system until a thread becomes available, and eventually clients terminate connections.

You can define a number of server processes and the threads per process (ThreadsPerChild) to handle the incoming connections to Oracle HTTP Server. The ThreadsPerChild property specifies the upper limit on the number of threads that can be created under a server (child) process.

Note:

ThreadsPerChild, StartServers, and ServerLimit properties are inter-related with the MaxClients setting. All of these properties must be set appropriately to achieve the number of connections as specified by MaxClients. See Table 5-1, "Oracle HTTP Server Configuration Properties" for a description of all the HTTP configuration properties.

2.9.1.2 KeepAlive

A persistent, KeepAlive, HTTP connection consumes an httpd child process, or thread, for the duration of the connection, even if no requests are currently being processed for the connection.

If you have sufficient capacity, KeepAlive should be enabled; using persistent connections improves performance and prevents wasting CPU resources re-establishing HTTP connections. Normally, you should not need to change KeepAlive parameters.

Note:

The default maximum requests for a persistent connection is 100, as specified with the MaxKeepAliveRequests directive in httpd.conf. By default, the server waits for 15 seconds between requests from a client before closing a connection, as specified with the KeepAliveTimeout directive in httpd.conf.

2.9.1.3 Tuning MOD_WL_OHS

The Oracle HTTP Server (OHS) uses the mod_wl_ohs module to route requests to the underlying Weblogic server or the Weblogic Server cluster. The configuration details for mod_wl_ohs are available in the mod_wl_ohs.conf file in the config directory.

2.9.2 Configuring Connection Pools

Connection pooling is configured and maintained per Java runtime. Connections are not shared across different runtimes. To use connection pooling, no configuration is required. Configuration is necessary only if you want to customize how pooling is done, such as to control the size of the pools and which types of connections are pooled.

You configure connection pooling by using a number of system properties at program startup time. Note that these are system properties, not environment properties and that they affect all connection pooling requests.

For applications that use a database, performance can improve when the connection pool associated with a data source limits the number of connections. You can use the MaxCapacity attribute to limit the database requests from Oracle Application Server so that incoming requests do not saturate the database, or to limit the database requests so that the database access does not overload the Oracle Application Server-tier resource.

The connection pool MaxCapacity attribute specifies the maximum number of connections that a connection pool allows. By default, the value of MaxCapacity is set to 15. For best performance, you should specify a value for MaxCapacity that matches the number appropriate to your database performance characteristics.

Limiting the total number of open database connections to a number your database can handle is an important tuning consideration. You should check to make sure that your database is configured to allow at least as large a number of open connections as the total of the values specified for all the data sources MaxCapacity option, as specified in all the applications that access the database.

2.9.3 Tuning the WebLogic Sever Thread Pool

By default WebLogic Server uses a single thread pool, in which all types of work are executed. WebLogic Server uses Work Managers to prioritize work based on rules you can define, and run-time metrics, including the actual time it takes to execute a request and the rate at which requests are entering and leaving the pool. There is a default work manager that manages the common thread pool.

The common thread pool changes its size automatically to maximize throughput. WebLogic Server monitors throughput over time and based on history, determines whether to adjust the thread count. For example, if historical throughput statistics indicate that a higher thread count increased throughput, WebLogic increases the thread count. Similarly, if statistics indicate that fewer threads did not reduce throughput, WebLogic decreases the thread count.

Since the WebLogic Server thread pool by default is sized automatically, in most situations you do not need to tune this. However, for special requirements, an administrator can configure custom Work Managers to manage the thread pool at a more granular level for sets of requests that have similar performance, availability, or reliability requirements. With custom work managers, you can define priorities and guidelines for how to assign pending work (including specifying a min threads or max threads constraint, or a constraint on the total number of requests that can be queued or executing before WebLogic Server begins rejecting requests).

Use the following guidelines to help you determine when to use Work Managers to customize thread management:

The default fair share is not sufficient.

This usually occurs in situations where one application needs to be given a higher priority over another.

A response time goal is required.

A minimum thread constraint needs to be specified to avoid server deadlock.

You use MDBs in your application.

To ensure MDBs use a well-defined share of server thread resources, and to tune MDB concurrency, most MDBs should be modified to reference a custom work manager that has a max-threads-constraint. In general, a custom work manager is useful when you have multiple MDB deployments, or if you determine that a particular MDB needs more threads.

See Also:

For more information on how to use custom Work Managers to customize thread management, and when to use custom work managers, see the following:

You can use Oracle WebLogic Administration Console to view general information about the status of the thread pool (such as active thread count, total thread count, and queue length.) You can also use the Console to view each application's scoped work manager metrics from the Workload tab on the Monitoring page. The metrics provided include the number of pending requests and number of completed requests.

2.9.5 Tuning BPEL Concurrency

2.10 Setting Logging Levels

The amount of information that applications log depends on how the environment is configured and how the application code is instrumented. To maximize performance it is recommended that the logging level is not set higher than the default INFO level logging. If the logging setting does not match the default level, reset the logging level to the default for best performance.

Once the application and server logging levels are set appropriately, ensure that the debugging properties or other application level debugging flags are also set to appropriate levels or disabled. To avoid performance impacts, do not set log levels to levels that produce more diagnostic messages, including the FINE or TRACE levels.

Scripting on this page enhances content navigation, but does not change the content in any way.