Before reading this chapter, Oracle recommends becoming familiar with WebLogic store administration and monitoring. See Using the WebLogic Persistent Store in Configuring Server Environments for Oracle WebLogic Server.

Overview of Persistent Stores

The following sections provide information on using persistent stores.

Using the Default Persistent Store

Each server instance, including the administration server, has a default persistent store that requires no configuration. The default store is a file-based store that maintains its data in a group of files in a server instance's data\store\default directory. A directory for the default store is automatically created if one does not already exist. This default store is available to subsystems that do not require explicit selection of a particular store and function best by using the system's default storage mechanism. For example, a JMS Server with no persistent store configured will use the default store for its Managed Server and will support persistent messaging. See:

Using Custom File Stores and JDBC Stores

In addition to using the default file store, you can also configure a file store or JDBC store to suit your specific needs. A custom file store, like the default file store, maintains its data in a group of files in a directory. However, you may want to create a custom file store so that the file store's data is persisted to a particular storage device. When configuring a file store directory, the directory must be accessible to the server instance on which the file store is located.

A JDBC store can be configured when a relational database is used for storage. A JDBC store enables you to store persistent messages in a standard JDBC-capable database, which is accessed through a designated JDBC data source. The data is stored in the JDBC store's database table, which has a logical name of WLStore. It is up to the database administrator to configure the database for high availability and performance. See:

Using JMS Paging Stores

Each JMS server implicitly creates a file based paging store. When the WebLogic Server JVM runs low on memory, this store is used to page non-persistent messages as well as persistent messages. Depending on the application, paging stores may generate heavy disk activity.

You can optionally change the directory location and the threshold settings at which paging begins. You can improve performance by locating paging store directories on a local file system, preferably in a temporary directory. Paging store files do not need to be backed up, replicated, or located in universally accessible location as they are automatically repopulated each time a JMS server restarts. See JMS Server Behavior in WebLogic Server 9.x and Later in Configuring and Managing JMS for Oracle WebLogic Server.

Using Diagnostic Stores

The Diagnostics store is a file store that implicitly always uses the Disabled synchronous write policy. It is dedicated to storing WebLogic server diagnostics information. One diagnostic store is configured per WebLogic Server instance. See "Configuring Diagnostic Archives" in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

Best Practices When Using Persistent Stores

For subsystems that share the same server instance, share one store between multiple subsystems rather than using a store per subsystem. Sharing a store is more efficient for the following reasons:

A single store batches concurrent requests into single I/Os which reduces overall disk usage.

Transactions in which only one resource participates are lightweight one-phase transactions. Conversely, transactions in which multiple stores participate become heavier weight two-phase transactions.

For example, configure all SAF agents and JMS servers that run on the same server instance so that they share the same store.

Add a new store only when the old store(s) no longer scale.

Tuning JDBC Stores

The location of the JDBC store DDL that is used to initialize empty stores is now configurable. This simplifies the use of custom DDL for database table creation, which is sometimes used for database specific performance tuning. For information, see "Create JDBC stores" in Oracle WebLogic Server Administration Console Help and "Using the WebLogic Persistent Store" in Configuring Server Environments for Oracle WebLogic Server.

Basic Tuning Information

The following section provides general information on tuning File Stores:

For basic (non-RAID) disk hardware, consider dedicating one disk per file store. A store can operate up to four to five times faster if it does not have to compete with any other store on the disk. Remember to consider the existence of the default file store in addition to each configured store and a JMS paging store for each JMS server.

There are three transactionally safe synchronous write policies: Direct-Write-With-Cache, Direct-Write, and Cache-Flush. Direct-Write-With-Cache is generally the best performer of these policies, Cache-Flush is generally the least performant, and Direct-Write is the default. Unlike other policies, Direct-Write-With-Cache creates cache files in addition to primary files.

The Disabled synchronous write policy is transactionally unsafe. The Disabled write-policy can dramatically improve performance, especially at low client loads. However, it is unsafe because writes become asynchronous and data can be lost in the event of Operating System or power failure.

Certain older versions of Microsoft Windows may incorrectly report storage device synchronous write completion if the Windows default Write Cache Enabled setting is used. This violates the transactional semantics of transactional products (not specific to Oracle), including file stores configured with a Direct-Write (default) or Direct-Write-With-Cache policy, as a system crash or power failure can lead to a loss or a duplication of records/messages. One of the visible symptoms is that this problem may manifest itself in high persistent message/transaction throughput exceeding the physical capabilities of your storage device. You can address the problem by applying a Microsoft supplied patch, disabling the Windows Write Cache Enabled setting, or by using a power-protected storage device. See http://support.microsoft.com/kb/281672 and http://support.microsoft.com/kb/332023.

When performing head-to-head vendor comparisons, make sure all the write policies for the persistent store are equivalent. Some non-WebLogic vendors default to the equivalent of Disabled.

Depending on the synchronous write policy, custom and default stores have a variety of additional tunable attributes that may improve performance. These include CacheDirectory, MaxWindowBufferSize, IOBufferSize, BlockSize, InitialSize, and MaxFileSize. For more information see the JMSFileStoreMBean in the Oracle WebLogic Server MBean Reference.

Note:

.The JMSFileStoreMBean is deprecated, but the individual bean attributes apply to the non-deprecated beans for custom and default file stores.

If disk performance continues to be a bottleneck, consider purchasing disk or RAID controller hardware that has a built-in write-back cache. These caches significantly improve performance by temporarily storing persistent data in volatile memory. To ensure transactionally safe write-back caches, they must be protected against power outages, host machine failure, and operating system failure. Typically, such protection is provided by a battery-backed write-back cache.

Tuning a File Store Direct-Write-With-Cache Policy

For most default and custom file stores, Oracle recommends using the Direct-Write-With-Cache synchronous write policy; it typically provides the best performance while still providing transactionally safe disk writes. It is typically not as performant as the Disabled synchronous write policy, but the Disabled policy is not a safe option for production systems unless you have some means to prevent loss of buffered writes during a system failure.

Direct-Write-With-Cache file stores write synchronously to a primary set of files in the location defined by the Directory attribute of the file store configuration. They also asynchronously write to a corresponding temporary cache file in the location defined by the CacheDirectory attribute of the file store configuration. The cache directory and the primary file serve different purposes and require significantly different locations. In many cases, primary files should be stored in remote storage for high availability, whereas cache files are strictly for performance and not for high availability and can be stored locally.

When the Direct-Write-With-Cache synchronous write policy is selected, there are several additional tuning options that you should consider:

Setting the CacheDirectory. For performance reasons, the cache directory should be located on a local file system. It is placed in the operating system temp directory by default.

Increasing the MaxWindowBufferSize and IOBufferSize attributes. These tune native memory usage of the file store.

Increasing the InitialSize and MaxFileSize tuning attributes. These tune the initial size of a store, and the maximum file size of a particular file in the store respectively.

The JMSFileStoreMBean is deprecated, but the individual bean attributes apply to the non-deprecated beans for custom and default file stores.

It is safe to delete a cache directory while the store is not running, but this may slow down the next store boot. Cache files are re-used to speed up the file store boot and recovery process, but only if the store's host WebLogic server has been shut down cleanly prior to the current boot (not after kill -9, nor after an OS/JVM crash) and there was no off-line change to the primary files (such as a store admin compaction). If the existing cache files cannot be safely used at boot time, they are automatically discarded and new files are created. In addition, a Warning log 280102 is generated. After a migration or failover event, this same Warning message is generated, but can be ignored.

If the a Direct-Write-With-Cache file store fails to load a wlfileio native driver, the synchronous write policy automatically changes to the equivalent of Direct-Write with AvoidDirectIO=true. To view a running custom or default file store's configured and actual synchronous write policy and driver, examine the server log for WL-280008 and WL-280009 messages.

To prevent unused cache files from consuming disk space, test and development environments may need to be modified to periodically delete cache files that are left over from temporarily created domains. In production environments, cache files are managed automatically by the file store.

Tuning the File Store Direct-Write Policy

Deprecation Note:

The AvoidDirectIO properties described in this section are still supported in this release, but have been deprecated as of 11gR1PS2. Use the configurable Direct-Write-With-Cache synchronous write policy as an alternative to the Direct-Write policy.

For file stores with the synchronous write policy of Direct-Write, you may be directed by Oracle Support or a release note to set weblogic.Server options on the command line or start script of the JVM that runs the store:

Globally changes all stores running in the JVM:

-Dweblogic.store.AvoidDirectIO=true

For a single store, where store-name is the name of the store:

-Dweblogic.store.store-name.AvoidDirectIO=true

For the default store, where server-name is the name of the server hosting the store:

-Dweblogic.store._WLS_server-name.AvoidDirectIO=true

Setting AvoidDirectIO on an individual store overrides the setting of the global -Dweblogic.store.AvoidDirectIO option. For example: If you have two stores, A and B, and set the following options:

Setting the AvoidDirectIO option may have performance implications which often can be mitigated using the block size setting described in Tuning the File Store Block Size.

Tuning the File Store Block Size

You may want to tune the file store block size for file stores that are configured with a synchronous write policy of Direct-Write (default), Direct-Write-With-Cache, or Cache-Flush, especially when using Direct-Write with AvoidDirectIO=true as described in Tuning the File Store Direct-Write Policy or for systems with a hard-drive-based write-back cache where you see that performance is limited by physical storage latency.

Consider the following example:

A single WebLogic JMS producer sends persistent messages one by one.

The network overhead is known to be negligible.

The file store's disk drive has a 10,000 RPM rotational rate.

The disk drive has a battery-backed write-back cache.

and the messaging rate is measured at 166 messages per second.

In this example, the low messaging rate matches the disk drive's latency (10,000 RPM / 60 seconds = 166 RPS) even though a much higher rate is expected due to the battery-backed write-back cache. Tuning the store's block size to match the file systems' block size could result in a significant improvement.

In some other cases, tuning the block size may result in marginal or no improvement:

The caches are observed to yield low latency (so the I/O subsystem is not a significant bottleneck).

Write-back caching is not used and performance is limited by larger disk drive latencies.

There may be a trade off between performance and file space when using higher block sizes. Multiple application records are packed into a single block only when they are written concurrently. Consequently, a large block size may cause a significant increase in store file sizes for applications that have little concurrent server activity and produce small records. In this case, one small record is stored per block and the remaining space in each block is unused. As an example, consider a Web Service Reliable Messaging (WS-RM) application with a single producer that sends small 100 byte length messages, where the application is the only active user of the store.

Oracle recommends tuning the store block size to match the block size of the file system that hosts the file store (typically 4096 for most file systems) when this yields a performance improvement. Alternately, tuning the block size to other values (such as paging and cache units) may yield performance gains. If tuning the block size does not yield a performance improvement, Oracle recommends leaving the block size at the default as this helps to minimize use of file system resources.

Setting the Block Size for a File Store

Deprecation Note:

The BlockSize command line properties that are described in this section are still supported in 11gR1PS2, but are deprecated. Oracle recommends using the BlockSize configurable on custom and default file stores instead.

To set the block size of a store, use one of the following properties on the command line or start script of the JVM that runs the store:

Globally sets the block size of all file stores that don't have pre-existing files.

-Dweblogic.store.BlockSize=block-size

Sets the block size for a specific file store that doesn't have pre-existing files.

-Dweblogic.store.store-name.BlockSize=block-size

Sets the block size for the default file store, if the store doesn't have pre-existing files:

-Dweblogic.store._WLS_server-name. BlockSize=block-size

The value used to set the block size is an integer between 512 and 8192 which is automatically rounded down to the nearest power of 2.

Setting BlockSize on an individual store overrides the setting of the global -Dweblogic.store.BlockSize option. For example: If you have two stores, A and B, and set the following options:

-Dweblogic.store.BlockSize=8192
-Dweblogic.store.A.BlockSize=512

then store B has a block size of 8192 and store A has a block size of 512.

Note:

Setting the block size using command line properties only takes effect for file stores that have no pre-existing files. If a store has pre-existing files, the store continues to use the block size that was set when the store was first created.

Determining the File Store Block Size

You can verify a file store's current block size and synchronous write policy by viewing the server log of the server that hosts the store. Search for a "280009" store opened message.

Windows NTFS: run fsutil fsinfo ntfsinfodevice letter: and look for "Bytes Per Cluster"

Converting a Store with Pre-existing Files

If the data in a store's pre-existing files do not need to be preserved, then simply shutdown the host WebLogic Server instance and delete the files to allow the block size to change when the store is restarted. If you need to preserve the data, convert a store with pre-existing files to a different block size by creating a version of the file store with the new block size using the compact command of the command line store administration utility:

File Locking and NFS

Important:

NFS storage may not fully protect transactional data, as it may be configured to silently buffer synchronous write requests in volatile memory. If a file store Directory is located on an NFS mount, and the file store's Synchronous Write Policy is anything other than Disabled, check your NFS implementation and configuration to make sure that it is configured to support synchronous writes. A Disabled synchronous write policy does not perform synchronous writes, but, as a consequence, is generally not transactionally safe. You may detect undesirable buffering of synchronous write requests by observing high persistent message or transaction throughput that exceeds the physical capabilities of your storage device. On the NFS server, check the synchronous write setting of the exported NFS directory hosting your File Store. A SAN based file store, or a JDBC store, may provide an easier solution for safe centralized storage.

Oracle strongly recommends verifying the behavior of a server restart after abrupt machine failures when the JMS messages and transaction logs are stored on an NFS mounted directory. Depending on the NFS implementation, different issues can arise after a failover or restart. The behavior can be verified by abruptly shutting down the node hosting the Web Logic servers while these are running. If the server is configured for server migration, it should be started automatically in the failover node after the corresponding failover period. If not, a manual restart of the WebLogic Server on the same host (after the node has completely rebooted) can be performed. Specifically, if Oracle WebLogic Server does not restart after abrupt machine failure when JMS messages and transaction logs are stored on NFS mounted directory, the following errors may appear in the server log files:

This error is due to the NFS system not releasing the lock on the stores. WebLogic Server maintains locks on files used for storing JMS data and transaction logs to protect from potential data corruption if two instances of the same WebLogic Server are accidentally started. The NFS storage device does not become aware of machine failure in a timely manner and the locks are not released by the storage device. As a result, after abrupt machine failure, followed by a restart, any subsequent attempt by WebLogic Server to acquire locks on the previously locked files may fail. Refer to your storage vendor documentation for additional information on the locking of files stored in NFS mounted directories on the storage device. If it is not reasonably possible to tune locking behavior in your NFS environment, use one of the following two solutions to unlock the logs and data files.

Use one of the following two solutions to unlock the logs and data files:

Solution 1 - Copying Data Files to Remove NFS Locks

Manually unlock the logs and JMS data files and start the servers by creating a copy of the locked persistence store file and using the copy for subsequent operations. To create a copy of the locked persistence store file, rename the file, and then copy it back to its original name. The following sample steps assume that transaction logs are stored in the /shared/tlogs directory and JMS data is stored in the /shared/jms directory.

With this solution, the WebLogic file locking mechanism continues to provide protection from any accidental data corruption if multiple instances of the same servers were accidently started. However, the servers must be restarted manually after abrupt machine failures. File stores will create multiple consecutively numbered .DAT files when they are used to store large amounts of data. All files may need to be copied and renamed when this occurs.

Solution 2 - Disabling File Locks in WebLogic Server File Stores

Note:

With this solution, since the WebLogic Server locking is disabled, automated server restarts and failovers should succeed. Be very cautious, however, when using this option. The WebLogic file locking feature is designed to help prevent severe file corruptions that can occur in undesired concurrency scenarios. If the server using the file store is configured for server migration, always configure the database based leasing option. This enforces additional locking mechanisms using database tables, and prevents automated restart of more than one instance of the same WebLogic Server. Additional procedural precautions must be implemented to avoid any human error and to ensure that one and only one instance of a server is manually started at any give point in time. Similarly, extra precautions must be taken to ensure that no two domains have a store with the same name that references the same directory.

You can also use the WebLogic Server Administration Console to disable WebLogic file locking mechanisms for the default file store, a custom file store, a JMS paging file store, and a Diagnostics file store, as described in the following sections: