Interpreting Message Output in Oracle Real Application Clusters Guard

Oracle Real Application Clusters Guard provides detailed error messages that can help in troubleshooting. Error messages from the Oracle database server and from third-party media vendors also provide useful troubleshooting output. This section contains the following topics:

Contains a chronological log of actions that are relevant to Oracle Real Application Clusters Guard, error messages that are generated by Oracle Real Application Clusters Guard and the Oracle database server, and administrative operations

Interpreting Oracle Real Application Clusters Guard Error Messages

Most of the messages in the log are information for capturing the root cause of a problem the first time it occurs. When a problem occurs, try to identify one or two errors that are most important.

Check for errors that are preceded by Warning. This indicates that a pack or a monitor has incurred a problem but is continuing to operate. It is an indication that actions may need to be taken before an outage occurs. The problem is described in the text of the message.

Check for errors that are preceded by Alert. This indicates the problem that the pack or monitor incurred.

Read the messages in chronological order. The errors before and after the warning or alert are usually the most informative.

Identify the type of error according to Table 8-2. Refer to Appendix A for information about the most important messages.

Example: Interpreting Oracle Real Application Clusters Guard Errors

The following is an example of messages from the Oracle Real Application Clusters Guard log file, pfs_SALES_hostA.log:

Use the pfsboot command to start the packs. The steps of the pfsboot command are as follows:

Check the prerequisites for executing the PFSBOOT command. These conditions cannot exist:

Packs are already running.

The Oracle instance is running outside of the packs.

Failover or restart is occurring.

Start the packs.

If the pfsboot command fails, then check the following items:

Are there errors in the Oracle Real Application Clusters Guard logs?

Are there errors in the alert logs?

Is the cluster up and running?

Is the network operating properly?

Oracle Corporation recommends setting up the call-home function to alert the user when the pfsboot command fails during normal processing.

The Oracle Real Application Clusters Guard logs should clearly describe why the pfsboot command failed. You may need to stop the database manually before reissuing the pfsboot command. The pfsboot command may also fail if the packs are running in foreign mode or if the monitors do not start successfully.

PFS-5074: Alert: System is not clear. Pack PFS_SALES_hostA is running. Use
PFSCTL PFSHALT first.

The message number indicates that the problem is in the PFSCTL command line. The text of the message indicates that the PFS_SALES_hostA pack is already running. Enter the STATUS command to find out the exact state of the packs:

The message numbers are in the 8000 range, so the problem has been reported from the heartbeat monitor. The message text indicates that there is a problem with service registration. The instance failed to register with the listener within 120 seconds.

Check the environment variable and the initialization parameters that affect service registration:

Is the TNS_ADMIN environment variable set correctly in the $PFS_HOME/include/$ORACLE_SERVICE.env file?

Are the following initialization parameters set correctly?

SERVICE_NAMES
ACTIVE_INSTANCE_COUNT
INSTANCE_NAME

Does the LOCAL_LISTENER parameter (for dedicated connections) specify a valid alias in the $TNS_ADMIN/tnsnames.ora file?

Does the LISTENER attribute of the DISPATCHERS parameter (for shared server connections) specify a valid alias in the $TNS_ADMIN/tnsnames.ora file?

For example, suppose a dedicated configuration has the following characteristics:

ORACLE_SERVICE=SALES

The relocatable IP address is 144.28.74

The port is 1524

Suppose that LOCAL_LISTENER is defined in the SALES_config.hostA.ded.pfs file as follows:

LOCAL_LISTENER=listener_SALES_hostA

Then listener_SALES_hostA must be resolved properly in the tnsnames.ora file:

Solution

There are several causes of failed service registration. The best practice is to look for the simplest solutions first. For example, it is common for service registration to fail because the LOCAL_LISTENER parameter is not set correctly. Ensure that the value of the LOCAL_LISTENER parameter in the initialization parameter file (init.ora) matches the entry in the tnsnames.ora file.

If you are not logged on as root, then you see output similar to the following:

PFSCTL for hostA: Version 9.2.0.0- Production on Jan 15 2001 16:49:59
(c) Copyright 2001 , Oracle Corporation. All rights reserved.
Welcome to PFSCTL. Type HELP for additional information.
pfsctl[38]: /home_oracle/901_sales/pfs/bin/PFSCTL.log: cannot create
ORACLE_SERVICE is set to SALES
DB_NAME is set to sales
PFSCTL>

You must be logged on as root.

Are the following environment variables set?

ORACLE_SERVICE
DB_NAME

If ORACLE_SERVICE is not set, then you see output similar to the following:

PFSCTL for hostA:Version 9.2.0.0- Production on Jan 15 2001 16:47:30
(c) Copyright 2001, Oracle Corporation. All rights reserved.
Welcome to PFSCTL. Type HELP for additional information.
Alert: ORACLE_SERVICE is not set. Set it and run PFSCTL again.
If DB_NAME is not set, then you see output similar to the following:
PFSCTL for hostA:Version 9.2.0.0- Production on Jan 15 2001 16:47:30
(c) Copyright 2001, Oracle Corporation. All rights reserved.
Welcome to PFSCTL. Type HELP for additional information.
Alert: DB_NAME is not set. Set it and run PFSCTL again.

Is $ORACLE_HOME/pfs/bin in the PATH variable?

If $ORACLE_HOME/pfs/bin is not in the PATH variable, then you will see output similar to the following:

# pfsctl
pfsctl: command not found

If $ORACLE_HOME/pfs/bin is not in the PATH variable, then either you can execute the PFSCTL command line utility from $ORACLE_HOME/pfs/bin or you can include $ORACLE_HOME/pfs/bin is in the PATH variable.

Troubleshooting the System Outside of the Packs

The packs cannot solve underlying performance or stability problems in the system. If such problems exist, then you must solve them outside of the packs. To troubleshoot outside of the packs, follow these steps: