Recovering a lost Data Guard broker configuration file (ORA-16572)

While teaching the Oracle11g Release 2 Data Guard course a while back I got a question about how to recover from the loss of a Data Guard broker configuration file. I didn’t know the answer right away so I did a bit of research and this is what I found out.

Basically there are 4 different recovery scenarios possible, listed by level of difficulty starting with the easiest one:

Recovering a lost broker configuration file on a standby database while it is still running

Recovering a lost broker configuration file on a standby database while it is not running

Recovering a lost broker configuration file on the primary database while it is still running

Recovering a lost broker configuration file on the primary database while it is not running

Before explaining how to recover a lost Data Guard broker configuration file, let’s explore my Data Guard configuration and take a look at the size and location of the actual broker configuration files itself.

As shown in the output above, my Data Guard configuration is called “PeppiEnKokki”, named after a famous Dutch children TV series. The primary database is named “peppi” and is hosted on a system with a hostname “el4:, and there is one physical standby database named “kokki” which is running on a system with the hostname “el5”. Furthermore the output shows that each database in the Data Guard configuration has two broker configuration files which are store in their $ORACLE_HOME/dbs directory. The reason behind two configuration files is that one of them stores the current configuration while the other stores the previous version of the configuration. When the configuration gets updated the new configuration gets written to the “oldest” configuration file and once this update is complete the broker switches from the current file to the updated file.

Now that my Data Guard setup is clear we can proceed with the 4 different recovery scenario’s.

Scenario #1: Recovering a lost broker configuration file on a standby database while it is still running

The first scenario deals with the loss of either one or both broker configuration files on a standby database while the standby database is still running. Before explaining how to recover this we first need to create the situation where there is actually something to recover ;-) This is easily achieved by brutally deleting both broker configuration files on host “el5”.

The broker clearly does not recognize the fact that the configuration files are missing. This is more or less true because the broker keeps the active configuration loaded in memory. However, if the standby database needs to be restarted a problem will show up as we will see in the next scenario. The question for now is: How do we recover the lost broker configuration files?

The answer is we don’t have to because the broker will recover from this error automatically without user intervention! Every time the broker configuration gets updated the new configuration will be written from memory to the inactive configuration file on disk. Thus the lost file will be recovered automatically upon the next change in the broker’s configuration.

If we want to trigger this process by hand we can simply re-enable the standby database in question as shown below:

We see that one broker configuration file has been re-created because we re-enabled the standby database. By re-enable the standby database another time the other broker configuration file will be re-created as well.

Scenario #2: Recovering a lost broker configuration file on a standby database while it is not running

The second scenario deals with the loss of the active broker configuration file while the standby database is not running. Again we start our journey by first creating the actual problem by shutting down the standby database instance followed by removing both broker configuration files as shown below:

YES! The broker knows that its configuration files are indeed missing and it starts recovery without user intervention. Behind the scenes it asks the primary database for the current configuration and writes it to disk. If we wait a while we can see that this is indeed the case:

As shown above the broker recovered one of the configuration files automatically and it reports that the overall status of the configuration is fine. Upon the next change in the configuration the other file will be recovered as well. If we want to trigger this process by hand we can simply re-enable the standby database just as we did in scenario #1.

Now that both files are recovered we can proceed to the next scenario.

Scenario #3: Recovering a lost broker configuration file on the primary database while it is still running

This scenario deals with the loss of either one or both broker configuration files on the primary database while the primary database is still running. This scenario is pretty much the same as the first scenario. As before we begin setting up the problem by removing the broker configuration files from disk.

The broker is not aware of the missing files and reports a successful overall status. Just like in the first scenario the broker will re-create the missing configuration files upon a configuration change by writing the new configuration to disk thereby in effect performing recovery. Again this recovery can be triggered by re-enable the primary database if we have the desire to do so.

They are now both recovered and we will move on to our final scenario.

Scenario #4: Recovering a lost broker configuration file on the primary database while it is not running

The final scenario deals with the loss of the active broker configuration file on the primary database while the primary database is not running. Again we start by knocking down the primary database instance followed by removing the broker configuration files from disk.

The primary database instance starts up normally and doesn’t complain about the loss of the broker configuration files. One might expect that the broker will recover them automatically, just as it did on the standby database, but this is simply not possible on the primary database. Even if we wait a long time, the broker configuration files will not show up automatically.

It is clear that the broker didn’t recover the lost broker configuration files automatically, and the question arises: Is the broker actually aware that its configuration files are indeed gone? Let’s see what the broker has to say:

Hmmm, the broker reports that there is no configuration at all! Is this really true or is the broker just incapable to distinguish between a non-existing configuration or the lost configuration? Examining the broker’s log file seems like a good thing to do in order to determine what is really going on here.

$ oerr ora 16572
16572, 00000, "Data Guard configuration file not found"
// *Cause: The Data Guard broker configuration file was either unavailable or
// did not exist.
// *Action: Verify that the configuration file was successfully created.
// If the DG_BROKER_CONFIG_FILE[1|2] initialization parameters were
// changed, ensure the file name on disk and the parameter value
// match, there is space on the device, and the proper permissions
// are granted. For a RAC database, ensure that these initialization
// parameters are set to file locations that are shared by all
// instances of the RAC database.

The above makes clear that the broker configuration files are either not there or they can’t be accessed. We already know that they are gone, because we removed them ourselves, so there is no need to verify this. The question is of course: How do we recover the lost broker configuration files?

The solution is either recreating the whole broker configuration from scratch or to copy a broker configuration file from one of its standby databases to the primary database. I’ll go for the second solution as shown below. It’s important to perform this action while the broker itself is not running.

After this step we need to give the broker a bit of time so it can do whatever it needs to do. After a while we can take a look at the broker configuration files and verify the overall configuration status.

Everything is fine now and it is about time to wrap up and summarize our findings.

In summary

In most situation recovering lost Data Guard broker configuration files is handled automatically by the broker itself, but can be manually triggered by forcing an update to the broker configuration. Only in case of the loss of all broker configuration files on the primary database user intervention is required. All we need to do is copy a broker configuration file from one of its standby databases to the primary database followed by re-enabling any disabled databases.-Harald

Like this:

LikeLoading...

Related

This entry was posted on October 26, 2010 at 22:09 and is filed under Oracle.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.