Oracle Blog

Availability Engineering

Getting the RGM to Step Aside

As mentioned in the discussion of the quiesce command,the RGM aggressively tries to maintain availability of a service. The "ping-pong" scenario was one example of that aggressive behavior, which occurs when a service is mis-configured and unable to start. However, even if a service is configured correctly, you might want to avoid the RGM exercising automated control over it. For example, you might want to switch a resource group offline for awhile, in order to perform some maintenance upon the application. Just switching the resource group offline using a command such as "clresourcegroup offline" (or in the old command line interface, "scswitch -F") does not keep it offline persistently. If a node dies or reboots,or if some other resource group changes state and there are configured resource group affinities or dependencies, then the resource group that you switched offline might automatically come back online again. Kind of like the Energizer bunny.

In Sun Cluster 3.2, we address this requirement by providing commands to suspend and resume automatic recovery actions on a resource group. The "clresourcegroup suspend" command tells the RGM to discontinue all efforts to automatically start, stop, restart, or fail over a specified set of resource groups. The suspend command not only quiesces a resource group, but also shuts off all of the RGM's automated recovery actions on that group. Resources are left in their current state, so if the service is running when you execute the suspend command,it will remain running.

After executing the command "clrg suspend myrg" (note that "clrg" is the short alias for "clresourcegroup"), you will find that the RGM will no longer automatically start or stop the resource group 'myrg'. You can still manually start or stop it, using commands such as "clrg online", "clrg offline", "clresource disable", etc.

If monitoring is enabled on a suspended resource group, the monitor will continue to run; but rather than initiate any restart or failover, it will operate in "log only" mode. You can also disable monitoring using "clresource unmonitor". Or, you can switch a group offline -- persistently -- by executing "clrg suspend" followed by "clrg offline" on that group. You can suspend multiple resource groups at once."clrg suspend +" will suspend all resource groups, effectively halting all RGM-initiated management of data services.

To resume automatic recovery actions on a resource group that has previously been suspended, you execute the "clrg resume" command. Note: If after suspending the resource group you used application-specific commands (rather than Sun Cluster commands) to start the application, then you might have to restart it -- for example, using "clresourcegroup restart" -- to bring it back under Sun Cluster control. However, if you only used Sun Cluster commands such as "clrg offline" and "clrg online" while the resource group was suspended, then you only need to _resume_ the resource group to bring it back under the automated control of Sun Cluster.

The suspend command takes the same -k option as the quiesce command, and it has the same effect. Specifying the -k flag causes the RGM to kill currently running methods to hasten the transition of the resource group to a quiescent state. As in the case of the quiesce -k command, killing a Start method leaves the resource in START_FAILED state, from which it can be switched offline. Killing a Stop method leaves the resource in STOP_FAILED state, which can be cleared by the "clresource clear" or "scswitch -c -f stop_failed" command.

I have configured a 2 nodes Solaris Cluster. Then, we have off line a configured Resource Group on the primary node via 'scswitch -F'. However, it is noticed that the Resource Group fail over to the secondary node after the secondary node is rebooted. So, is that the normal behavior?

In your latest comment, you say that the RG occasionally restarts on the primary node when the secondary reboots. Is the RG initially online on the primary node and restarting (going offline and online) on it? Or is the RG initially offline and goes online on the primary when the secondary reboots?

The former case (RG restarts from being initially online) would be unexpected and erroneous behavior. The second case (coming online on the primary) is valid behavior. There are various factors that could cause the RG to prefer the primary node over the secondary that just booted.

To have more control over this behavior, you should (as noted above) suspend the RG to prevent automated recovery actions by Sun Cluster. Then you can use clrg(1cl) with the switch, offline or online subcommand to specify exactly where to bring the RG offline or online. To restore automated recovery behavior, you'd execute "clrg resume".