Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and laterInformation in this document applies to any platform.

Symptoms

11gR2 Grid Infrastructure node 2 evicted as expected after private network issue, however, after network is restored, ora.cssd and ora.crsd on the evicted node fails to start.

• <GI_HOME>/log/<node>/alert<node>.log - evicted node

2014-07-25 11:12:26.510: [cssd(12284)]CRS-1713:CSSD daemon is started in clustered mode...2014-07-25 11:15:44.348: [cssd(12284)]CRS-1601:CSSD Reconfiguration complete. Active nodes are wvfyapp1 wvfyapp2 .2014-07-25 11:15:45.432: [ohasd(22341)]CRS-2878:Failed to restart resource 'ora.cssd' ######## CSSD started and reconfig complete, but reported CSSD failed to start immediately.2014-07-25 11:15:47.523: [ctssd(13009)]CRS-2403:The Cluster Time Synchronization Service on host wvfyapp2 is in observer mode.2014-07-25 11:15:47.927: [ctssd(13009)]CRS-2407:The new Cluster Time Synchronization Service reference node is host wvfyapp1.2014-07-25 11:15:47.929: [ctssd(13009)]CRS-2401:The Cluster Time Synchronization Service started on host wvfyapp2.2014-07-25 11:15:48.596: [/oracle/app/11.2.0/grid/bin/oraagent.bin(22560)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/oracle/app/11.2.0/grid/log/wvfyapp2/agent/ohasd/oraagent_grid/oraagent_grid.log"2014-07-25 11:15:49.512: [ohasd(22341)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE2014-07-25 11:15:49.512: [ohasd(22341)]CRS-2769:Unable to failover resource 'ora.diskmon'.2014-07-25 11:15:54.417: [/oracle/app/11.2.0/grid/bin/oraagent.bin(22560)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/oracle/app/11.2.0/grid/log/wvfyapp2/agent/ohasd/oraagent_grid/oraagent_grid.log"2014-07-25 11:15:59.046: [ctssd(13009)]CRS-2409:The clock on host wvfyapp2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.2014-07-25 11:16:10.714: [ohasd(22341)]CRS-2878:Failed to restart resource 'ora.crsd'