Solution

Reviewing the emagent.trc and emdctl.trc files did not
provide any useful trace infomation to diagnose the problem in this case.
I did find several notes on Metalink regarding this error, however, none
of the solutions applied to my configuration.

I then decided to check the dbsnmp login credentials to the
OEM Grid Control Repository. The database password in this case was valid, however
the password was about to expire:

2010-05-10 12:28:56,007 Thread-149101456 ERROR pingManager: Did not receive valid
response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync
with repository. This most likely means that the agent was reinstalled or recovered.
Please contact an EM administrator to unblock the agent by performing an agent
resync from the console. Please contact EM adminstrator to unblock the agent"

Cause

The Agent Re-synchronization is a new feature in 10.2.0.5 Grid Control
which verifies whether an Agent that been uploading earlier to the OMS is
re-installed or restored from a backup. If yes, the OMS blocks further updates
from this Agent until the information about the Agent and all its targets are
synchronized between the Repository and the Agent.

Factors which can cause the Agent to go out-of-sync with the Repository:

Agent is re-installed or restored from a backup, on the same port number as before.

The <AGENT_HOME>/sysman/emd/agntstmp.txt is manually deleted for some reason.

Solution

To solve the issue you have to perform following from the OEM grid control console:

Login to 10.2.0.5 Grid Console as sysman user

Navigate to Setup > Agents > [Click on problematic Agent name].

In Agent home page click on "Resycnhronize" Button
on right hand top of the page. Choose the 'Unblock Agent'
option and click [Continue].

Ensure that the Agent is up and running when attempting the Re-synchronization.

Once the re-sycnchronization is completed successfully,
the Agent should be able to communicate with the OMS:

Re-synchronization Failure

It is possible for the re-synchornization operation to complete with
the following error:

Agent Operation completed with errors. For those targets that
could not be saved, please go to the target's monitoring
configuration page to save them. All other targets have
been saved successfully. Agent has not been unblocked.

The above indicates that the Repository has been unable to completely save
the details of the all targets that the Agent currently has. In this case,
the Agent was re-installed and has newly discovered all the Database targets
and iAS targets, which are not completely configured as yet (i.e the monitoring
password is missing).

To resolve the above:

Navigate to the Agent home page in the Grid Console (Setup > Agents).
Choose one database at a time and click on the [Configure] button.

In the "Monitoring Configuration" page, enter the password for the monitoring
user (the dbsnmp user) and save.

Perform the same for any other target that is shown in the result from the
Agent re-synchronization.

Once, all the targets are configured in this manner go to:
Setup > Agents > [choose this Agent name] and
click on the [Unblock] button.

If you're like me, you may opt to keep the CRS Cluster Name set to its default
value of "crs". Even when configuring a second or third Oracle RAC within
an organization, we many times simply leave the cluster name for the new
clusters set to their default value of crs. Although I believe it is bad
practice to configure multiple Oracle RAC clusters with the same CRS cluster name, it
doesn't necessarily cause any conflicts given these clusters don't interact
with each other. The clustered databases work independently from each other
without incident. That is, until you register multiple clustered databases with the
same CRS cluster name in Oracle Enterprise Manager Grid Control!

An issue can arise when multiple Oracle RAC clusters with the same CRS cluster name
are registered as targets in EM Grid Control. While not causing a problem with the
cluster itself, it does cause EM Grid Control to think both clusters are the same.

The following article will explain the steps used to change the EM cluster target name(s)
in Grid Control from their default name of "crs" to avoid conflict with other
cluster targets.
In this article, a new EM cluster target name will be created in EM Grid Control
for two Oracle RAC database environments that were registered in EM Grid Control with
the same EM cluster target name.

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author,
and an Oracle ACE.
Jeff currently works as a Senior Database Administrator for
The DBA Zone, Inc. located in Pittsburgh, Pennsylvania.
His work includes advanced performance tuning, Java and PL/SQL programming, developing
high availability solutions, capacity
planning, database security, and physical / logical database design in a UNIX /
Linux server environment. Jeff's other interests include mathematical
encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters)
in Java and C, LDAP, writing web-based database administration tools, and of
course Linux. He has been a Sr. Database Administrator and Software Engineer
for over 20 years and maintains his own website site at:
http://www.iDevelopment.info.
Jeff graduated from Stanislaus State University in Turlock,
California, with a Bachelor's degree in Computer Science and Mathematics.

Copyright (c) 1998-2017 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter
and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express,
prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate,
but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from
relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.