Today I have faced a problem on my "OCM" environment. Before I went to vacation, everything was working fine. but today, when I was checking the agent after I have started the oms, I found out, that its not working:
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /oracle/product/Middleware/agent11g
Agent binaries : /oracle/product/Middleware/agent11g
Agent Process ID : 31370
Parent Process ID : 31348
Agent URL : https://grid11g.localdomain:3872/emd/main/
Repository URL : https://grid11g.localdomain:4900/em/upload
Started at : 2011-10-13 16:07:56
Started by user : oracle
Last Reload : 2011-10-13 16:10:20
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 219
Size of XML files pending upload(MB) : 13.12
Available disk space on upload filesystem : 41.64%
Last attempted heartbeat to OMS : 2011-10-13 16:10:29
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready
I tryed manual upload:
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.
In the log, I found following error:
Thread-149101456 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"
So I have to refresh the agent.
I have to logon to the grid control and click on setup
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /oracle/product/Middleware/agent11g
Agent binaries : /oracle/product/Middleware/agent11g
Agent Process ID : 31370
Parent Process ID : 31348
Agent URL : https://grid11g.localdomain:3872/emd/main/
Repository URL : https://grid11g.localdomain:4900/em/upload
Started at : 2011-10-13 16:07:56
Started by user : oracle
Last Reload : 2011-10-13 16:10:20
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 219
Size of XML files pending upload(MB) : 13.12
Available disk space on upload filesystem : 41.64%
Last attempted heartbeat to OMS : 2011-10-13 16:10:29
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready
I tryed manual upload:
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.
In the log, I found following error:
Thread-149101456 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"
So I have to refresh the agent.
I have to logon to the grid control and click on setup
On the next scree, I have chosen the "Agents" from the menu.
There, I can see, taht the agent is blocked. See the green check in the blocked column. I have clicked on the affected agent (grid11g.localdomain)
Here, I can see the error message. The agent needs to by resychronized. So i clicked the "Agent Resynchronization" button.
On the next screen, I have chosen the "Continue" button.
After the agent was resynchronized, everything was working fine again:
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /oracle/product/Middleware/agent11g
Agent binaries : /oracle/product/Middleware/agent11g
Agent Process ID : 31370
Parent Process ID : 31348
Agent URL : https://grid11g.localdomain:3872/emd/main/
Repository URL : https://grid11g.localdomain:4900/em/upload
Started at : 2011-10-13 16:07:56
Started by user : oracle
Last Reload : 2011-10-13 16:26:13
Last successful upload : 2011-10-13 16:30:21
Total Megabytes of XML files uploaded so far : 1.11
Number of XML files pending upload : 2
Size of XML files pending upload(MB) : 0.02
Available disk space on upload filesystem : 41.66%
Last successful heartbeat to OMS : 2011-10-13 16:30:14
---------------------------------------------------------------
Agent is Running and Ready
After the agent was resynchronized, everything was working fine again:
[oracle@grid11g log]$ /oracle/product/Middleware/agent11g/bin/emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /oracle/product/Middleware/agent11g
Agent binaries : /oracle/product/Middleware/agent11g
Agent Process ID : 31370
Parent Process ID : 31348
Agent URL : https://grid11g.localdomain:3872/emd/main/
Repository URL : https://grid11g.localdomain:4900/em/upload
Started at : 2011-10-13 16:07:56
Started by user : oracle
Last Reload : 2011-10-13 16:26:13
Last successful upload : 2011-10-13 16:30:21
Total Megabytes of XML files uploaded so far : 1.11
Number of XML files pending upload : 2
Size of XML files pending upload(MB) : 0.02
Available disk space on upload filesystem : 41.66%
Last successful heartbeat to OMS : 2011-10-13 16:30:14
---------------------------------------------------------------
Agent is Running and Ready