May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Troubleshooting HPOM agents

News Agents Recommended Links HP HPOM Reference itochecker HPOM Health Check Tracing
opcagt opcragt ovc ovcert bbcutil Unix System Monitoring Shadow IT
Changing OVcoreid of the node due to mismatch Certificate server is not set Changing coreid of the node due to mismatch Undefined certificate state opcmsgm message manager aborted Message from opcmsg not reaching HPOM console Error opc_dflt_lang
Starting and stopping HPOM agent Troubleshooting of communication problems with the agent Deinstallation of agent Agent patching Change of IP for the Agent Multihomed Agenets Policies troubleshooting
Unix System Monitoring Unix Configuration Management Tools Enterprise Job schedulers Sysadmin Horror Stories Support Forum Humor  

Note: HP renamed the product called now HP operations manager way too many times. Also it is very inconsistent with using abbreviations. Here we will assume that the term "HP Operations manager" and abbreviations HPOM, OMU, and OVO  mean the same thing :-)


HPOM agents are complex (multicomponent) and thus extremely unreliable with some components that tend to fail spontaneously. For each 20 servers with HPOM typically there is at least one failure in 24 hours. That means for 200 servers you might well around a dozen of such problem in 24 hours (some days more, some days less).  This is a major problem with HPOM.

If agent health is monitored in a primitive way -- by creating a ticket for each failure, such helpdesk tickets about them became major nuisance for system administrators and source of well-deserved hate of HPOM, and, especially, people in the organization who are responsible for it.  BTW minor problems usually can be fixed by just restarting the agent. If this is not done automatically via a cron script, your HPOM people are extremely stupid or lazy or both. And the same is true for you  :-).

This monitoring script should have a "map of running components" stored as a file and compare it with the current "map" on each invocation (say, once an hour). If there is a difference it needs to restart the agent and compare again. If difference persists,  than it should write a "do not do it again" file that will prevent this operation from repeating indefinitely on the next invocation  and many be send email that the agent needs manual intervention to  the primary administrator of the server. This "do not do this again" file can expire in 24 hours (you can delete it via find is it is older then one day).  Such a script can be implemented in less then a hundred lines of Perl, or bash code or can be implmented as a part private "parralel" monitoring system that admistor uses for his servers ("Shadow IT" solution ;-) .  

Agents are extremely unreliable with some components that tend to fail spontaneously. Minor problems usually can be fixed by just restarting the agent. If this is not done automatically via a cron script

In order properly troubleshoot the agent one needs to understand the agent architecture. See HPOM Agent Architecture. You chances to find problem are higher is you have a snapshot of the "working" configuration and list of processes as in large organization there might be difference version of agents installed on different servers.  

If your agent was recently switched from one management server to another it is important to establish to which manager it is now reporting. You can do it with the command:

 /opt/OV/bin/ovconfget | grep MANAGER
For example:
/opt/OV/bin/ovconfget | grep MANAGER

If this is correct OV manager server then the next thing to check  is whether the agent can communicate with the management server
          status=eServiceOK coreID=b90b1206-cab9-7557-1510-990e458abb81
          bbcV=11.05.005 appN=ovbbccb appV=11.05.005 conn=131 time=564 ms

Creating a snapshot of the current configuration

Having a snapshot of a working configuration greatly simplifies troubleshooting.  For each agent you can get a snapshot on installed components using the commands

PATH=$PATH:/opt/OV/bin # for AIX PATH=$PATH:/usr/lpp/OV/bin
ovc -status
opcagt -status # see manpage for opcagt

After that you need to create a tar of relevant directories such as /opt/OV, /var/opt/OV, /etc/OV.  Difference of the current directory content with that tar of "working" configuration often suggest areas where to search the problems.  Pkease note that GNU diff can compare two different directories directly if you put them as arguments to the command.

The following three commands provide a snapshot of the configuration of the  management server

ovcoreid -show
ovcert -list
opcragt <node name>

Troubleshooting of communication problems with the agent

The most typical source of troubles with the HPOM agent are communication problems. They happen probably less often the "spontanious" crashes of some agent-related processes, but they need to be checked first.  Among them most common reasons: 

  1. Linux firewall is enabled on agent host, but port 383 is not opened.
    In this case attempt to get status of the agent from the HPOM server with the command

    opcragt -status

    produces the following error: 

    Cannot get status information from node  (OpC40-428)
    Network communication problems occurred. (OpC40-427)
     CTRL - CommunicationException:
    (ctrl-21) Communication error when executing 'Status' method.
     (bbc-42) Unable to connect to the OV Communication Broker. The connection error returned was: 
      (xpl-316) connect() to '' failed.
       (RTL-239) Connection refused (OpC40-2130)

    Connection refused above suggests that we are dealing with the firewall problem.

    NOTE:  itochecker  version 09.00.160 diagnoses this error incorrectly as certificate error:

    Nodename             Node state Agent State Cert. State Config State Overall  OK         ERROR       ERROR       OK           ERROR	
  2. Check if NTP is working correctly on server and agent. HPOM will not function properly if clocks are not in sync and, for example, messages are coming "from the future".
  3. Check if IP is correct or the node was moved. Also check /etc/hosts file if it contains and entry for the node. In  case the node was moved and IP changed, but /etc/hosts was not corrected this is a difficult to find bug. It is also very typical. 
  4. Check whether the node exists on the server and is assigned to any node group.  Check if IP is correct (it is editable) Check if core ID on the node record on the server is the same as core ID reported by agent. See Changing OVcoreid of the node due to mismatch
  5. Run the following commands from management server and save output: 

    Compare with the output from other similar working node or, better, with your snapshot.

  6. From Managed node execute the following:

Sometimes version of the agent is just too old. To get a version of the agent from HPOM server use the command:

opcragt -agent_version <node>

There are some cases when the node and the server can't communicate with each other. HPOM uses name resolution services (DNS and/or a host file) and that can get wrong too. Here are some potentially useful questions to start with:

Communication Broker problems checklist

  1. On all systems, the Operations Manager Server, Certificate Authority server and HTTPS managed node, check if the Communication Broker ovbbccb is running. Use the following commands:
    ovc -status

    The ovbbccb process must be listed as running. The output takes the following form:

    ovc -status
    coda        OV Performance Core                 COREXT       (835828) Running
    opcacta     OVO Action Agent                    AGENT,EA     (925700) Running
    opcmona     OVO Monitor Agent                   AGENT,EA     (491590) Running
    opcmsga     OVO Message Agent                   AGENT,EA     (798748) Running
    opcmsgi     OVO Message Interceptor             AGENT,EA     (495654) Running
    ovbbccb     OV Communication Broker             CORE         (1081380) Running
    ovcd        OV Control                          CORE         (614606) Running
    ovconfd     OV Config and Deploy                COREXT       (860202) Running
  2. Then the command
    show system/proc="ovbbccb"

    ovbbccb must be listed.

  3. Now use command
    bbcutil -status

    Status of ovbbccb must be ok. For example

    bbcutil -status
    NOTE:    Sending status request to: 'localhost'.
    (Namespace, Port, Bind Address, Open Sockets)
          383    ANY    2
    HP OpenView HTTP Communication Incoming Connections
      BBC 06.21.501; bbcutil unknown version
      BBC 06.10.251; bbc.http.ext.ctrl 00.00.000

    If non standard ports are used make a note of the ports listed using the command:

    bbcutil -getcbport <hostname>

    • on HTTPS managed node as <AGENT-PORT>

    • on Operations Manager Server as <MGMT-SRV-PORT>

    • on Certificate Authority server as <CA-SRV-PORT>

    Alternatively, you can use the command:

    ovconfget bbc.cb.ports PORT

    Installed OpenView components have associated configuration settings files that contain one or more namespaces. A namespace is a group of configuration settings that belong to a component. All configurations specified in the settings files are duplicated in the settings.dat configuration database. For each specified namespace, ovconfget returns the specified attribute or attributes and writes them to stdout. Used without arguments, ovconfget writes all attributes in all namespaces to stdout. For details of how to use this tool, use ovconfget -h.

    COPY=scp  @:
    EXEC=ssh -q -2 @ 
    OPC_INSTALLATION_TIME=Tue Jun 10 14:40:00 EDT 2014
    timestamp=Wed Jan 12 11:53:20 2011
    LAST_CERT_UPDATE=Tue Jun 10 14:41:46 2014
    LAST_TRUSTED_CERT_UPDATE=Tue Jun 10 14:41:46 2014
    ... ... ...
  4.  If communication broker is not running you can start it with the command:
    ovc -start ovbbccb

    No error messages should be displayed.

Restarting failed agent process

Following are descriptions of the HTTPS agent processes:

The most common process the "accidentally dies" is coda. Depending on the status on communications broker you can start it with the following commands:

  1. If communication broker is not running you need to execute command on the node (or using ssh from from the other server):
    ovc -start coda
  2. If communication broker is running  you can do it on the HPOM server with the command:
    # opcragt -status
    Cannot get status information from node  (OpC40-428)
    Network communication problems occurred. (OpC40-427)
     CTRL - CommunicationException:
    (ctrl-21) Communication error when executing 'Status' method.
     (bbc-42) Unable to connect to the OV Communication Broker. The connection error returned was: 
      (xpl-316) connect() to '' failed.
       (RTL-239) Connection refused (OpC40-2130)

Certificate related problems

Those are typical after the installation of the agent. Rarely happen with working agent. See Certificates management

Key commands  used in troubleshooting

BBCUTIL - used to test HTTPS communications between managed node and management server.

If there are any problems in the area of HTTPS communication, the bbcutil command can be used to test
basic communication and verify whether RPC servers have registered correctly.

A common test is by using bbcutil -ping, as shown from the following example output:

C:\Program Files\HP OpenView\data\bin\instrumentation>bbcutil -ping

localhost: status=eServiceOK coreID=079f1de2-a609-7512-13c2-d6c7194b9864
bbcV=05.20.010 appN=ovbbccb appV=05.20.010 conn=1 time=156 ms

Output of BBCUTIL help

C:\Documents and Settings\root>bbcutil -help

Usage: bbcutil [-count|-size|-list] -path [-t|-target ]
bbcutil -deregister {|*} [-force]
bbcutil -migrate {[namespace] [appname] [filename]}
bbcutil -getcbport []
bbcutil -gettarget []
bbcutil -ping {[|[:]] | []} [count]
bbcutil {-reg|-registrations} [|]
bbcutil -status {[|[:]] | [] [status|version|
bbcutil [-v|-verbose]
bbcutil [-h|-help]
bbcutil [-ovrg ]
bbcutil [-version]

HP OpenView BBC Utility Program.
The following options are recognized by bbcutil.

Upon completion, bbcutil returns one of the following values:

OPCAGT - command used to check current status of OVO Agents on managed node (for backwards compatability with previous OVO versions)

See also opcragt

Output of OPCAGT help

C:\Documents and Settings\root>opcagt -help
Usage: opcagt [ -help| -h | -stop | -kill | -start | -restart | -status | -type
[verbose] ]
-help Prints this message.
-stop Stop Event/Action Agent and Coda.
-kill Stop OV Core functionality.
-start Start Event/Action Agent and Coda.
-restart Restart OV Core functionality.
-status Show status of Event/Action Agent and Coda.
-type Show type of Event/Action Agent.
-verbose Show detailed info of Event/Action Agent.
-version Show version information.

Example usage (checking status of the agents):

C:\Documents and Settings\root>opcagt -status
opcacta OVO Action Agent AGENT,EA (3772) Running
coda OV Performance Core AGENT,CODA (3788) Running
opcmsgi OVO Message Interceptor AGENT,EA (3820) Running
opcle OVO Logfile Encapsulator AGENT,EA (3896) Running
opcmona OVO Monitor Agent AGENT,EA (4016) Running
opcmsga OVO Message Agent AGENT,EA (3836) Running

OVC - command used to check current status of OVO8 Agents on managed node (similar to opcagt command)

Output of OVC help

C:\Documents and Settings\root>ovc -help
(ctrl-76) Usage: ovc -help|-h
-start [ ... ] [-boot]
-stop [ ... ] [-nostart]
-restart [ ... ]
-status [ ... ] [-level ]
-trace [ ... ]
-notify [ ... ] [-value ]

Example usage (checking status of the agents):

C:\Documents and Settings\root>ovc -status
ovcd OV Control CORE (1204) Running
opcacta OVO Action Agent AGENT,EA (3772) Running
coda OV Performance Core AGENT,CODA (3788) Running
opcmsgi OVO Message Interceptor AGENT,EA (3820) Running
opcle OVO Logfile Encapsulator AGENT,EA (3896) Running
opcmona OVO Monitor Agent AGENT,EA (4016) Running
opcmsga OVO Message Agent AGENT,EA (3836) Running
ovconfd OV Config and Deploy COREXT (3848) Running
ovbbccb OV Communication Broker CORE (3296) Running

OVCERT - Command to check, request, or modify the certificate on the managed node

Output of OVCERT help

C:\Program Files\HP OpenView\data\bin\instrumentation>ovcert -h
Usage: ovcert -importcert -file [-pass ]
[-ovrg ]
ovcert -exportcert -file [-alias ] [-pass ]
[-ovrg ]
ovcert -importtrusted -file [-ovrg ]
ovcert -exporttrusted -file [-alias ]
[-ovrg ]
ovcert -certreq [-instkey [-pass ]]
ovcert -list [-ovrg ]
ovcert -remove [-f] [-ovrg ]
ovcert -certinfo [-ovrg ]
ovcert -check [-ovrg ]
ovcert -status
ovcert -updatetrusted
ovcert -version
ovcert [-h|-help]

Example usage (checking the status of the certificate):

C:\Program Files\HP OpenView\data\bin\instrumentation>ovcert -status
Status: Certificate is installed.

Top Visited
Past week
Past month


Old News

[Dec 01, 2010] OVO Agent Error in UNIX Servers

Feb 13, 2006 | IT Resource Center forums

Ujjwal Gupta:


I am using OVOU8.x environment. When i have already installed agent in UNIX/LINUX/Solaris server. Now Management server not communicating with some servers. When i check these servers with help of using commands i.e :-

opcagt -status

Then it gives error just like as:-

" Network Communication problem occurred during distribution to node <Node Name>. "

Please suggest me, How do remove this error.

Reymond Keiluhu:

did you execute that command on managed node? why is the error about distribution? https agent or dce agent? have you verified all the required ports between mgmt server and managed nodes are ok?

Ujjwal Gupta:


Thanks for immediate response. I am using https agent. I executed this command on Management Node. it runs sucessfully. Yah, I have checked all the ports which are required for mgmt server & mgmt node.

Any another way to resolve this issue.

Reymond Keiluhu:

what is the output of this command:


Hi Ujjwal,

Please try the following from management server

  1. bbcutil -ping <managed nodename>
  2. opcragt -status <managed nodename>
  3. telnet <Managed node IP> 383 (i.e., default https port used by OVO 8.x)
  4. opcsv -status

From Managed node execute the following

Even after checking these conditions, check whether the node is assigned to any node group.

Still if you are facing the same problem, remove and re-install the agent on managed node.

Regards, Ravindra

Ujjwal Gupta:

Hi Ravindra,

Thanks for reply. I check all the command which you have send me in rply.

One of them in Magement Server when i run the command i.e opcragt <Node Name>

It gives the error " Network Communication problem occurred during distribution to node <Node Name>. "

Please suggest me how do resolve it


Shivam Tiwari:

Hi Ujjwal,

To start with

Just check the output of this from Management Server

#bbcutil -ping http://FQDNOFNODE

and check that the COREID is same as what known to the management server by

#opcnode -list_id node_list=FQDNOFNODE

If they are not same then change the id of the node on the management server

On the Node do

#bbcutil -ping https://FQDNOFSERVER

If this is OK then the communication is OK otherwise there is some network routing issues.


Shivam Tiwari


Try with reinstalling the agent. First remove the node from node bank. it asks for automatic deinstalling of agent say "yes". once agent is uninstalled and node is not visible in node bank, restart the GUI. then discover node once again and install the agent automatically. You need the root password of managed node to install the agent. because before it is ftp'ing the packages, it asks for the password.

Regards, Ravindra

Shivam Tiwari :

I believe reinstalling is the last step to perform as this doesnt give us the oppurtunity to troublehshoot the root cause of the Incident.


if the bbcutil -ping http:// is working both the ways then try for bbcutil -ping https://FQDN both the ways to check the Secure Communication is OK.

Id it is then delete all the certificate on the node and issue a new certificate request from the node by #ovcert -certreq

grant the certificate from management server and hope this will reolve your issue.


Shivam Tiwari

IT Resource Center forums - HP Operations HTTPS agent processes aborted


I believe that from agent (installation and processes related to agent) perspective the Linux nodes will be just that - no difference whethere it is in Oracle RAC.

The log file you need to look for is the System.txt file - /var/opt/OV/log/System.txt file.

Hope this helps.

Nov 4, 2010 21:35:45 GMT N/A: Question Author


thanks for your reply,
regarding the processes aborted problem, i have been searching in other threads and almost all recommed this procedure:

1. ovc -kill
2. make sure all corresponding services (OvCtrl, HPOvTrcSvc) and processes (ov* opc* ,coda, agtrep) are shut;
3. remove all agt queues & buffers in
4. ovc -start

I already did it with no success, the problem persists:

ovc -start
(ctrl-7) Error in the target component.
ovc -status
ovcd OV Control CORE (5153) Running
ovbbccb OV Communication Broker CORE (5178) Running
agtrep OV Discovery Agent AGENT,AgtRep (5917) Running
opcle OVO Logfile Encapsulator AGENT,EA Aborted
opcacta OVO Action Agent AGENT,EA Aborted
opcmsga OVO Message Agent AGENT,EA (5648) Running
coda OV Performance Core COREXT (18011) Running
opcmona OVO Monitor Agent AGENT,EA Aborted
ovconfd OV Config and Deploy COREXT (5215) Running
opcmsgi OVO Message Interceptor AGENT,EA Aborted

Q: is there any other way to fix this before reinstall the agent?

best regards.



Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July 28, 2019