HP operations manager 9 (HPOM, former OVO) agents

Note: HP renamed the product called now HP operations manager way too many times. Also it is very inconsistent with using abbreviations. Here we will assume that the term "HP Operations manager" and abbreviations HPOM, OMU, and OVO  mean the same thing :-)

HPOM agents are complex (multicomponent) and thus extremely unreliable with some components that tend to fail spontaneously. For each 20 servers with HPOM typically there is at least one failure in 24 hours. That means for 200 servers you have around a dozen of such problem in 24 hours (same days more, some days less).  This is a major problem with HPOM. If agent health is monitored, helpdesk tickets about them became major nuisance for system administrators and source of well-deserved hate of HPOM.

HPOM 9 agents use HTTPS for communication which provide secure communication between agent and the "mothership".  Message format based on XML. Managed nodes can be identified by their unique OvCoreID and not necessarily by their IP addresses.

The architecture of the agent as described in   HTTPS Agent Concepts and Configuration Guide Software Version: 9.01 (Figure 1-3, p. 31) looks pretty similar to classic Tivoli agent.

The first agent installation is the creation of agent on the management server. This way the server becomes the first managed node. After the server is installed and configured,  agents on other servers can be installed from the management server semi-automatically (ssh connection is needed).

The main difference is that Tivoli agent is just one process and HPOM  agent consists of several processes. Also some tasks that in Tivoli were performed by server are delegated to the agent level. In other works HPOM  has more powerful, much more heavyweight agent then Tivoli. Due to that the agent that has potential of causing many troubles during installation (troubles with certificates are pretty common during installation), but after it is installed and configured is runs pretty reliably with almost no other problem then when one of multiple processes die (restart in this case almost always cure the problem and can be done automatically).  Port 383 is used for communication of HTTP agent with the server. 

Like in Tivoli, the HPOM agent by default runs  as root on UNIX, but if necessary can be configured to run under a regular account with less capabilities.

 Running of the daemons is controlled by two RC scripts. On Linux the following startup scripts are used:

• Start Trace Daemon: /etc/rc.d/rc<num>.d/S90OVTrcSrv
• Start HPOM Agent:  /etc/rc.d/rc<num>.d/S92OVCtrl <num> = 3 | 4 | 5
• Stop HPOM Agent:  /etc/rc.d/rc<num>.d/K01OVCtrl
• Stop Trace Server:  /etc/rc.d/rc<num>.d/K02OVTrcSrv <num> = 0 | 1 | 2 | 6

HTTPS agents on Linux kernel 2.6  require the standard C++ library (libstdc++.so).  There are two main versions of HTTPS agents used with HPOM 9:

• Version 8.60. 

• Version 11.0x. Later version substantially different from 8.6. Can be used with SLES 11 SP1 and with some tricks with SLES 11 SP2 (which have linux kernel version 3 that agent does not recognize but that can be fixed by manual modification of one of the script on the installation disk (or ISO file). Assuming that agent ISO disribution is mounted to /tmp/ISO) this is the script:  /tmp/ISO/packages/LIN/Linux2.6_X64/oareqcheck.cfg. See SLES 11 SP2 installation trick for details

Managed nodes must have a valid, industry standard, X509 certificate issued by the HP Certificate Server to be able to communicate with HP Operations management servers. Certificates, signed by 1024 bit keys, are required to identify managed nodes in a managed environment using the Secure Socket Layer (SSL) protocol. The “SSL handshake” between two managed nodes only succeeds if the issuing authority of the certificate presented by the incoming managed node is a trusted authority of the receiving managed node. The main communication security components responsible for creating and managing certificates are:

• HP Certificate Server

• HP Key Store

• HP Certificate Client

Generic Directory Structure on a Managed Nodes

The files associated with the HTTPS agent are found in the following directory structures by default:

1. OvInstallDir This directory (usually /opt/OV/ on Unix, but /usr/lpp/OV/ on AIX) contains static files that are installed from the product media and never change, for example, executables. Since these files never change, you can mount . <InstallDir> as “read-only” for increased security in highly sensitive environments. It is not necessary to back up these files as they can be re-installed from the product media. All other files change during operation and must be backed up regularly.
2. OvDataDir This directory (usually /var/opt/OV on Unix) contains configuration and runtime data files that are used only on the local system.
   • The most important subdirectory in /var/opt/OV contains the instrumentation files such as actions, commands and monitors /var/opt/OV/bin/instrumentation. For example after installation and distribution of Examples the directory looks like:

     -rwxr-xr-x 1 root bin 12966 Sep  3 13:06 cpu_mon.sh*
     -rwxr-xr-x 1 root bin 13216 Sep  3 13:06 disk_mon.sh*
     -rwxr-xr-x 1 root bin 10556 Sep  3 13:06 dist_mon.sh*
     -rwxr-xr-x 1 root bin  9538 Aug  5 13:57 libsis2om.pm*
     -rwxr-xr-x 1 root bin 12611 Sep  3 13:06 mailq_l.sh*
     -rwxr-xr-x 1 root bin 41912 Sep  3 13:06 mondbfile.sh*
     -rwxr-xr-x 1 root bin   352 Aug  5 13:57 OM-SiSAlert*
     -rwxr-xr-x 1 root bin   366 Aug  5 13:57 OM-SiSAlert_full*
     -rwxr-xr-x 1 root bin 12515 Sep  3 13:06 proc_mon.sh*
     -rwxr-xr-x 1 root bin   886 Aug  5 13:57 sis2om_perl.bat*
     -rwxr-xr-x 1 root bin 13995 Aug  5 13:57 sis2om.pl*
     -rwxr-xr-x 1 root bin 96162 Aug  5 13:57 sis2om_samples_95*
     -rwxr-xr-x 1 root bin 14487 Aug  5 13:57 sis2om-setup.pl*
     -rwxr-xr-x 1 root bin   937 Aug  5 13:57 sisconfigdir*
     -rwxr-xr-x 1 root bin  2065 Aug  5 13:57 sis_control.pl*
     -rwxr-xr-x 1 root bin 26062 Aug  5 13:57 sis_disc.pl*
     -rwxr-xr-x 1 root bin 12615 Sep  3 13:06 swap_mon.sh*
     -rwxr-xr-x 1 root bin 27672 Aug  5 13:57 TreePP.pm*
     -rwxr-xr-x 1 root bin 12330 Sep  3 13:06 vp_chk.sh*
     		

   • Examples are provided in

     /var/opt/OV/share/databases/OpC/mgd_node/instrumentation/examples/Unix/Linux

See /opt/OV/OpC/examples/progs/README

HTTPS Communication can be controlled using the following commands.

• ovcoreid (Unique System Identifier) The ovcoreid command is used to display existing OvCoreId value and, in addition, create and set new OvCoreId values on the local system. For details of how to use this tool, refer to the ovcoreid(1) man page.
• ovc (Process Control) ovc controls starting and stopping, event notification, and status reporting of all components registered with the Control service, ovcd . A component can be a server process, an agent (for example, the Performance Agent or the Discovery Agent), an event interceptor, or an application delivered by an integrator. For details of how to use this tool, refer to the ovc(1) man page.
• bbcutil The bbcutil command is used to control the HP Communication Broker. For syntax information and details of how to use this tool, refer to the bbcutil(1) man page. Communication parameters are set in the file: <OVDataDir>/conf/confpar/bbc.ini
• ovconfget Installed HP BTO Software components have associated configuration settings files that contain one or more namespaces and apply system wide or for a specified High Availability Resource Group. A namespace is a group of configuration settings that belong to a component. All configurations specified in the settings files are duplicated in the settings.dat configuration database.

Remote Control of HTTPS Nodes

The opcragt utility is used to control agents from the HP Operations Management server. The operations includes:

• start,
• stop,
• get status,
• primary manager switch,
• get and set configuration variables,
• configuration distribution.

There is a wrapper called opcagt on HTTPS nodes. This utility can be used to perform remote control tasks by application launch from the operator's desktop. It allows to setup a common action definition for any kind of managed nodes. Along with starting and stopping agent they can be used to verify the status of the agent

/opt/OV/bin/OpC/opcagt -status
coda        OV Performance Core                 COREXT       (1603)   Running
opcacta     OVO Action Agent                    AGENT,EA     (1486)   Running
opcmsga     OVO Message Agent                   AGENT,EA     (1487)   Running
opcmsgi     OVO Message Interceptor             AGENT,EA     (1605)   Running

Typical operations

• Listing nodes
• Listing nodes with status
• Adding nodes
• Removing nodes
• Renaming nodes
• Starting and stopping HPOM agent

NEWS CONTENTS

• 20120203 : How to list all nodes their Layout groups and node hierarchies ( How to list all nodes their Layout groups and node hierarchies, Feb 03, 2012 )
• 20101106 : Linux Agent Update results in Failure of ovbbccb -status ( IT Resource Center forums )
• 20101106 : Tips and tricks for HP Operations Manager, including how to ... ( Tips and tricks for HP Operations Manager, including how to ..., )

Old News

[Feb 03, 2012] How to list all nodes their Layout groups and node hierarchies

2011-May-02

The Document KM793358 will show you the list of all nodes in a layout group, however it is required to specify the node hierarchy as a parameter.

If node hierarchy is not specified it takes "Node Bank" as default, but those nodes which are not in the Node Bank are not listed.

Attached is an SQL Query which lists all nodes under all layout groups irrespective of the node hierarchy.

[Nov 06, 2010] Linux Agent Update results in Failure of ovbbccb -status

IT Resource Center forums

Aug 31, 2009

HPOM 8.33 HPUX 11.31 IA Server with latest patches (Server and Agents) effective 2009-08.
Updated Linux agent from
Operations Agent 08.11.000 to
Operations Agent 08.60.005

Just prior to update
ovbbccb -verbose -status omuserver.org
succeeds with no errors.

Post agent install (no install errors reported)
root@agent> /root #
ovbbccb -verbose -status omuserver.org

NOTE: Sending status request to: 'https://omuserver.org:383/
Hewlett-Packard/OpenView/BBC/status/'.

ERROR: (bbc-303) An exception occurred while querying the server
'pluto.dpsk12.org'. Exception message: (xpl-117) Timeout occurred
while waiting for data..

On the omu server executes with no errors:
ovbbccb -verbose -status agent
NOTE: Sending status request to: 'https://agent:383/Hewlett-Packard/OpenView/BBC/status/'.
Status: OK
(Namespace, Port, Bind Address, Open Sockets)
<default> 383 ANY 1
HP OpenView HTTP Communication Incoming Connections
BBC 06.10.205; ovbbccb 06.10.205
10.10.2.43.59629 10.10.10.98.383

From HPOM server to agent:
ovbbccb -ping agent (from omu server):

agent: status=eServiceOK coreID=b0407b96-5198-7517-0e49-8d6c1910f2e7 bbcV=06.20.050 appN=ovbbccb appV=06.20.050
conn=1 time=73 ms

From Agent to Server:
ovbbccb -ping pluto

omuserver: status=eServiceOK coreID=401c626c-6433-7537-14e4-94396352425a
bbcV=06.10.205 appN=ovbbccb appV=06.10.205 conn=39 time=77 ms

RamkumarA

Hi,

Try the following on linux node.

1) Check whether any errors logged in System.txt file
2) try ovc -kill
3) Remove the ovbbccb.dat and queue files.
4) Restart the agents using ovc -start
5) Check ovbbccb -status

Thanks,
Ramkumar A.

jason_lee

Hi SB,
8.60 is a new & different animal. It is a huge "jump" HPOM agents wise. So, I suggest that you read the 8.6 Release notes. I've just have an (unpleasant) experience, where installation of 8.6 failed, but just going back to 8.53, successful without a single problem.

Good luck.

SB:

Ramkumar, thanks for the suggestions...

>>>1) Check whether any errors logged in System.txt file >>>2) try ovc -kill >>>3) Remove the ovbbccb.dat and queue files. >>>4) Restart the agents using ovc -start >>>5) Check ovbbccb -status

I have repeatedly tried ovc -kill and removed the queue files to no avail.

I tried also removing the ovbbccb.dat flie with similar unsucessfull results.

The only thing of interest in the System.txt file is a repeated instance of: "... Unknown monitor 'DBSPI-0088'... " "... Unknown monitor 'DBSPI-0086'... "

I've now determined that this not only applies to linux but HPUX agents updated (at HP Support request mind you) to the latest version 8.60.005. This behavior of breaking things that work just because Support demands you run the latest patches needs to stop. HP needs to provide support for EXISTING releases not simply the most current.

I now will probably have to remove my 8.60.005 patch on the management server and push the older agent 8.53.xx to all nodes.

What a P I T A!!!

SB:

Jason:

As mentioned in my previous reply, the update to the latest patch level was requested by HP support while troubleshooting other issues.

The resulting update causing additional completely unrelated problems is something rather typical of HP. I will likely have to remove the agent patches installed on the HPOM server and the redeploy the agent software to all those ugents recently updated (LINUX and HPUX).

Its experiences like this that make me so darn frustrated at HP.

SB:

Jason: Thanks for the response however, I'm not finding the 8.60 Agent Release Notes. I have read the Patch Description and nothing seems to stand out.

Can you post the release notes or provide a pointer?

SB

Tips and tricks for HP Operations Manager, including how to ...

Recommended Links

OML9.01_Linux_HTTPSAgent.pdf - KM772798
http://support.openview.hp.com/selfsolve/document/KM772798/binary/OML9.01_Linux_H
TTPSAgent.pdf

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019