HPOM Troubleshooting

Note: HP renamed the product called now HP operations manager way too many times. Also it is very inconsistent with using abbreviations. Here we will assume that the term "HP Operations manager" and abbreviations HPOM, OMU, and OVO  mean the same thing :-)

Troubleshooting, problem analysis and root cause determination requires patience, determination, and experience. Even if you just deleted important directory you should rush to action. It is important to analyze the problem and see available options in order to find the correct path. Although you may start out on one path and end up on another in order to resolve a complex problem, over time the most important skill will become patience.  You need to deliberately slow down yourself, maintain log of your actions and think before jumping to action. In may cases haste really bring waist and one hour problem became 10 hour problems due to some badly thought out "recovery" action.

You can benefit from using HP Support Forum. There is also access to online HP OpenView technical support

When an event occurs that cause system or application downtime, this downtime already started often additional 20-30 min does not change anything. So the number one priority is forget idea of getting it back as quickly as possible and work as reliably and safely as possible. That means making intermediate backups as you progress.  

Each event can be analyzed against the baseline of working system. In this sense detecting configuration or system changes from the known good configuration is an important step that prevent hasty action and likely force you to tread with a more cautious mindset.  Production environment is often stupidly stable and differences that you might find can be the key to detecting what coused a problem. Three major sources of problems are

• yourself or other admins with root access.
• Patches of operating system or HPOM itself
• Networking

Helpdesk support from HP is critical for serious problems as you by definiton can't know the HPOM to the depth that developers in HP know. Collaboration with colleagues is important when the issue falls outside the realm of familiar territory. In this sense excluding possibility of actions of other sysadmins is important first step. It never hurts to ask a question or two of those who might have seen a similar situation and have more current experience. Finally, bulldog style determination is the final essential quality for someone facing the HPOM troubleshooting task, especially if the situation is unfamiliar.

There are multiple tools built into the HPOM user environment that might help you solve the problem. They make possible to use structured approach to determination of the root cause of the problem and components involved.  The HPOM structurally be viewed as several subsystems each of which can be investigated independently:

1. Networking and Client/Server Communications

2. Database(s) and Event and Message processing

3. Policies

4. Server Configuration

5. Applications (out-of-the-box and add-on)

6. Agents  and their components (smart Plug-Ins)

7. Security (built-in and add-on)

8. High Availability (backup server, cluster, etc)

The task of troubleshooting a problem has three key components

• Read the error message thoroughly and collect relevant information from all system and application logs. Reading the description of the error message and acting accordingly can solve many problems
• Correlated corresponding error logs and event logs. Other errors might have occurred before the problem and might be the cause for the problem.
• Try to isolate the problem and refer to it to particular element of HPOM infrastrcture. HPOM uses a client-server architecture made up of a management server, management console, and agents running on the managed nodes. The management server uses a database to store the service model, information about the managed nodes, methods used to manage the nodes, and events that have occurred on the managed nodes.

Information about HPOM server problems
that you need to collect before calling HP support

• What is the status of the HPOM processes on the server. What is the status of agent processes on the server.

• Did any error messages appear in the log files?
• Are there any errors in the itochecker report?
• Do the processes start and stop properly? Can you stop and restart the server?
• Do you have the current patches installed for the agent, server, and operating system?
• Did the failed process produce a core file?

• Do you have a current system backup?

• What is the complete error message?

• Versions of the HPOM software products in use

• Versions of the operating system for the server

• Server platform is involved?

• Where did the error occur?  When did the error initially occur?

• Can the problem be replicated?

• What recent changes were made to the operation system and/or HPOM system (such as patches, configuration changes, etc)?

Check for Errors

Error messages from HPOM are reported to the user via a variety of sources. The error message sources are the log files, the graphical user interface, and the shell. In the graphical user interface, the error messages may appear in a pop-up window as the result of an illegal operation or in the message browser within the message text. There are a few log files that contain important information about the normal operation of the system and when necessary error messages when an operation within the OV environment did not complete successfully. For example, after installing an operating system or OV patch you should check the installation log files for any errors.

Here are topics from Advanced Administration class that can be helpful:

Architecture

• Global Architecture
• HTTPS versus DCE Communication
• HPOM Queue Files
• Open Agent Overview
• Component Architecture - Terminology
• OV Foundation
• Component Architecture (HTTPS Agent)
• Architecture - Message Flow, Policy Management and Processes

HPOM Communication and Firewalls

• HPOM in Secure Environments
• HTTP Communication
• HPOM and HTTPS - Managing through a Firewall
• Using HTTP Proxies
• Proxies and Content Filtering
• HPOM and HTTPS -- Using Proxies
• Proxies - HPOM Configuration
• Proxy Configuration - Syntax
• HTTPS Agent and NAT
• Traditional HPOM Communication
• Use of the Reverse Channel Proxy
• Configuring Outbound-Only Communication
• DCE Communication Across Firewalls
• DCE RPC - Details
• DCE Nodes -- Firewall Configuration

Certificate Use and Management

• Certificates, Encryption and Signatures
• Certificates and HPOM
• Use of Certificates and Private Key for SSL Handshake and Signature
• Basic Certificate Commands
• Certificate Deployment and Backing Up Certificates
• Granting Certificate Requests in HPOM GUI
• Certificate Client and Server Commands
• Manual Certificate Deployment and Installation
• Using an Installation Key

Advanced Troubleshooting

• XPL Distributed Architecture
• XPL Trace Server
• Tracing -- Configuration
• Viewing Traces with the Windows GUI Client
• Restricting Trace Information
• Searching and Saving Trace Data
• Command Line Trace Utilities
• DCE Agent Tracing
• Verifying Core IDs and Certificates
• Invalid or Lost Certificates
• Compromised Certificates or Keys
• HTTPS Communication Problems
• HPOM Test Tools
• Testing Data Flow

Online Help

During troubleshooting, it is helpful to have all the necessary information and resources at your fingertips. The online resources provided within the HPOM platform make access to important information easy. Inside each graphical window there is a HELP button on the menu.

# tail /var/opt/OV/log/OpC/opcerror|grep ERROR
					    

• actagtq <OvAgentInstallDir>\tmp\OpC\
• agentid <OvAgentInstallDir>\conf\OpC\
• agtreg <OvAgentInstallDir>\conf\OpC\
• default.txt <OvDataDir>\conf\BBC\
• managedNodeId.txt <OvAgentInstallDir>\conf\OpC\
• mgrconf <OvAgentInstallDir>\conf\OpC\
• monagtq <OvAgentInstallDir>\tmp\OpC\
• mpicmaq <OvAgentInstallDir>\tmp\OpC\
• mpimaq <OvAgentInstallDir>\tmp\OpC\
• msgagtdf <OvAgentInstallDir>\tmp\OpC\
• msgagtq <OvAgentInstallDir>\tmp\OpC\
• msgiq <OvAgentInstallDir>\tmp\OpC\
• nodeinfo <OvAgentInstallDir>\conf\OpC\
• opcdfchk <OvAgentInstallDir>\contrib\OpC\
• opcinfo <OvAgentInstallDir>\bin\OpC\install\
• (AIX: /usr/lpp/OV/OpC/install/)
• opcqchk <OvAgentInstallDir>\contrib\OpC\
• opcrpccp <OvAgentInstallDir>\contrib\OpC\
• OvChangeMsg <OvInstallDir>\Support
• OvDebugSlot <OvInstallDir>\Support
• OVDumpEvtLog <OvInstallDir>\Support
• OvInstaller <OvInstallDir>\Support
• OvModelValidatorEx <OvInstallDir>\Support
• Ovntkill <OvInstallDir>\bin
• OVSuptInfo <OvInstallDir>\Support
• OvUnInstaller <OvAgentInstallDir\{20a6…}\
• Pids <OvAgentInstallDir>\tmp\OpC\
• NodeUpdateQueue <OvInstallDir>\temp
• UnmanagedAgentQueue <OvInstallDir>\temp

Log files

Installation creates a number of log files, all located in the folder <OvDataDir>\HPOVInstall.

Each log file tracks a specific portion of the overall product installation or uninstallation. The information contained in the logs is mostly the output of the msiexec.exe application (See Other Tools); however, some of the information generated is specific to the portion of the install executed. See Log file examples.

Log file filename format follows these rules:

<package>_install.log (installation)

<package>_remove.log (uninstallation)

where <package> is any of the subpackages listed in the tables above.

One additional log file (non-Windows Installer format) is created during installation that logs

the status of the many configuration tasks performed during the installation of management servers and consoles. The name of the file is ServerConsoleInstallResults.txt.

See Log file examples.

NEWS CONTENTS

• 20100729 : HPSWSC - corefiles on AIX Managed Nodes - ( HP Software Solutions Community online forum )

Old News ;-)

[Jul 29, 2010] HPSWSC - corefiles on AIX Managed Nodes -

HP Software Solutions Community online forum

om-m:

We have AIX managed nodes running https agents 8.6. Sometimes one the subagents crashes and a large core file is written to /var/opt/OV/tmp. Is it possible to configure the agent in a way that there is no core file written in case of a crash ? The agent is restart by the selfmanagement policies.

pituhdo, HP Knowledge Expert:

HI,

I'm afraid that the core dumps are controlled by the OS rather than the agent itself, so AFAIK there is no way to configure the agent to not create core files.

You can configure the size of the core files on the configuration file /etc/security/limits as can be seen on

http://www.ibm.com/developerworks/aix/library/au-aixuseradmin/index.html

If you want that your system doesn't create any core file just set the core variable to 0.

Core files are caused normally by a bug in the software so I would recommend contacting HP Software Support in order to get it solved.

Hope this helps,

Pitu

Recommended Links

How to check the used and spare space of tablespaces for the HPOM oracle database

How to uploading Policies on HPOM 9 using the AdminUI

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019