itochecker - a utility for HP Operations Manager Health checking

itochecker is a Born shell script that produces various reports about configuration of HPOM management server. It also attempts to identify configuration items which may cause problems. itochecker should be used on a regular basis for proactive determination of HPOM problems as well as troubleshooting tool when a particular problems occurs.

While a useful script with the unique and very educational set of checks (studying the code gives you some additional insights about HPOM), it is badly maintained and have a serious weaknesses some of them stemming from the use of Born shell and others from the lack of maintenance:

• The itochecker is weak in detecting network communication problems with the agent and its summary table does not have such a category. That's a serious weakness.
• Also if the node is disabled the itochecker  does not explicitly detect this state.

It is an unsupported large shell script (around 7K lines), but HP continues to maintain the tool and welcomes feedback on problems as well as enhancement requests. The logic is simple that the code has some educational value. The main structure is a set of procedures each of which performs one particular check:

check_mgmt_sv()
check_DNS()
check_required_patch()
check_nslookup()
check_system_file_perm()
check_kernel()
check_opcinfo()
check_distrib_dir()
check_cluster_info()
check_ito_files()
check_opcinfo_opcerror()
check_server_processes()
check_permissions()
check_and_init_db()
check_num_audit_files()
check_supp_unmat_cond()
check_templ_not_group()
check_msg_node_grp()
check_oracle_diskspace()
check_node_groups()
check_mpi_templ()
check_agent_processes()
check_templ_assigned()
check_ns()
check_if_pingable()
check_node_snmp()
check_licenses()
check_ito_op_dir()
check_java_files()
check_svcnav_files()
check_services_and_loggings()
check_pre_req()

This is strictly reporting tool. Internally it is uses regular HPOM commands such as: 

/opt/OV/bin/OpC/opcragt -status -all
/opt/OV/bin/OpC/opcragt -agent_version -all
/opt/OV/bin/OpC/opcragt -get_config_var eaagt:OPC_PRIMARY_MGR -all
/opt/OV/bin/OpC/opcragt -get_config_var sec.core.auth:MANAGER -all

It doesn’t make any changes on the system it checks, but you can try remotely restart agent for nodes that have problem only reported only in  "Agent State" column (WARNING or ERROR):

  Nodename                                 Node state   Agent State  Cert. State  Config State    Overall
  lux26.dmz2.firma.com                        OK           WARNING      OK           OK           WARNING

Default report files created are

• /tmp/ito_rpt/ITO.rpt -- text version of the report
• /tmp/ito_rpt/report.html -- html report index (there additional multiple file sin the directory referenced by this index)

They are created in the output directory, which is set to /tmp/ito_rpt by default. The output directory can be changed by setting the ITOCHECKDIR environment variable before calling the itochecker / itosend scripts.

If run without options it presents interactive menu. To run the tool in non-interactive mode you need to supply at least one option. For example to check HPOM , you can run: 

		# /opt/OV/contrib/OpC/itochecker -2

It will generate the report in both text and HTML formats. The HTML is structured as index page and multiple sections (one for each category of checks).

Syntax

itochecker [ -1234567 [-l] [-x] [-t] [-a] [-m] [-s nodename] ]

Options

• -1 System Environment Check: Domain name, whether DNS is in use, number of lines in /etc/hosts, whether nslookup is working for user opc_op, Server Memory, Swap space, and product versions, Number of processes. Installed OS patches and patchbundles, if required patches are installed.
   
• -2 HPOM environment check. Produces report about Installed HPOM bundles, HPOM and ECS designer versions, What proesses are running on the server, Server kernel parameters and whether the requirements are met, Installed HPOM and NNM patches, version and patchlevel of all HPOM binary and library files, Disk space in DB and HPOM directories, Pending agent data in the distribution directory, Decoded agent configuration files, Cluster information, Core file information, opcsvinfo(only HPOM 7.x), opcinfo, configuration settings and System.txt files, Elements in server and agent queues, file permissions of some critical files.
   
• -3 General Database Check: Database version, How many messages are in the active and history message tables, Whether all node group / message group pairs are assigned to some operator, Test the number of files in the audit directory of Oracle (A large number of files in this directory can cause the monitor mondbfile to fail).
• -4 Database / Node Verification: Compares agent entries in the database with the file system, Gets the free disk space in the oracle directory
• -5 Nodes Check: Hostname inconsistencies, Type of node (message allowed, controlled, ...), Nodes belonging to node groups, Long vs Short host names (DNS in use, DNS not in use), OPC_IP_ADDRESS points to a IP address on the local host, Checks the case of host names, returns warning if upper case is used. Checks if ip.ipaddr in the mib tree does not exist, Non-NCS based systems are not checked via LLBD (NT Boxes)
• -6 Java GUI / Service Navigator Check: Java version, size of service repository, number of configured ser vices and loggings, output of ‘stacktrace -svc’
• -7 Performance Checks: measuring the time it needs to get a service tree from opcsvcm, (other checks to be implemented)
• -l Check licenses when checking nodes: how many nodes need a license, how many nodes are set to get a license from the server
• -x Check via snmp when checking nodes
• -t tar output files to one tar-file, including cluster, core, opcinfo, System.txt information
• -s nodename Check only the given node, when checking nodes
• -m Send messages to the HPOM Management Server for every WARNING and ERROR found during the check. itosend can also be used to send the output file (ITO_rpt.tar.Z or ITO_rpt.tar) to a specified email address. It is necessary to run itochecker to create the report files, before using itosend. NOTE: You must run itochecker with the option -t, or choose the additional t option in the start menu of itochecker! Only this will create a packed .tar or .Z file! (the option -a contains the t option) itosend will search this file in the /tmp/ito_rpt/ directory, and send it to the specified email address. ( itosend will ask you for an email address)
• -a All options activated, except the -l, -x, -s and -7 options

Return Value

In case errors were detected during the check, itochecker returns the following value:

Files

1. /opt/OV/contrib/OpC/itochecker -- The main script
2. /opt/OV/contrib/OpC/itochecker.conf -- Configuration file, used by itochecker it can be used to change settings, e.g. Kernel Parameters,...
3. /opt/OV/contrib/OpC/itosend -- Script to send the packed itochecker output file per email to HP Operations support
4. /opt/OV/contrib/OpC/agent_platform.conf -- Config file, used by itochecker to decode machine numbers to platform paths and detect the machine types of managed nodes.
5. /tmp/ito_rpt/ITO.rpt -- Plain text report file
6. /tmp/ito_rpt/report.html -- html report index (there additional multiple file sin the directory referenced by this index)

Examples

Invoke itochecker and presents a user menu:

itochecker

Invoke itochecker and check nodes:

itochecker -5

Checks the HPOM environment, the database, the node stroppy and tars all output files to one tar file:

itochecker -235 -t -s stroppy

NEWS CONTENTS

• 20120130 : HP Communities - itochecker, node check problem - Enterprise Business Community ( HP Communities - itochecker, node check problem - Enterprise Business Community, Jan 30, 2012 )

Old News ;-)

[Jan 30, 2012] HP Communities - itochecker, node check problem - Enterprise Business Community

Re: itochecker, node check problemOptions

Could anyone help me understand what itochecker is doing when it checks nodes, and why our node reports are empty?

Goran Koruga 04-20-2010 09:17 PM

Hello.

Are you referring to the "-5" option of the 'itochecker' tool?

It uses opcragt, nslookup, SNMP checks and the likes.

If it's about the option '-3', then it uses SQL queries to get a list of nodes from the OPC_NODES table.

In any case, itochecker is a shell script so it should be trivial for you to find out what it's doing.

OK here's what I recommend.

1. Copy itochecker to /var/tmp and run 'chmod 755 /var/tmp/itochecker'

2. Edit the script, find function "do_nodes_check" and make its start look like so:

do_nodes_check()
{
set -x

We only added 'set -x' to assist with debugging.

3. Run it:

/var/tmp/itochecker -5 >/var/tmp/chk.txt 2>&1

4. Look in the file "/var/tmp/chk.txt" and find a section similar to this (note that your values will be different):

+ + wc -l /tmp/opc.22782/node_list
+ awk {print $1}
NUMNODES=6
+ + date +%H %M %S

What does it set NUMNODES to?

Below that it should say "Checking 6 node(s)" - again, the number will be different in your case. After this it shall process all the nodes it has found previously.

It uses the SQL query I used before and you said it lists a lot of nodes, so that part looks OK. The NUMNODES value should be he same number as is the number of rows returned by the SQL query you used before.

Regards,
Goran

Recommended Links

HPOM Health Checks

HP Communities - itochecker, node check problem - Enterprise Business Community

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019