System Heartbeats with TEC

This type of monitoring is somewhat similar to Heart rate monitoring which might explain the origin of the term. Heartbeat monitoring can be implemented either as part of endpoint health monitoring (the simplest way is to use “wep status” and “wadminep view_version but Framework v4.1.1  has a new feature: endpoint health check ) or as a central service. In the latter case you can monitor remote hosts independently of whether they have endpoint installed or not from one central location. The most primitive type of this type of monitoring is periodic pings.

ITM 5.1 also can be used, see Heartbeating in 30 minutes with ITM 5.1.1 - part 1

Netview contains build in service for heartbeat monitoring so if Tivoli installation includes Netview it is easier to do it this way.  See also NetView rule set (netview.rls). This is covered is IBM Tivoli software training course - IBM Tivoli Enterprise Console 3.9 Solutions:

NetView Integration

• Lesson 1: Architecture Overview
• Lesson 2: Netview Rulesets
• Lesson 3: Troubleshooting Netview Rule Sets

Note:  Orb Data website contains a valuable collection of Tivoli TEC tips.  You can search for the word "TEC" (or more specific search string) from the main page to get the list of  available articles or visit Knowledge Base

NEWS CONTENTS

• 190313 : Framework v4.1.1 endpoint health check by Gary Hamilton ( November 2002 , The Tivoli Advisor )
• 190313 : Heartbeating in 30 minutes with ITM 5.1.1 - part 1 ( Orb Data )
• 190313 : System Heartbeats with TEC - Monitoring by Exception ( System Heartbeats with TEC - Monitoring by Exception, )
• 190313 : System Heartbeats with TEC - a state engine approach ( Orb Data )
• 190313 : RE ITM heartbeat endpoint unreachable alert question - msg#00823 - ( RE ITM heartbeat endpoint unreachable alert question - msg#00823 - , )
• 190313 : HeartBeat Event to Monitor NV6K Alive Status ( Jun 1. 1998 lists.tivoli DOT com )

Old News

Framework v4.1.1 endpoint health check By Gary Hamilton

November 2002 | The Tivoli Advisor

Why is the health check necessary?

In framework version 4.1.1, a new feature was added to help customers with the perennial problem of awareness of the health of endpoints in the environment. Various custom solutions have been written in the past, utilizing either the CLI commands "wep status" and "wadminep view_version" or Distributed Monitoring/ IBM Tivoli Monitoring heartbeats. The problems associated with both these approaches are:

1. The CLI commands were never designed with this function in mind, so they are ineffective, inconsistent and incomplete. If a problem occurs there is inadequate information to understand and address the problem.
2. The heartbeats are reliant on the underlining products and framework. The products were not designed with the endpoint process' health in mind. The expectation is that the endpoint will be healthy.
3. To use the heartbeat you must have a monitoring product installed. The new health check is built into the framework. Therefore no additional products are necessary.

What does the health check do?

The new feature is concerned with the health of the endpoint. This means it tests whether the endpoint is able to perform the operations the products will require it to do.

These operations are:

a. Downcalls

b. Upcalls

c. Method and task execution

d. Creation and deletion of files

e. Token creation on Windows platforms

These five operations encompass all the general functions that products perform on the endpoints.

Heartbeating in 30 minutes with ITM 5.1.1 - part 1

Orb Data

The IBM solution comes as part of ITM (IBM Tivoli Monitoring) 5.1.1 and it can be used to monitor the basic availability of endpoints attached to the gateway at which it is enabled. Events can then be sent to the Tivoli Business Systems Manager (provided that the Tivoli Business Systems Manager Adapter component is installed at the gateway), the Tivoli Enterprise Console (TME® events only) or the Tivoli Monitoring Notice Group (default).

This tip is split into 2 parts. In this first part of the tip I will discuss the physical architecture, some design decisions you need to make and the actual implementation of a heartbeating solution. In part 2 of the tip I discuss the events that you might expect to see if an endpoint is shut down, concentrating on TEC and Notice group integration.

At the end of part 2 of the tip I also detail a simple script that I have written named heartbeat.pl. This script can complete some of the steps that are shown in this tip with only a few basic options, and will hopefully allow your organization to implement heartbeating in 30 minutes.

System Heartbeats with TEC - Monitoring by Exception

Provides the second part of a two part explanation of the use of system heartbeats to display the status of your key systems. This method monitors by exception, displaying only those systems that have missed a specified number of heartbeats for a specified time. Included in this tip are all of the rules, classes and external scripts required.

System Heartbeats with TEC - a state engine approach

Orb Data

One of the requirements of the majority of our customers has been to implement some kind of heartbeat mechanism from their most critical systems. Generally the requirements for this are the same, the customer wants know at any time whether a system is up and running or not.

Even though the requirements are the same, we have seen many different solutions. Over the next two weeks we will discuss two such solutions.

This tip outlines the first solution to some very specific requirements.

RE ITM heartbeat endpoint unreachable alert question - msg#00823 -

sysutils.tivoli.general

Same here, I ended up creating a new severity, TIVOLI, and redirecting these
'heartbeating' events to there, as there are way too many false alarms.

Best Regards

David Weir
IBM Certified Deployment Professional - Tivoli Monitoring V5.1.1

-----Original Message-----
From: owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx
[mailto:owner-tme10@xxxxxxxxxxxxxxxx] On
Behalf Of Duffy, Brian (OFT)
Sent: Friday, 30 January 2004 5:36 AM
To: tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx
Subject: RE: [tme10] ITM heartbeat "endpoint unreachable" alert question

It means that the endpoint didn't send a heartbeat upcall when it was
supposed to, and the gateway was unable to send a downcall to it.

In our environment, it tended to throw a lot of false alarms. We ended
up writing TEC rules and a couple of scripts to check out the status of
an endpoint before alerting people.

Brian J. Duffy
NYS Office for Technology
brian.duffy-zVgB07JTx4c4Xc9VQZkPiQ@xxxxxxxxxxxxxxxx

-----Original Message-----
From: owner-tme10-XtjxT7Vmt5b1ENwx4SLHqw@xxxxxxxxxxxxxxxx
[mailto:owner-tme10@xxxxxxxxxxxxxxxx]
On Behalf Of D'Apice, Dominic
Sent: Thursday, January 29, 2004 1:09 PM
To: Tme10 (E-mail) (E-mail)
Subject: [tme10] ITM heartbeat "endpoint unreachable" alert question

What mean exactly the "endpoint unreachable" alert in ITM heartbeat ?

Does it mean that the server doesn't response to a ping command or it
mean that the Endpoint process doesn't response to the wep status command ?

Please can someone clarify this question ?

thanks
Dominic D'Apice
* D.D'Apice-XB5swX22BCD3fQ9qLvQP4Q@xxxxxxxxxxxxxxxx

HeartBeat Event to Monitor NV6K Alive Status

Jun 1. 1998 lists.tivoli DOT com

Good Day Netview Admins. We are running NV V4.1.2 on AIX platform. We are running the secured tecadapter to our TEC box (Solaris 2.5.1 TEC3.1). Once events pass through the TEC rulesbase, trouble tickets that are deemed critical or fatal are passed to our Trouble Ticket System - ala..Remedy.

Here's the question.

We want to monitor the entire process from the Netview Box through the adapter to TEC and onto Remedy. Knowing that NV6K polls the entire network, what do you need to do to ensure the NV6K box and all the necessary processes are up themselves all the time. Does NV6K issue a trap or other events before the processes on the box crash? We have Tivoli Distributed Monitor installed on NV6K monitoring files structures, and other processes. But then if the oserv dies on this box, we are out of luck for any of its monitors. Our attempt here is NOT to rely on Tivoli for all monitoring purposes. Sort of a contingency backup failover system to make sure our Enterprise Systems Management boxes are healthy.

Our thinking here is creating a new event within NV6K EventConfiguration panel which would fire every 30-60 minutes. Using Tivoli rulesbase we could have the event pass eventually to Remedy where it could handle filtering and deem the system is up. If the event is not received at Remedy by a certain point, then we can assume somewhere in the process has experienced a failure.

Thanks for any tips anyone can offer that may have configured a similar "HeartBeat" event to monitor the health of the entire process.

John Mull
Hershey Foods Corporation
Information Technology & Integration
Enterprise Systems Management
(717)534-7959
email:jmull AT hersheys DOT com

Recommended Links

Heartbeating in 30 minutes with ITM 5.1.1 - part 1

System Heartbeats with TEC - Monitoring by Exception

System Heartbeats with TEC - a state engine approach

1. [PDF]

   PDF - No Slide Title

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019