Automatic acknowledgement

Note: HP renamed the product called now HP operations manager way too many times. Also it is very inconsistent with using abbreviations. Here we will assume that the term "HP Operations manager" and abbreviations HPOM, OMU, and OVO  mean the same thing :-)

Sometimes, you may want to acknowledge messages automatically. The following example illustrates the approach that can be used:

• The first message reports a problem (opening message).  If multiple messages are produced they can be deduplicated.
• The second message reports that the problem disappeared or has been solved( Closing message). For example, if a user fails to log on because of an incorrect password, but succeeds with a second attempt. Or the second message might report that the problem has deteriorated. In either case, the first message is no longer be relevant and should be acknowledged. 

If you would want to the second message to automatically acknowledge the first message. HPOM enables you to automate this type of correlation (for example, an application shutting down and starting up again).

Please note that if  you acknowledge messages automatically, just two states can exist in the browser.

• "Opening message" exists. That means that the problem exists.  In other words the last  message reflects the current status of the object.
• Neither opening nor closing messages exist or only the closing message exists. That means that the problem was acknowledged.

In effect, the message browser has become a state-based browser. (For suggestions on how to implement this concept for threshold monitors, see “A Default Message Key and Message Key Relation”)

Acknowledging Messages with Message Keys

When you work with related message, the relationship between message that "opens" the situation and message the "closes" it is established through message keys.

A message key acknowledges all previous messages with  matching the message key. The pattern used for performing matching is specified with MSGKEYRELATIONS ACK keywords in the policy body.

Annotating Acknowledged and Acknowledging Messages

When a message is acknowledged automatically by another message, both messages are annotated, as follows:

• Acknowledged Message The acknowledged message is annotated automatically with details about the acknowledging message. These details about the acknowledging message include its message ID, condition ID, and message key relation.
• Acknowledging Message The acknowledging message is annotated with details about the acknowledged message. These details about the acknowledged messages include its message key relation, the number of messages it has acknowledged, its message IDs, and its condition IDs. These annotations can be useful when troubleshooting. NOTE The status of the message has no influence on whether it will be acknowledged: owned messages, pending messages, and messages with running actions are also acknowledged.

A Default Message Key and Message Key Relation

For threshold monitor policies, HPOM can generate a default message key, along with a message key relation, for each condition. To generate a default message key and a message key relation for each condition, use the AUTOMATIC_MSGKEY keyword. The following default values are generated:

• Message Key <$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<$THRESHOLD>
• Message Key Relation ^<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>$

This message key indicates that the monitor disk_util has detected more than 90% used disk space in the root directory (/) on the node managed_node.hp.com.

The message key relation ensures that all those messages are acknowledged that are triggered by the monitor disk_util and report that disk utilization of / on the node managed_node.hp.com has exceeded or fallen below a threshold. Sending a Reset Message Automatically

HPOM also automatically acknowledges the last message of a monitored object by sending a reset message if a threshold was first exceeded and the monitored value then drops below all possible reset values. In other words, no condition matches any longer. The reset message is provided with a message-key relation that acknowledges the last message that was sent for a certain monitor. This last message must contain a message key. Otherwise, no reset message is sent.

The reset message cannot be configured, but has the following default text: <monitor_name>[(<instance>)]:<value> (below reset). The reset message does not display in the message browser, but is sent directly to the history database. The severity of the reset message is set to normal. Automatic or operator-initiated actions are not defined.

For details on how the monitor agent behaves when multiple conditions exist for one monitored object, see “Threshold Monitoring with Multiple Conditions” on page 403. Example of an Automatic Reset Message Figure 4-9 shows an example of a reset message sent automatically.

Implementing Message Policies
Strategies for Optimal Message Filtering

MONITOR "my_mon"
...
MAXTHRESHOLD
...
DESCRIPTION "my_mon 1"
CONDITION
...
THRESHOLD 99.0000
RESET 95.0000
...
SET
...
SEVERITY Critical
...
MSGKEY "<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<$THRESHOLD>"
MSGKEYRELATION ACK "^<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>$"
DESCRIPTION "my_mon 2"
CONDITION
...
THRESHOLD 90.0000
RESET 85.0000
...
SET
...
SEVERITY Major
...
MSGKEY "<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<$THRESHOLD>"
MSGKEYRELATION ACK "^<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>$"
DESCRIPTION "my_mon 3"
CONDITION
...
THRESHOLD 80.0000
RESET 75.0000
...
SET
...
SEVERITY Warning
...
MSGKEY "<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<$THRESHOLD>"
MSGKEYRELATION ACK "^<$NAME>:<$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>$"
Reset Message: monitored value below lowest possible reset value (< 75%).

• my_mon 1 Generates a critical message when the monitored value exceeds 99%. When the value falls below 95%, the counter is reset.
• my_mon 2 Generates a major message when the monitored value exceeds 90%. When the value falls below 85%, the counter is reset.
• my_mon 3 Generates a warning message when the monitored value exceeds 80%. When the value falls below 75%, the counter is reset. The message key relation of each message ensures that the previous message is automatically acknowledged. The last message (generated by “my_mon 3”) is automatically acknowledged by the reset message when the monitored value falls below the lowest possible reset value (in this example, below 75%).

NEWS CONTENTS

• 20110124 : open message interface acknowledge message with message key question ( Feb 28, 2010 , IT Resource Center )
• 20110124 : Message Key ( Sep 20, 2002 , IT Resource Center forums )
• 20110124 : Message correlation in HP openview ( Jun 22, 2006 , IT Resource Center forums )

Old News

[Jan 24, 2011] open message interface acknowledge message with message key question

Feb 28, 2010 | IT Resource Center

John Hawkins

OMW 8.10

I've got an open message interface policy with 5 rules, each with a different threhold of Critical, Major, Minor and Warning and each has a message key of .....

<$OPTION(hostname)>:<$MSG_OBJECT>:<$MSG_SEV>

On the Correlation tab in the acknowledge messages with message key field i've entered this .......... <$OPTION(hostname)>:<$MSG_OBJECT>:<*>

In hopes that each time a rule in the policy is matched the new message that is created will acknowledge any messages in the browser that were generated previously by another rule in the policy. This is not working. Instead what i'm seeing is that if the same rule (eg. warning) is matched multiple times the counter is increased (which is fine by me) and then if a different rule is matched (eg. minor) it creates a new message in the message browser leaving the previous warning message. Acknowledgement is not working and i'm not sure why not. Any thoughts?

Drew Dimmick:

You have to send the message to the acknowledged browser, not the active browser.

John Hawkins:

I tried that and its better, except that I don't (obviously) have the lastest alarm in my active broswer. Also, if i use a pretty aggressive polling interval like i'd like to i'm going to get notified via email each time it polls. I was hoping i could suppress and count duplicates of the same severity and then once any change in severity was detected by the policy it would send those to the active message broswer and ack previous messages. Is that not possible?

Dennis O'Quinn:

Hi, use <![$MSG_SEV]> for the message severity clause (instead of '<*>')

i.e., <$OPTION(hostname)>:<$MSG_OBJECT>:<![$MSG_SEV]>

This will close any active events with a severity that is different from the 'current' event.

John Hawkins:

That did the trick. I tried passing that in every which way but that one. Seems so obvious now. Thanks very much!

John Hawkins:

I spoke too soon. I thought it was working, then went back to do some more testing and it wasn't after all.

Seems like it should be ok. Below is what is in my policy for msg key and ack msg key Message Key: <$OPTION(hostname)>:<$MSG_OBJECT>:<$MSG_SEV> ack message key: <$OPTION(hostname)>:<$MSG_OBJECT>:<![$MSG_SEV]>

This is what i get in the msg properties of the active message browser...... Message Key: lfkndrac01.consolidated.com :DATA:Minor Ack Mesg Key: lfkndrac01.consolidated.com :DATA:<![$MSG_SEV]>

I've got to be overlooking something, I'm just not seeing it. Any ideas?

John Hawkins:

Should probably add more detail. What is happening is still that if there is, for instance, a minor message in the browser triggered by the policy rule and then another policy rule catches a major violation it creates a major event in the browser an leaves the minor as well instead of ack'ing it.

Dennis O'Quinn:

John, what you see in the keys looks right... We do this all the time. I also ran a test to verify via a test open message interface policy, and, it ran just fine. Did all the things it sounds like you are wanting... Same severity messages incremented the dup count and differing severity messages acknowledged any current messages. If this is OMW, we (frequently) have WMI problems and have to bounce WMI to straighten things out... You may try that and test agin...

Dennis

John Hawkins:

Thanks for taking the time to respond again. I'm glad to hear it should work as expected and that i'm on the right track.

This is omw8.10 and a linux 8.65 agent. I bounced wmi but still the same thing. I'm perplexed, I've been staring at this thing for hours and am now at a loss.

I suppose it could still be something in the policy but I did like the wmi thought. Seems like the issue still might be on the mgt sever side. Any other thoughts on some possible troubleshooting i can do to try and figure out what the problem might be?

Dennis O'Quinn:

John, I would simplify first try to isolate the location of the problem.

For instance, set up an open message interface policy that traps on something like app=tstapp, obj=tstobj, and all severities. Then have that policy set the msg key and ack key to something like...

<$MSG_NODE>:<$MSG_APPL>:<$MSG_OBJECT>:<$MSG_SEV>

and

<$MSG_NODE>:<$MSG_APPL>:<$MSG_OBJECT>:<![$MSG_SEV]>

deploy to a given server and, then, from that server (I just did mine on the mgt server), just start issuing opcmsg commands like...

opcmsg a=tstapp o=tstobj msg_t="some text" sev=.... (normal, minor, major, critical for sev).

and see what happens... if it works like you want, then it is (maybe?) the other policy, if not, then it is *maybe?) your server or agent.... Then, just issue an opcmsg command from a commandline somewhere

Dennis

John Hawkins:

Dennis, thx for the tips, working great now. Once i tested as you suggested i realized it had to be the policy.

At some point, early on, when testing the policy I couldn't get <$MSG_NODE_NAME> to come through in the message key and couldn't figure out why not. So, in the perl script that makes the opcmsg call I was grabbing the $HOST_NAME=`uname -n`; and passing it to the opcmsg call as an -option and then set the msg key to ............

<$OPTION(hostname)>:<$MSG_OBJECT>:<$MSG_SEV>

I switched the message key back to <$MSG_NODE_NAME>:<$MSG_OBJECT>:<$MSG_SEV>

noticed the node name was now being populated in the $MSG_NODE_NAME and things are fine.

I was actually passing a couple of variables using the opcmsg -option, so maybe that contributed to the problem.

I wanted to let you know you were extremely helpfull and I appreciate you taking the time to respond back.

Dennis O'Quinn:

No problem, glad you got it fixed... I remember having some problem in the past with $MSG_NODE_NAME vs. $MSG_NODE before, but, that was long ago and don't remember the details... I see that I am using $MSG_NODE now... so... not sure what the difference is... but, as long as it works... D

Message Key

Sep 20, 2002 | IT Resource Center forums

kevinb:

I have a message key set with the same message key relation set for a message template. I would expect that new messages arriving from the same node with the same object and severity to acknowledge the old out of the browser but that is not happening. The new message is becoming an annotation of the old message. The key is <$MSG_NODE_NAME>:<$MSG_OBJECT>:<$MSG_SEV>

Charles G.:

Hello,

Did you specify a pattern for message key acknoledgement in the message correlation window of the template definition ? I don't have acces to ITO, it's the field under the message key definition.

To have the result you expect, you have to specify : Message key : <$MSG_NODE_NAME>:<$MSG_OBJECT>:<$MSG_SEV> And in the message key acknoledment pattern : ^<$MSG_NODE_NAME>:<$MSG_OBJECT>:<$MSG_SEV>$

With these settings, any new message arriving will have the message key set, and will acknowledge all previous messages with the same key.

I guess the new message is added as an annotation of the previous message because the suppress and count duplicate messages is enabled on your management server, with the "add duplicates as annotation" checked.

kevinb:

no go with that set up

Greg Vaidman:

sounds like you have the duplicate counter set to annotate duplicates. try it with that feature turned off (Actions->Server->Configure)

If that doesn't work, try setting the relationship to ^<$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>$

That's the normal relationship, so that any subsequent message that refers to the same object (not just the same condition, which is what the object and severity together refer to) would be acknowledged automatically.

I haven't seen anyone do what you're trying, which is to use message correlation to ack the same exact message. usually the duplicate counter is enough to keep the browser from filling up because of the same message occurring.

hope this helps, greg

kevinb:

Greg I have tried this and still it is not working. I have two conditions, one for warning and one for minor. the warning comes in and I pump up the FS size to get the minor alarm and it just becomes a counted duplicate even though it is a different severity. Any thoughts.

Greg Vaidman:

I think i misunderstood your original message. you don't want the severity in your key, you want the value of the object you're measuring. you should use pattern matching in your message text to get the value into a variable, and then put that variable at the end of your message key.

your key will look like: <$MSG_NODE_NAME>:<$MSG_OBJECT>:<value>

("value" is the variable you create in the main part of the condition)

and your relationship like: <$MSG_NODE_NAME>:<$MSG_OBJECT>:<*>

note that if the exact same value comes in, you'll get a duplicate, but if a different value (even with the same severity) comes in, you'll get the old message ack'd and the new one in the browser.

regards, greg

kevinb:

Greg thanks for that bit of info, it works with only one issue. When an alarm comes in, for example "Warning node1 /tmp space is at 18%". Ten minutes later another comes in, "Minor node1 /tmp space is at 25%". The Minor alarm acknowledges the Warning and continues that way if the % goes up. But when the % goes back down into the Warning level the alarm is surpressed and never comes into the browser. I have tested dropping the % back down into the Normal range and still the higher alarm stays in the browser. Any thoughts?

Message correlation in HP openview

The concept of message key is very important

Jun 22, 2006 | IT Resource Center forums

Remco Dhaenen

In that case,

In the template/policy condition for the active message, in the 'message key', enter a key e.g. <node>:<app>:<object>:down.

This if for event that specs something is down.

Then for your next message, the UP, mention in the condition to acknowledge active messages with the above key.

Make sure every key is tuned, to avoid that other messages are ack'd as well.

All variables you can use are in the help pages as well.

BTW : you posted this in NNM thread, better place it in Operations

remco

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019