Processing SNMP Traps

Trap is the only PDU sent by an agent on its own initiative. It is used to notify the management station of an unusual event that may demand further attention (like a link going down). In version 2, traps are named in MIB space. Newer MIBs specify management objects that control how traps are sent. Includes current sysUpTime value, an OID identifying the type of trap and optional variable bindings.

The simplest was to catch SMTP trap is to use tcpdump tuned to the specific server from which trap is expected (or group of such servers) and SNMP ports. Here is a relevant quote from Watching SNMP on the wire

To illustrate more accurately how SNMP actually works on the wire, lets look at sniffed packets captured during an SNMP GET. Only 2 packets are exchanged, a request by the manager and a response by the agent. The following was gathered using Ethereal:

No.     Time        Source      Destination Protocol Info
    112 1.936155    10.10.1.110 10.10.1.224 SNMP     GET SNMP...
Simple Network Management Protocol
    Version: 1 (0)
    Community: public
    PDU type: GET (0)
    Request Id: 0x2a7ee1af
    Error Status: NO ERROR (0)
    Error Index: 0
    Object identifier 1: 1.3.6.1.2.1.1.4.0 (SNMPv2-MIB::sysContact.0)
    Value: NULL

No.     Time        Source      Destination Protocol Info
    113 1.943425    10.10.1.224 10.10.1.110 SNMP     RESPONSE SN...
Simple Network Management Protocol
    Version: 1 (0)
    Community: public
    PDU type: RESPONSE (2)
    Request Id: 0x2a7ee1af
    Error Status: NO ERROR (0)
    Error Index: 0
    Object identifier 1: 1.3.6.1.2.1.1.4.0 (SNMPv2-MIB::sysContact.0)
    Value: STRING: Ben Rockwood

Each packet is referred to as a PDU in SNMP speak. The first packet (with the Ethernet, IP, and UDP headers removed) is a version 1 GET request with an OID to get and a community name of "public". Notice that the GET PDU has a Value field, but its NULL. The second packet is the response which you'll notice is identical to the GET PDU with 2 exceptions: the PDU type is RESPONSE instead of GET and the Value field is populated.

Given this example you can see that when you make a request you hand a PDU to an agent and say "Please fill this in".

Putting this into the context of the Net-SNMP C API, look at the snmp_PDU structure.

typedef struct snmp_pdu {

    long            version;
    int             command;
    long            reqid;
    ...
    long            errstat;
    long            errindex;
    ...
    netsnmp_variable_list *variables;
    ...
    u_char         *community;
    ...

The version value maps directly the value seen in the packet above. The command value is the PDU type as seen in the packet above. And you can also see the reqid in the packets above as the Request ID. The important point here is that when a PDU structure is manipulated using the API your actually preparing an SNMP packet for transmission.

Forming a request, such as a GET, can be done in several ways. One method is to create a single PDU per OID you wish to request. As we saw above this would mean sending one packet per OID and thus receiving one packet for each request. Another method, which is completely valid, is to pack a single GET PDU with multiple OIDs for retrieval. The following is a break capture of such an exchange as seen on the wire:

 

No.Time        Source      Destination Protocol Info
28 2.240789    10.10.1.110 10.10.1.224 SNMP     GET SNMPv...
Simple Network Management Protocol
    Version: 1 (0)
    Community: public 
    PDU type: GET (0)
    Request Id: 0x30549f5c
    Error Status: NO ERROR (0)
    Error Index: 0
    Object identifier 1: 1.3.6.1.4.1.318.1.1.1.1.1.1.0 (SNMP...
    Value: NULL
    Object identifier 2: 1.3.6.1.4.1.318.1.1.1.1.2.3.0 (SNMP...
    Value: NULL

No.Time        Source      Destination Protocol Info
41 2.328751    10.10.1.224 10.10.1.110 SNMP     RESPONSE SNM...
Simple Network Management Protocol
    Version: 1 (0)
    Community: public
    PDU type: RESPONSE (2)
    Request Id: 0x30549f5c
    Error Status: NO ERROR (0)
    Error Index: 0
    Object identifier 1: 1.3.6.1.4.1.318.1.1.1.1.1.1.0 (SNMP...
    Value: STRING: "Silcon DP340E"
    Object identifier 2: 1.3.6.1.4.1.318.1.1.1.1.2.3.0 (SNMP...
    Value: STRING: "SE0XXXXXXX   "

This exchange looks identical to the previous exchange but this time we've packed multiple (8 total, but I only show 2 for brevity) OIDs into the PDU. Hence, we send one packet and we are returned one packet.

Looking back at the PDU structure, it should not be obvious why there is no fixed value for the OID but instead an included netsnmp_variable_list pointer. This variable list is otherwise referred to as a VARLIST or VARBIND.

 

typedef struct variable_list netsnmp_variable_list;

struct variable_list {
   struct variable_list *next_variable;
   oid            *name;
   size_t          name_length;
   u_char          type;
   netsnmp_vardata val;
   size_t          val_len;
   oid             name_loc[MAX_OID_LEN];
   u_char          buf[40];
   void           *data;
   void            (*dataFreeHook)(void *);
   int             index;
};

So looking at the variable list we immediately notice that its a linked list, hence the ability to pack multiple values into a single OID.

We can pull all this information together to get a much clearer idea of how the API really works. As seen in the captures what we really need to do is create, send, receive and then process one or more PDUs; thats the basics. In order to do this we need to initialize a PDU structure, append one or more variables into the variable list and then send that PDU out. What we'll get back is a fully populated (we hope) PDU, but this time each OID in the variable list will have a value referenced by the data pointer.

Next lets look at the actual code.

Destination IP address for traps is usually configured via agent interface of configuration file and is generally  application-specific. It can be part of MIB.

The format of the trap message was changed in SNMPv2 and the PDU was renamed SNMPv2-Trap.

NEWS CONTENTS

• 20120703 : Receiving SNMP Traps in Nagios ( Receiving SNMP Traps in Nagios, Jul 03, 2012 )

Old News ;-)

[Jul 03, 2012] Receiving SNMP Traps in Nagios

Askar Ali Khan Blog

SNMP traps are alerts and notifications generated by SNMP-enabled devices. The traps con-tain information about the status or an event on an SNMP-enabled device. For example, an authentication event or the change in status of an interface on a router may generate an SNMP trap that is sent to a management station of some sort, such as HP OpenView, CiscoWorks, Nagios.

Pre-requisites:

1. Net-SNMP with snmptrapd configured.
2. SNMPTT, SNMP trap translator.
3. Nagios.
4. Mib definition files for the equipment or software you need to monitor.

Installing Net-SNMP packages:

The Net-SNMP package is available as a series of installable packages on many distributions. Indeed, it may already be installed on your system or you may be able to install it via your distribution's package management system, such as yum, apt, or the like. On Red Hat, SuSE, Debian, and Mandrake distributions, the required packages are called

• net-snmp
• net-snmp-libs
• net-snmp-utils.

Installing Net-SNMP packages on Centos 5.5

# yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-perl perl-Net-SNMP 
net-snmp-devel

Configuring and Running the snmptrapd Daemon

When incoming traps are received from the snmptrapd daemon, they are passed to the SNMPTT tool. The SNMPTT tool will then try to match the incoming trap against the collection of trap definitions that it has translated. If the trap matches, SNMPTT will see if the translated trap definition contains logic to output it to Nagios and execute that logic. The trap is then out-put to Nagios as a passive check result.

On Centos 5.5

# vi /etc/sysconfig/snmptrapd.options

#OPTIONS="-On -Lsd -c /etc/snmp/snmptrapd.conf -p /var/run/snmptrapd.pid"

OPTIONS="-On -Lsd -p /var/run/snmptrapd.pid"

Make sure to remove the -c /etc/snmp/snmptrapd.conf part, otherwise you will receive TRAP twice, as snmptrapd' is compiled with the default configuration file path being already set to '/etc/snmp/snmptrapd.conf'.

As quoted from SNMP Trap Translator documentation: "The -On is recommended. This will make snmptrapd pass OIDs in numeric form and prevent SNMPTT from having to translate the symbolic name to numerical form."

# vi /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler
disableAuthorization yes
#donotlogtraps yes

The traphandle directive tells the snmptrapd daemon how to handle incoming traps and where to send them. Adding the default option tells the daemon that this is the default way to handle all incoming traps. All traps will be sent to the snmptthandler script located in the /usr/sbin directory, the "disableAuthorization yes" tells to accept SNMP traps from all you can configure it to do authentication for detail refer to snmptrapd.conf manual.

Installing SNMPTT (SNMP Trap Translator)

You can get the SNMPTT tool from Sourceforge at http://snmptt.sourceforge.net/. This line shows how to download and unpack the SNMPTT tool:

Download snmptt_1.3.tgz which the latest version/stable release.

tar -zxvf snmptt_1.3.tgz

The SNMPTT package has no installation script, so a number of manual installations steps need to take place. First, copy the SNMPTT binaries to a suitable directory and mark them as executable. I recommend using the /usr/sbin directory

# cp snmptt snmptthandler /usr/sbin/ # chmod +x /usr/sbin/snmptt /usr/sbin/snmptthandler

I specified the snmptthandler binary as the value of the traphandle option in the snmptrapd.conf configuration file in the previous section. When a trap is received, this binary is executed by default and the trap sent to the snmptt daemon

Next, copy the SNMPTT configuration file, snmptt.ini, to the /etc/snmp directory and snmpttconvertmib utility

cp snmptt.ini /etc/snmp/
cp snmpttconvertmib /usr/sbin/

Also needed are a user and group to run the SNMPTT daemon as.

groupadd snmptt
adduser -g snmptt snmptt
chown snmptt:snmptt /etc/snmp/snmptt.ini

The SNMPTT tool also needs a spool directory to hold the incoming traps. I usually use the default directory of /var/spool/snmptt. It needs to be owned by the user and group that will run SNMPTT. Create and change the ownership of the directory like so

# mkdir /var/spool/snmptt
chown snmptt:snmptt /var/spool/snmptt

Finally, in order to start the SNMPTT tool, you can either execute it from the command line or use the init script provided with the package. On the following line

SNMPTT started in daemon mode:

# /usr/sbin/snmptt -daemon

Or copy the init script provided with the package, you can then add it to your startup process.

# cp snmptt-init.d /etc/init.d/snmptt

To start/stop/reload you can do with ..

/etc/init.d/snmptt start/stop/reload

Configuring SNMPTT

The first is configuring the /etc/snmp/snmptt.ini file. The file contains quite a large number of directives, but I'll only look at those relevant to the process of translating and transmitting the received traps to Nagios

mode = daemon
daemon_fork = 1
daemon_uid = snmptt
spool_directory = /var/spool/snmptt/
sleep = 5
dns_enable = 1
strip_domain = 1
log_enable = 1
syslog_enable = 0
exec_enable = 1
snmptt_conf_files = <
/etc/snmp/snmptt.conf
END

The sample snmptt.ini file contained in the SNMPTT package has detailed explanations of all the directives and options that you can specify. I recommend reading this file for further information and explanations about SNMPTT's configuration options

Tip: If you enable DNS resolution, I recommend you add all the hostnames that need to be resolved to the local /etc/hosts file on your host server. This prevents your DNS server from being a bottleneck or preventing SNMPTT from functioning if your DNS server is unavailable.

Compiling MIBs

You must gather all MIBs for monitored software, so you can feed SNMPTT with them. Compiling consists in extracting each OID of type "trap" and its associated comments, and generate a configuration file in SNMPTT format from these information.

Run the following command on each of your MIB files:

snmpttconvertmib --in= -- out=/etc/snmp/snmptt.co \
	--exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

The resulting SNMPTT configuration file will contain blocks (one per selected OID)

Catchall Trap Definition

SNMPTT also has a regular expression–matching capability that allows you to use an EVENT line that matches multiple incoming traps, a catchall trap definition. This means you don't need to define individual translated trap definitions for each possible incoming trap.

Catchall Trap Definition

EVENT CatchAll .1.* "SNMP Traps" Critical
FORMAT $D
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result "$r" 
	
"snmp_traps" 2 "$O: $1 $2 $3 $4 $5" 

I could also be more selective and select OIDs from a particular vendor or class of trap either using a wildcard or regular expression pattern matching. I've added a category called SNMP Traps and severity of Warning.

For example here is example to catchall traps from a specific vendor OID.

EVENT CatchAll .1.3.6.1.4.1.20916.* "Status Events" Normal
FORMAT A room-alert-4e-snmp-trap indicates that an alarm $*
EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $r "snmp_traps" 1 
"A room-alert-4e-snmp-trap indicates that an alarm $*"
SDESC
A room-alert-4e-snmp-trap indicates that an alarm
condition has occurred on the sensor indicated
by the alarmmessage variable.
Variables:
1: alarmmessage
EDESC

When done, add to SNMPTT configuration file /etc/snmp/snmptt.ini the path to compiled configuration files:

[...]
snmptt_conf_files = < 
/etc/snmp/snmptt.conf. 
END

Configuring Nagios

You will use passive checks to receive SNMP traps but they also will be volatiles. If ever two traps are received from the same host, the second one coming in before the first one was reset to OK, we want to be notified twice, although there is no state change. That's why we use a volatile service.

You might define (for example) a service template for SNMP traps, inheriting from a generic service template:

define service{
name generic-service 
active_checks_enabled 1 
passive_checks_enabled 1 ; Passive service checks are enabled/accepted
parallelize_check 1
	
obsess_over_service 1 
check_freshness 0 
notifications_enabled 1 ; Service notifications are enabled
event_handler_enabled 1 ; Service event handler is enabled
flap_detection_enabled 1 ; Flap detection is enabled
failure_prediction_enabled 1 ; Failure prediction is enabled
process_perf_data 1 ; Process performance data
retain_status_information 1 ; Retain status information across program restarts
retain_nonstatus_information 1 ; Retain non-status information across program restarts
is_volatile 0 ; The service is not volatile
check_period 24x7 ; The service can be checked at any time of the day
max_check_attempts 3 ; Re-check the service up to 3 times in order to determine 
its final (hard) state
normal_check_interval 10 ; Check the service every 10 minutes under normal conditions
retry_check_interval 2 ; Re-check the service every two minutes until a hard state 
can be determined
contact_groups admins ; Notifications get sent out to everyone in the 'admins' group
notification_options w,u,c,r ; Send notifications about warning, unknown, critical, 
and recovery events
notification_interval 60 ; Re-notify about service problems every hour
notification_period 24x7 ; Notifications can be sent out at any time
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
}

define service{
name trap-service
use generic-service
register 0
service_description snmp_traps
is_volatile 1
check_command check-host-alive ;Used to reset the status to OK when 'Schedule an 
immediate check of this service' is selected.
flap_detection_enabled 0 ; Flap detection is disabled
process_perf_data 0 ; Do not Process performance data
max_check_attempts 1 ; Leave as 1
normal_check_interval 1 ; Leave as 1
retry_check_interval 1 ; Leave as 1
passive_checks_enabled 1 ; Enables passive checks
check_period 24x7
notification_interval 31536000 ; Notification interval. Set to a very high number 
to prevent you from getting pages of previously received traps (1 year - restart 
Nagios at least once a year! - do not set to 0!).
active_checks_enabled 0 ; Prevent active checks from occuring as we are only using 
passive checks.
notification_options w,u,c ; Notify on warning, unknown and critical.
contact_groups sysadmins
}

define service{
host_name AVT-Room-Alert ; hostname is define /etc/hosts file
use trap-service
contact_groups sysadmins
}

TIP: You could also use a wildcard to create this service for all hosts or use the hostgroup_name directive to create the service for all members of a host group or groups.

I've defined the service as volatile and set the maximum check attempts to 1. This will cause Nagios to immediately set a HARD service state and trigger any configured notifications or event handlers. I've also configured it for passive checks only and disabled active checks.

Putting It All Together

The SNMPTT tool is called via the trap handler defined in the snmptrapd.conf configuration file I defined in the "Configuring and Running the snmptrapd Daemon" section.

This trap handler calls the /usr/sbin/snmptthandler script. The script reads the trap and then writes it to the spool directory defined in the spool_directory directive from the snmptt.ini configuration file. The script then exits.

From here the SNMPTT daemon takes over. It reads the trap from the spool file and searches for a match in its trap definitions. If it finds a match, it executes the EXEC statement in the matching trap definition. This EXEC statement sends the passive check result to the Nagios server using the submit_check_result script. The daemon then sleeps for the period specified in the sleep directive in the snmptt.ini file and checks the spool directory for additional traps; if it finds matches, it processes them and sends the check results to Nagios.

The Nagios server has to have host objects defined for every host that generates SNMP traps. Additionally, you need to define service objects for those hosts to receive the service check results. You should configure them to receive passive check results and as volatile services.

Posted by Askar Ali Khan

Aimen May 3, 2012 2:01 AM

Great tutorial, thanks a lot. SNMPTT is not well packaged in my opinion and can easily have an install script, but this tutorial was very clear and easy to follow. I had some package dependencies issue and had to install perl-Config-IniFiles to make it work. It might be a good idea to add the required perl packages to the Pre-requisites. Running #perl -MCPAN -e 'install Config::IniFiles' seems to resolve the compilation errors as well. This is the error I got when tried to start the daemon: Can't locate Config/IniFiles.pm in @INC (@INC c... at /usr/sbin/snmptt line 3894

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019