Pandora: Documentation en: SNMP traps Monitoring

From Pandora FMS Wiki
Jump to: navigation, search

Go back to Pandora FMS documentation index

1 Operations with SNMP Traps

1.1 Introduction

Network devices that support SNMP, such as switches, routers, servers, printers, or AP's can send alarms (or SNMP traps) when certain events occur, such as interface failure, CPU or network load is too high, a UPS changes status or a disk partition gets filled up. Each device has its own "collection" of possible events, which is reflected in a collection, called MIB, in this case, different from the MIB we used to query the device.

Trap-example.png

Pandora FMS features a TRAP Reception Console which allows TRAPs to be displayed which were sent by monitored objects which allows alerts to be added to those traps. SNMP traps are received by the operating system's daemon which starts the SNMP Server of Pandora FMS in the moment of server-startup. This server stores received traps in a log file under

/var/log/pandora /pandora_snmptrap.log

Traps are usually received in RAW format. Regarding numerical OIDs, that means unless an MIB is installed on the Operating System which is capable of resolving them, the Pandora FMS Enterprise SNMP Console enables rule creation for renaming numeric OIDs to alphanumeric ones or simple descriptive text strings (e.g. 'interface is down') in order to make working with TRAPs more intuitive. Pandora FMS allows loading trap MIBs of any sort in order to automatically define such rules.

First, you have to modify the following parameter in /etc/pandora/pandora_server.conf to enable the SNMP Console:

snmpconsole 1

If you want the traps to be translated (either the variable bindings or the Enterprise string), please enable the following options too (only Enterprise):

translate_variable_bindings 1
translate_enterprise_strings 1

Also, you need to set up the /etc/snmp/snmptrapd.conf file with the necessary parameters. For instance:

authCommunity log public
disableAuthorization yes

As for this configuration, traps will be created for the "public" community and won't need any authorization.

1.1.1 SNMPv3

SNMPv3 traps are rejected unless the sending user is added to /etc/snmp/snmptrapd.conf using the createUser directive. For example:

disableAuthorization yes
createUser -e 0x0102030405 snmpv3user SHA mypassword AES

Template warning.png

The engineID must be specified with the -e option. Otherwise, only SNMPv3 INFORMs will be received.

 


1.2 Access to the TRAP Reception Console

To use the trap reception console, please go to Monitoring > SNMP > SNMP Console, where you may take a look at the list of TRAPs which have been received so far. There is an eye-shaped icon which displays all the trap information. You're able to learn all detailed information regarding SNMP traps here.


Traps.JPG


For each trap, the following columns are going to be displayed:

Status

A green box appears if the trap is validated. It turns red if not.

SNMP Agent

The Agent which has sent the trap.

OID

The OID of the sent trap. A trap can only send one piece of data in this field.

Value

The value field of the sent trap. A trap can only send one piece of data in this field.

Custom OID, Custom Value

The customized fields, sent within the trap. They sometimes consist of very complex data which bears a specific logic, which sends the trap. A trap is able to send several types of data in this field.

Time Stamp

The elapsed time since the trap reception.

Alert

A yellow box appears if any alert was launched by this trap. It's gray if no alert was launched.

Action

The field for deleting or validating the trap.

Traps also bear the following colors, depending on the type:

  • Blue: Maintenance traps.
  • Purple: Information traps.
  • Green: 'normal' traps.
  • Yellow: 'warning' traps.
  • Red: 'critical' traps.

In the upper part of the trap console, the option named 'Toggle Filter' is displayed. By clicking on that option, the trap's filtering field options appear or disappear.


Alert.png


1.2.1 TRAP Validation

In order to effectively manage the traps it's possible to validate them, so the Administrator is able to distinguish between pending traps and the ones already seen.

To validate a trap, click on the green circle to the left of the trap. It is also possible to validate multiple traps by marking them and clicking on the "validate" button. The procedure is similar to how events work in Pandora FMS.


Traps2.JPG


1.2.2 TRAP Deletion

It is possible to delete traps once they have been edited, either individually or by multiple selection and the "Delete"action. To prevent traps from accumulating, there is a default setting option that automatically deletes traps older than 10 days.


Traps3.JPG


1.3 SNMP Trap Alerts

Pandora FMS also has an alert system for the SNMP traps that it receives. They are mainly based on filtering rules, searching for matches in all possible fields according to rules that we set up to trigger the alert. Before reading on, bear in mind you should can know more about Pandora's alerts [1]

1.3.1 Alert Creation

SNMP traps alerts have several fields that will be used to search for matches in SNMP traps received in the console. You can optionally use any fields you want to create more general or more specific rules as required:

  • Description: A combo box for an alert description.
  • Enterprise String: The main OID of the trap. It will look for the presence of the string. For example, if you're looking for a piece of the OID, we can use 1.21.34.2.3 and every OID that contains that one will be filtered, in a similar way of *1.21.34.2.3.* But there is NO need to use the * character.
  • Custom Value / OID: This element is going to search within the trap's 'value', 'custom OID', 'custom value' and in the rest of the TRAP fields. Regular expressions are supported here. For example, if we have a trap that sends the "Testing TRAP 225" string, we can search for any trap with the subchain "Testing TRAP" with the regular expression "Testing. *TRAP.*"
  • SNMP Agent: The agent's IP which has sent the trap. You're able to use a regular expression or a substring here, too.
  • Trap type: The filter by trap type. Accepted values are 'cold start', 'warm start', 'link down', 'link up', 'authentication failure' or 'other'. Most of the generated traps are usually "Other" type; if we don't specify anything, it will look for any type of trap.
  • Single value: It filters by trap value. In this example, it's .666. Please keep in mind this refers only to the MAIN OID value, not to any additional OIDs within the custom data section.
  • Variable bindings/Data #1-20: These are regular expressions which try to match the binding variables from '1' to '20'. If there is a match, the alert is fired. The value of the variable is stored in the corresponding '_snmp_fx_' macro (e.g. _snmp_f1_, _snmp_f2_, etc.). Although only twenty binding variables are able to search for matches, the _snmp_fx_ macros are set for all of them (_snmp_f11_, _snmp_f12_, etc.).
  • Field 1: Field to set the Field 1 alarm command parameter. This is the field that will be used in the case of choosing to generate an event, or the destination mail in the case of choosing an email action (if we want to overwrite the default email in the action). To fully understand how custom fields work in actions/alerts templates, read the documentation chapter that explains the alerts in Pandora FMS [2]
  • Field 2: Field to set the command parameter of the Field 2 alarm. In the case of sending an email, e. g. will be the subject of the message. If left blank, it would use what it had defined in the action.
  • Field 3: Field to set the command parameter of the Field 3 alarm. In the case of sending an email, it would be the text of the message. If left blank, it would use what it had defined in the action.
  • Min. Number of Alerts: The field to define the min. amount of traps which have to be received to trigger the alarm.
  • Max. Number of Alerts: Field where the maximum number of times the action will be executed in the given interval (or time threshold).
  • Time Threshold: The field for determining the time to elapse before resetting the alarm counter. This counter is the one which is getting used for the field named 'min. number of alerts'.
  • Priority: Combo where the alarm priority is set. The priorities of the alerts are different and have nothing to do with the priority of the traps, nor with the Pandora FMS events.
  • Alert Action: The combo box for selecting the action which is going to execute the alert. If you select an event, the normal event of an alert creation is not going to be created.
  • Position: The alerts with a lower position are evaluated first. If several alerts match with a trap, all matched alerts with the same position will be triggered. Although lower position alerts match with the trap, they will not be triggered.

1.3.1.1 Traps Alert Example

Let's suppose we get a trap like the next one:

Trap sample for alert.jpg

In this case, we have a main OID (. 1.3.6.1.4.1.2789.2005) that identifies a trap that can contain CPU overheating messages (we don't know if about anything else) but we know that in two variables it shows that the CPU heats up and the temperature of that CPU at that time, in variables 1 and 2 respectively. As we want to identify only the CPU heating traps, we match the Heat alert string in the first variable of the trap (remember that we have up to 20 for searching).

Defining the first part of the trap is simple, we only use the main OID to make the first and most important prefilter:

Trap alert definition 1.jpg

The second part of the trap definition is the one containing the essential part. In the first variable of the trap, we look for the "Heat alert" string if traps arrive with the main OID but in the first variable they do not contain that text string, the alert would not be triggered.

Trap alert definition 2.jpg

And finally, by choosing a "Pandora Event" type alert, we conform the message using the variables that contain the value of variables 1 and 2 of the trap received:

Trap alert definition 3.jpg

This way, when the alert goes off, the generated event will look like this:

Event result of alert.jpg

1.4 Working with environments with many traps

1.4.1 TRAP-Storm Protection

There are a couple of parameters on the server which are designed to protect the system against the arrival of a Trap Storm, originating from a single location. We're going to use the following settings in the file 'pandora_server.conf' for this:

  • snmp_storm_protection: The max. number of processed SNMP traps by the same source IP in a given interval (see below).
  • snmp_storm_timeout:: The interval in seconds for protection against an SNMP Trap Storm. During this interval, the system will only process 'snmp_storm_protection' type traps from the same source (IP).

Trap storm protection combined with traps filtering (see below) allows that if we receive hundreds of thousands of traps per day, we work with only a few thousand traps to eliminate redundant or unhelpful traps.

1.4.2 TRAP Filtering in the Server

Some systems receive a large number of traps from which only a small percentage is interesting to monitor. With Pandora FMS it is possible to filter the traps that the server receives in order to avoid unnecessary loading of the application.

From Administration>Manage SNMP Console>SNMP Filters you can define different filters. A trap that will marry any of them will be automatically discarded by the server:

Snmp filter editor new.png

The filter is applied as a regular expression on the trap entry in the SNMP log (default /var/log/pandora/pandora_snmptrap. log), which has the following fixed format:

%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n

Them being:

  •  %y: current year.
  •  %m: current month (numerical).
  •  %l: current day of the month.
  •  %h: current hour.
  •  %j: current minute.
  •  %k: current second.
  •  %a: Origin address (traps version 1 only).
  •  %N: OID.
  •  %w: trap type (numerical).
  •  %W: trap description.
  •  %q: trap sub-type (numerical).
  •  %v: List of variables separated by tab (custom OID).

For example, to filter all the traps from host 192.168.50.20 we could set the following filter:

Snmp filter example.png

Since more than one filter can be created simultaneously, the search will take into account those traps that meet all filtering conditions.

1.4.3 SNMP Trap stats

This view allows you to see statistics of traps, both by origin (IP) and by OID, this allows an effective management of the filters, by identifying the IP's that generate more traps and the OIDs that repeat more. The system displays statistics of traps for the last 30 days.

Captura de pantalla 2014-09-23 a la(s) 17.59.56.png

Captura de pantalla 2014-09-23 a la(s) 19.11.40.png

1.5 Customizing SNMP Traps

In order to give the operator a better understanding of the traps sent by the monitored devices, it's possible to either load the manufacturer's MIBs into Pandora FMS or to edit the traps to your liking.


Info.png

The functions mentioned so far are all features of the Pandora FMS Enterprise Version.

 


1.5.1 Renaming and Customizing Traps

We call the process "edit a trap" where it is allowed to "customize" the appearance of a trap in the console. If you look at the next capture, all your traps may be difficult to distinguish because of the cryptic information they contain.

To edit a trap, go to Operation > SNMP Console. Click the edit icon for the trap you want to customize:


Traps7.JPG


This way, when you find traps with the OID ". 1.3.6.6.1.4.1.2789.2005" you will see them as "Blue box sample". And it'll be easier to tell them apart. Its content (including the original OID) will remain unchanged.

"Custom OID" is a Perl compatible regular expression that will be matched against the part of the trap string containing variable bindings. It is generally not needed to translate a trap.

Template warning.png

"Custom OID" is not meant to be the whole variable binding string, which can be larger than its supported maximum length, but rather a regular expression that matches one or more variables.

 


Keep in mind that all previous traps will not change their appearance, this will start working with the new traps that enter the system from now on.

Finally, this is what a user-defined trap would look like:


Traps8.JPG


1.5.2 Loading the MIBs of the Manufacturer

This option is suitable for uploading traps MIBS (exclusively) and expanding Pandora's internal translation database, so that when a trap arrives, it will be automatically translated by its description.

To upload the Manufacturer's MIBs click on "Browse", choose the file that should be with txt extension and click on "Upload MIB".

Cima6.png

Once uploaded, the system is going to incorporate it to its trap library.

1.6 Associated Alerts to complex SNMP Traps

The previously described alerts are only going to be used where the trap is appropriately defined. It's always the same and it doesn't bear any relevant data to recover.

In certain situations, we however might find a trap which exhibits the following structure:

OID: .1.3.6.1.4.1.2789.2005
Value: 666
Custom data: 1.3.6.1.4.1.2789.2005.1 = STRING: "ID-00342" .1.3.6.1.4.1.2789.2005.2 = STRING: "Automated check"  
.1.3.6.1.4.1.2789.2005.3 = STRING: "NIC Offline" .1.3.6.1.4.1.2789.2005.4 = STRING: "4897584AH/345"

This is a 'complex' trap which contains complex data alongside an OID and a value, based on many other OIDs and values. In its complex part, a trap can contain a completely randomized structure which is based on OIDs or value pairs (e.g. counters, numerical, alphanumerical, dates, etc.).

Within the trap console, this trap would appear as in the picture below:


Traps9.JPG


If you carefully read the extended info (Custom data), there might be some pieces of useful data. In the first field, bearing the OID ending of '2005.1' looks like an identifier. The third field, bearing the OID ending of '2005.3' looks like an error message. Fields 2 and 4 don't look pretty useful, since they seem to be of unknown code to us.

Let's assume for a moment we could create an event from a trap, moving specific parts from the trap data into the text. Let's suppose we intend to build an event which contains the following information:

Chassis Alert: <error message> in device <identifier>

The challenge is to make an alert which produces a 'match' in those fields, obtaining the piece of data and using it to create a message in the alert later. We're able to perform this task using Pandora FMS and advanced regular expressions and selectors. You can find more info about regular expressions here.

The selectors are using brackets (), allowing us to 'copy' information by utilizing a search expression.

The regular expression to obtain the identifier would be as follows:

.*.1.3.6.1.4.1.2789.2005.1 \= STRING\: \"([0-9\-\_A-Za-z]+)\"

The regular expression to obtain the error message would be as follows:

.*.1.3.6.1.4.1.2789.2005.3 \= STRING\: \"([\sA-Za-z]+)\".*

Once we've obtained the data fields, we're required to use them in the alert. For this purpose, the special macros '_snmp_f1_', '_snmp_f2_' and '_snmp_f3_' are used. Using these macros doesn't make any sense outside any SNMP trap alert.

To build the message, we're going to use the following string:

Chassis Alert: _snmp_f2_ in device _snmp_f1_

The picture below shows how the complete alert is created.



Compex trap def1.png



This is how it would look from the perspective of the event viewer.



Compex trap def2.png



In order to successfully create this type of alert, an extensive knowledge of regular expressions is required, since a simple blank space, a symbol or another character in the wrong location could make it work inappropriately. Please keep in mind that SNMP alerts imply the use of regular expressions. An easier way to establish an alert by regular expressions would be the following:



Compex trap def4.png



This alert would be 'compatible' to the previous one in such a way that two different events could be displayed for the same trap:



Compex trap def5.png




1.6.1 Additional Example

This additional example uses an email alert to send information about the interface name each time you receive a specific trap. You're going to produce an email containing a device name, IP and interface name as received in the trap.

This is a trap, received from a switch:



Another trap alert sample.png




This is the received mail.



Trap email.png



This is the trap definition.



Another snmp trap sample.png



1.7 TRAP Association to the Rest of Pandora FMS Alerts and SNMP Agent Trap Forwarding

The alerts defined on traps are completely independent from Pandora's Alert Engine, so correlations of them trigger an alert if the temperature reaches 29 degrees and the trap for secondary power supply is on. This type of alerts cannot be displayed since they're -possibly- not associated to any modules of Pandora FMS. Therefore, the trap console monitoring cannot be related to elements such as reports or maps.



Special ModuleModule SNMPTrap,with the trap sent from the SNMP console: Snmptrap agent.png

In order to achieve this, we've created a method called 'Agent SNMP Trap Forwarding'. This server-wide option forwards the trap to a special agent's module named 'SNMPTrap' as a text string, but only if the trap's originating IP address is defined as the agent's IP. Whenever this occurs, the trap arrives as a text line to the agent within that module, which is one that's only defined on arrival of the first trap.

Text alerts can be specified within that module, these being completely standard, just as any other module. This enables a customization of SNMP Monitoring in order for certain traps from certain origins to be treated as yet another type of module, which are thereby integrated into the rest of the monitoring, including Alert Correlation.



SNMPTrap data sample': Snmptrapforward2.png

This is an Enterprise feature which could be activated on the main setup screen as shown below.



Configuration option to activate the trap forward to the agents:
Snmptrap agent forwardsetup.png

If this setting is changed, the Pandora FMS Server must be restarted to enable it.

Another solution is to mount an alert on the trap to activate an agent's module. If the trap has a '1' in a certain log file, there is an agent reading that file, ready to run it when it finds this '1'. In this way, the module is going to be triggered if the desired trap arrives and the correlation can be established, based on the arrived trap.

1.8 External SNMP Trap Handler

The SNMP console is made for the sole purpose of obtaining traps. It only processes TRAPs as individual items. One trap is able to contain a lot of information. Sometimes, you have a situation when the only monitoring we're able to conduct is based on traps. For doing so, we might choose to post-process the obtained information of one trap by an external script working as a plug in.

To process the data of one trap in detail, you may send all contained information to a script as an alert result. I have used the trap shown below for the example. It's the trap view as it would be in Pandora's SNMP console log:

2010-08-26 12:01:46 pandora 10.201.246.2 .1.3.6.1.4.1.1722 .1.3.6.1.4.1.1722.2.10.1.1.1 233 .1.3.6.1.4.1.1722.2.10.1.1.3 = STRING: 
AIX_Software_Failure .1.3.6.1.4.1.1722.2.10.1.1.2 = STRING: 08 25 2010 08:23:43:697685 .1.3.6.1.4.1.1722.2.10.1.1.8 = STRING: 1: A 
software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource 
SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED. .1.3.6.1.4.1.1722.2.10.1.1.6 = STRING: 8 
.1.3.6.1.4.1.1722.2.10.1.1.11 = STRING: An application may not work properly .1.3.6.1.4.1.1722.2.10.1.1.10 = STRING: An application 
may not work properly .1.3.6.1.4.1.1722.2.10.1.1.12 = INTEGER: 4 .1.3.6.1.6.3.1.1.4.3.0 = OID: .1.3.6.1.4.1.1722



Snmp plugin1.png



Snmp plugin2.png


On the screen shots below, you can see how a special alert is created. It executes a script which contains the complete content of the trap (_data_) and shows how an SNMP type of alert is created. In such a case, it has been mapped for the specific OID (.1.3.6.1.4.1.1722.2.10.1.1.1), although it could have been more general, e.g. (.1.3.6.1.4.1.1722) to call the script when there would be any type of traps like these (.1.3.6.1.4.1.1722, I suppose it would be part of the AIX specific MIB).

A script which processes this data is executed. It also 'analyzes' the trap to write data directly to Pandora FMS by creating an XML file, moving it to '/var/spool/pandora/data_in' as data, as if it had come from an agent. A basic script for this case would be one to generate complex information. We already possess enough information regarding this trap, which constitutes the following:

  • The originating IP
  • The main event (cold start)
  • The secondary events (descriptives): AIX_Software_Failure, 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly

When designing a script which 'parses' each of these data, e.g. 'miscript.pl' and storing it under '/var/spool/pandora/data_in' in the XML file along with a generic name and one random number, e.g. 'snmp_gateway.31415.data'.

The generated XML is supposed to look like the one on the picture below.


<?xml version='1.0' encoding='ISO-8859-1'?>
<agent_data description='' group='' os_name='aix' os_version='' interval='300' version='3.1(Build 100608)' timestamp='2010/08/26 12:20:26' agent_name='10.201.246.2'>
  <module>
    <name><![CDATA[Critical_Event]]></name>
    <description><![CDATA[]]></description>
    <type>async_proc</type>
    <data><![CDATA[1]]></data>
  </module>
<module>
    <name><![CDATA[events]]></name>
    <description><![CDATA[]]></description>
    <type>generic_string</type>
    <datalist>
      <data><value><![CDATA[AIX_Software_Failure]]></value></data>
      <data><value><![CDATA[A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC.]]></value></data>
      <data><value><![CDATA[Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.]]></value></data>
    </datalist>
  </module>
</agent_data>

The application is endless, but any script should be customized and it would have a very dynamic structure. Under a lot of systems, the information which gets received doesn't solely consist of text. It's also numerical and therefore able to feed numerical information to the modules in order to represent e.g. graphics, etc. Note that the data generated in XML should always be asynchronous.

1.8.1 Practical Example: ESX Monitoring using Traps

One of the most problematic things to monitor is the monitoring of distributed infrastructure. It gets even harder if each version changes its implementation to gather information (like VMware or ESX). In this chapter, we're going to explain how to monitor ESX systems by using an external SNMP Trap Handler.

ESX traps consist of the following data:

.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "c7000-06-01.tsm.inet"      .1.3.6.1.4.1.6876.4.3.303 = ""  
.1.3.6.1.4.1.6876.4.3.304 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.305 = STRING: "Yellow"    .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host 
cpu usage - Metric Usage = 1%"
.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "dl360-00.tsm.inet" .1.3.6.1.4.1.6876.4.3.303 = ""  
.1.3.6.1.4.1.6876.4.3.304 = STRING: "Yellow".1.3.6.1.4.1.6876.4.3.305 = STRING: "Green"     .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host 
memory usage - Metric Usage = 84%"
.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = ""  .1.3.6.1.4.1.6876.4.3.303 = ""  .1.3.6.1.4.1.6876.4.3.304 = 
STRING: "Red"       .1.3.6.1.4.1.6876.4.3.305 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Datastore usage on disk - Metric 
Storage space actually used = 55%"

As you can see, traps could be used to gather information from the CPU, the hard drive or the memory. The general idea behind the trap handler is to write a small script which is able to 'understand' the trap and to create an XML file which emulates a software agent. It's recommended to write a trap handler for each technology. The process of doing so is pretty common and explained in four steps:

  1. Please create the handler script. You may base your work on the script which is provided below.
  2. Please create an alert command.
  3. Please create an alert action using the previous command. You can add some custom options for each 'destination' agent you want (if you possess several farms of ESX, you're probably going to like to host the data on different agents).
  4. Please create an SNMP trap-alert which maps the enterprise OID (information trap contains for all kind of this specific technology) and / or the source trap IP address.

Now we're going to show how to create the trap-handler script.

1.8.1.1 Step 1 - The Trap Handler Script (esx_trap_manager.pl)

#!/usr/bin/perl
# (c) Sancho Lerena 2010 <[email protected]>
# Specific Pandora FMS trap collector for ESX 

use POSIX qw(setsid strftime);

sub show_help {
	print "\nSpecific Pandora FMS trap collector for ESX\n";
	print "(c) Sancho Lerena 2010 <[email protected]>\n";
	print "Usage:\n\n";
	print "   esx_trap_manager.pl <destination_agent_name> <TRAP DATA>\n\n";
	exit;
}

sub writexml {
	my ($hostname, $xmlmessage ) = @_;
	my $file = "/var/spool/pandora/data_in/$hostname.".rand(1000).".data";

	open (FILE, ">> $file") or die "[FATAL] Cannot write to XML '$file'";
	print FILE $xmlmessage;
	close (FILE);
}

if ($#ARGV == -1){
	show_help();
}

$chunk = "";

# The first parameter is always the destination host for the virtual server.
$target_host = $ARGV[0];

foreach $argnum (1 .. $#ARGV) {
	if ($chunk ne ""){
		$chunk .= " ";
	}
	$chunk .= $ARGV[$argnum];
}

my $hostname = "";
my $now = strftime ("%Y-%m-%d %H:%M:%S", localtime());
my $xmldata = "<agent_data agent_name='$target_host' timestamp='$now' version='1.0' os='Other' os_version='ESX_Collectordime ' interval='9999999999'>";

if ($chunk =~ m/.1.3.6.1.4.1.6876.4.3.302 \= STRING\: ([A-Za-z0-9\-\.]*)\s\.1/){
	$hostname = "_".$1;
}

if ($chunk =~ m/Host cpu usage \- Metric Usage \= ([0-9]*)\z/){
	$value = $1;
	$module_name = "CPU_OCUPADA$hostname";
}

if ($chunk =~ m/Host memory usage \- Metric Usage = ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "MEMORIA_OCUPADA$hostname";
}

if ($chunk =~ m/Datastore usage on disk \- Metric Storage space actually used \= ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "DISCO_OCUPADO$hostname";
}

$xmldata .= "<module><name>$module_name</name><type>async_data</type><data>$value</data></module>\n";

$xmldata .= "</agent_data>\n";
writexml ($target_host, $xmldata);


1.8.1.2 Step 2 - Creating the Alert Command

In this example, I've put the command script in '/tmp' to put in a safer place, and to be sure it's executable (chmod 755):



Trap handler step2.png


1.8.1.3 Step 3 - Creating the Alert Action

We're going to create a specific action to send all information to specific trap agents. In this case, all information is going to be sent to an agent named 'WINN1247VSR'. The above mentioned command accepts the name of the agent as a parameter which is going to receive all the information (ESX Virtual Center), and 'chunks off' " the data from the trap, which can be unlimited and also includes all the information you're sending to the trap.



Trap handler step3.png


1.8.1.4 Step 4 - Creating the SNMP Alert

You may set alert traps by using the action you've just created.



Trap handler step4.png


In order to process all the traps the of ESX type by using the specific OID '.1.3.6.1.4.1.6876.4.3.301' to map the ESX type traps. You also have the option of filtering by source IP for each virtual center by filtering for the originating IP address (which is contained in the trap).

1.8.1.5 Data Visualization

This is an example of how the information is going to look. You may manage them as standard modules by this data.



Trap handler step5.png


1.9 SNMP trap forwarding ( > v5.0 )

With Pandora FMS, it's also possible to forward SNMP traps to an external host by enabling the token named snmp_forward_trap within the server configuration file.

1.9.1 Configuration Example to forward Traps by SNMP v1

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 1
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.9.2 Configuration Example to forward Traps by SNMP v2c

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 2c
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.9.3 Configuration example to forward Traps by SNMP v3

This example is particularly challenging because of the implied knowledge regarding SNMP v3 traps. We are considering the remote SNMP agent to be defined under snmp_forward_ip. It contains the following entry in its file under '/etc/snmp/snmptrapd.conf':

createUser -e 0x0102030405 myuser MD5 mypassword DES myotherpassword

The Pandora server configuration file should look like this:

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 3
snmp_forward_secName myuser
snmp_forward_engineid 0x0102030405
snmp_forward_authProtocol MD5
snmp_forward_authPassword mypassword
snmp_forward_privProtocol DES
snmp_forward_privPassword myotherpassword
snmp_forward_secLevel authNoPriv

If you'd like to learn more about it, you can find more info here.

1.10 Independent Management of the snmptrapd daemon

If, for some reason, you would like to manage the snmptrapd daemon independently from Pandora FMS (stop or start it independently from the main Pandora FMS daemon) then you need to take into account several things:

1. The snmpconsole parameter must be active for the pandora fms server.

2. The logs configured in the pandora fms server must be the same as the ones in the independent call to snmptrapd.

3. The call to snmptrap must be in a specific format the standard system call can't be used. That call must be like the following (the parameter: -A is very important!):

/usr/sbin/snmptrapd -A -t -On -n -a -Lf /var/log/pandora/pandora_snmptrap.log -p /var/run/pandora_snmptrapd.pid --format1=SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n --format2=SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

4. The snmptrapd token must be configured into the pandora fms configuration file:

snmp_trapd manual

5. To enable this functionality you need to complete the following procedure::

  • Change the configuration in /etc/pandora/pandora_server.conf
  • Stop the Pandora FMS server.
  • Check that the snmptrapd process isn't running (if it is running you can wait till the process stops or kill it)
  • Start snmptrapd manually (with the format indicated above).
  • Start the Pandora FMS Server.

1.10.1 Management of the trap log file

The snmptrapd process can be stopped and started without having to stop or start the pandora server process if the pandora_snmptrap.log.index and pandora_snmptrap.log haven't been modified. If those files are modified it is necessary to restart the pandora server. If you need to rotate externally the traps log files then you should restart the pandora server, after deleting the two files previously mentioned.

1.11 SNMP trap buffering

If SNMP traps are sent to an external manager through an unreliable connection some information will be lost. Pandora FMS allows you to instead forward traps from a local snmptrapd to your Pandora FMS Server in a reliable way.

1.11.1 Architecture

Remote snmp trap schema.jpg

  • An SNMP agent sends traps to a local snmptrapd.
  • A local Pandora FMS agent reads traps from snmptrapd's log file and sends them to the designated Pandora FMS Server inside an XML data file, saving it to the XML buffer and retrying at a later time if necessary.
  • The Data Server reads traps from XML data files and dumps them to a plain text file.
  • The SNMP Console processes traps from the plain text file.

Template warning.png

Having the SNMP Console directly process traps from snmptrapd's log file is more efficient. This setup should only be used if direct connectivity or reliability are a concern.

 


1.11.2 Prerequisites

  • A local snmptrapd that is receiving traps.
  • A local Pandora FMS Agent.
  • A Pandora FMS installation.

1.11.3 Configuration

1.11.3.1 snmptrapd

Edit the file /etc/snmp/snmptrapd.conf and make sure it is logging to a file in a format compatible with Pandora FMS (you can change the name of the log file if necessary):

[snmp] logOption f /var/log/snmptrapd.log
format1 SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n
format2 SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

1.11.3.2 Pandora FMS Agent

We will use the grep_snmptrapd plugin that comes with the Pandora FMS agent to read data from snmptrapd's log file.

Edit the local agent's configuration file, /etc/pandora/pandora_agent.conf, and add the following line, adjusting the path of snmptrad's log file if necessary:

module_plugin grep_snmptrapd /var/log/snmptrapd.log

1.11.3.3 Pandora FMS Server

We need to tell the SNMP Console to process traps from an external log file that will be written by the Data Server.

Edit the server's configuration file, /etc/pandora/pandora_server.conf and:

  • Make sure the SNMP Console is enabled:
snmpconsole 1
  • Make sure the Data Server is enabled:
dataserver 1
  • Configure an external SNMP log file. If it does not exist, the SNMP Console will create it:
snmp_extlog /var/log/pandora/pandora_snmptrap.ext.log

Template warning.png

snmp_extlog can be any file writable by the Pandora FMS Server, but it must be different from snmp_logfile (defined in /etc/pandora/pandora_agent.conf too).

 


1.12 Trap Generator

This tool is used to generate traps that we will be able to observe later in the SNMP console, where if we click on the generated trap it will take us to the agent that contains it and we will be able to modify it freely.

Generador de traps.png


In order to be able to configure correctly the trap generator we will have to fill in the following fields:

  • Host Address: This is the destination IP to which we are going to send the trap.
  • Community: Where we will set the password of the SNMP community we are trying to access with the trap generator.
  • Enterprise String: The OID (Object Identifier) of the trap must be configured here. For example: 1.3.6.1.2.1.2.2.1.8
  • Value: The value that we want to give the trap and that will then appear as Data.
  • SNMP Agent: We have to insert the agent where we are going to simulate the trap, putting obligatorily the IP of the same one.
  • SNMP Type: We will have to choose a SNMP type between the following options:
    • Cold Start: Indicates that the agent has been started or restarted.
    • Warm Start: Indicates that the agent configuration has been modified.
    • Link down: Indicates that the communication interface is out of service (inactive).
    • Link up: Indicates that a communication interface has been activated.
    • Authentication failure: Indicates that the agent received a request from an unauthorized NMS (community-controlled)
    • EGP neighbor loss: Indicates that on systems where routers using the EGP protocol, a nearby host is out of service.
    • Enterprise: In this category you will find all the new traps. Including traps from suppliers.

Next, we can see an example of a trap configuration that will be sent to our localhost, with the agent 192.168.10.2, OID 1.2.3.4, value 20 and type "Link down":

Generador de traps ejemplo.png


Go back to Pandora FMS documentation index