Pandora: Documentation en: SNMP traps Monitoring

From Pandora FMS Wiki
Jump to: navigation, search

Go back to Pandora FMS documentation index

1 Operations with SNMP Traps

1.1 Introduction

Pandora FMS features a TRAP Reception Console which allows TRAPs to be displayed which were sent by monitored objects which allows alerts to be added to those traps. SNMP traps are received by the operating system's daemon which starts the SNMP Server of Pandora FMS in the moment of server-startup. This server usually stores traps in a log file under '/var/log/pandora /pandora_snmptrap.log'.

Traps are usually received in RAW format. Regarding numerical OIDs, that means unless an MIB (Management Information Base) is installed on the Operating System which is capable of resolving them, the Pandora FMS Enterprise SNMP Console enables rule creation for renaming numeric OIDs to alphanumeric ones or simple descriptive text strings (e.g. 'interface is down') in order to make working with TRAPs more intuitive. Pandora FMS allows loading trap MIBs of any sort in order to automatically define such rules.

First, you have to modify the following parameter in /etc/pandora/pandora_server.conf to enable the SNMP Console:

snmpconsole 1

If you want the traps to be translated (either the variable bindings or the Enterprise string), please enable the following options too (only Enterprise):

translate_variable_bindings 1
translate_enterprise_strings 1

Also, you need to set up the /etc/snmp/snmptrapd.conf file with the necessary parameters. For instance:

authCommunity log public
disableAuthorization yes

As for this configuration, traps will be created for the "public" community and won't need any authorization.

1.2 Access to the TRAP Reception Console

To use the trap reception console, please go to Monitoring > SNMP > SNMP Console, where you may take a look at the list of TRAPs which have been received so far. There is an eye-shaped icon which displays all the trap information. You're able to learn all detailed information regarding SNMP traps here.

Snmp trap sample.png

For each trap, the following columns are going to be displayed:

Status: A green box appears if the trap is validated. It turns red if not.
SNMP Agent: The Agent which has sent the trap. An address of the agent is obtained from the 'agent-addr' field of the PDU (Protocol Data Unit) if the sent trap is of the SNMPv1 type. If not, the source address of the packet is going to be used.
OID: The OID of the sent trap. If the sent trap is of the SNMPv1 type, the 'enterprise' field of the PDU is going to be used. If not, the 'snmpTrapOID' field is going to be used.
Value: The value field of the sent trap. A trap can only send one type of data in this field. It's not going to be used in the SNMPv2 trap.
Custom OID, Custom Value: The customized fields, sent within the trap. They sometimes consist of very complex data which bears a specific logic, which sends the trap. A trap is able to send several types of data in this field.

Time Stamp: The elapsed time since the trap reception.
Alert: A yellow box appears if any alert was launched by this trap. It's gray if no alert was launched.
Action: The field for deleting or validating the trap.

Traps also bear the following colors, depending on the type:

  • Blue: Maintenance traps.
  • Purple: Information traps.
  • Green: 'normal' traps.
  • Yellow: 'warning' traps.
  • Red: 'critical' traps.

1.2.1 TRAP Filtering

In the upper part of the trap console, the option named 'Toggle Filter' is displayed. By clicking on that option, the trap's filtering field options appear or disappear.

Alert.png

Within the trap console, it's possible to filter by the following fields:

  • Agent: A combo box where Pandora agents are displayed.
  • OID: A combo box where OIDs are displayed.
  • Alert: A combo box to select either triggered or non-triggered alerts.
  • Search value: A combo box to input any descriptive text.
  • Severity: A combo box where the different trap types are displayed, e.g. 'maintenance', 'information', 'severity', 'warning', 'critical', etc.
  • Status: A combo box to select between 'SNMP validated alert', 'SNMP not validated alert' or 'all'.
  • Free search: A search field to search for any alphanumeric field within the trap.
  • Type: A type filter, intended to be used within SNMP alerts. Accepted values are: 'cold start', 'warm start', 'link down', 'link up', 'authentication failure' or 'other'.

On top of these search fields, there is an option named 'Block size of pagination' which allows the amount of displayed traps per page to be defined.

1.2.2 TRAP Validation

In order to effectively manage the traps it's possible to validate them, so the Administrator is able to distinguish between pending traps and the ones already seen.

In order to validate a trap, please click on the green circle on the left of the trap.

Trap1.png

It's also possible to validate multiple traps by marking them all and to click on the green 'validate' button.

Trap2.png

1.2.3 TRAP Deletion

Furthermore, it's possible to delete traps once they have been attended to. To delete a trap, please click on the red cross on the left of the trap.

Borrar1.png

1.3 SNMP Trap Alerts

It's possible to associate an alert to a trap, so Pandora FMS warns in case of an arrival of a specific one. To manage alerts which are associated to traps, please click on 'Alerts' and 'SNMP Alerts'. SNMP traps have nothing to do with the rest of the system alerts, even if both reuse the action system.

I think you've already noticed the absence of an 'email template' here. The mechanism for conditions and fields (which are getting passed as a parameter to the command) differs from a normal alert. The alert fields within the SNMP alert are going to overwrite the defined fields under 'alert actions' - and there are no recovery alerts within SNMP alerts. If you receive a trap which notifies you about a solved situation, you're required to manage it as a different trap. There is no way to both 'connect' and to 'close' the alert.

1.3.1 Alert Creation (Pandora FMS versions 5 and above)

To create an alert associated to a trap, please go to 'Alerts' and 'SNMP Alerts' and click on 'Create'.

The SNMP alert traps come with several fields which could be used to filter out data within the SNMP trap. The fields (which could be used both separately and in combination) are the following:

  • Description: A combo box for an alert description.
  • Enterprise String: The main OID of the trap. It's possible to use a regular expression. If a regular expression is not used, it will look for the presence of the string. For example, if you're looking for a piece of the OID, we can use 1.21.34.2.3 and every OID that contains that one will be filtered, in a similar way of *1.21.34.2.3.* But there is NO need to use the * character.
  • Custom Value / OID: This element is going to search within the trap's 'value', 'custom OID', 'custom value' and in the rest of the TRAP fields. Regular expressions are also supported here.
  • SNMP Agent: The agent's IP which has sent the trap. You're able to use a regular expression or a substring here, too.
  • Trap type: The filter by trap type. Accepted values are 'cold start', 'warm start', 'link down', 'link up', 'authentication failure' or 'other'.
  • Single value: The Filter by trap value. In this example, it's '.666'. Please keep in mind this refers only to the MAIN OID value, not to any additional OIDs within the custom data section.
  • Custom OID / Data #1-20: These are regular expressions which try to match the binding variables from '1' to '20'. If there is a match, the alert is fired. The value of the variable is stored in the corresponding '_snmp_fx_' macro (e.g. _snmp_f1_, _snmp_f2_, etc.). Although only twenty binding variables are able to search for matches, the _snmp_fx_ macros are set for all of them (_snmp_f11_, _snmp_f12_, etc.).
  • Field 1: The first field to set the parameters for the alert's command. This could also feature as an event description if you select the event action.
  • Field 2: The second field to set the parameters for the alert's command.
  • Field 3: The third field to set the parameters for the alert's command.
  • Min. Number of Alerts: The field to define the min. amount of traps which have to be received to trigger the alarm.
  • Max. Number of Alerts: The field to define the max. amount of times the action will be executed.
  • Time Threshold: The field for determining the time to elapse before resetting the alarm counter. This counter is the one which is getting used for the field named 'min. number of alerts'.
  • Priority: The combo box for establishing the alarm priority.
  • Alert Action: The combo box for selecting the action which is going to execute the alert. If you select an event, the normal event of an alert creation is not going to be created.
  • Position: The alerts with a lower position are evaluated first. If several alerts match with a trap, all matched alerts with the same position will be triggered. Although lower position alerts match with the trap, they will not be triggered.

Snmp trap alert.png

You may use up to 20 variables to conduct the filtering and reuse them for macros later. They don't have to follow a specific order. The position of the variable is defined in the field's preceding value. If we intend to create an alert, seeking the values ​'one' in the first variable received by the trap, 'three' in the third variable and the same for 'five' and 'seven', the configuration might look like the one displayed in the picture below.

Alerta snmp2.png

We can use the value of the variables in macros _snmp_f1_ coincidence .. so _snmp_f7_ to define the alert, the alert action allows us to use these macros:

Alerta snmp3.png

If an alert is configured to use an event, the result would be an event like the one shown in the picture below.

Snmp trap alert event.png

The generated alert (an internal audit) is going to have a text like this one: "SNMP Alert of 192.168.5.2 with OID .1.3.6.1.4.1.2789.2005 Binding 1: 'one' Binding 3: 'three' Binding 5: 'five' Binding 7: 'seven'." So if the trap has 200 variables, you can use up to 20 filter variables (bindings) and take the value of up to 20 variables - regardless if they are in the 10th, 50th or 170th position.

Once the fields are filled out, please click on 'Create'.

1.4 TRAP-Storm Protection

There are a couple of parameters on the server which are designed to protect the system against the arrival of a Trap Storm, originating from a single location. We're going to use the following settings in the file 'pandora_server.conf' for this:

  • snmp_storm_protection: The max. number of processed SNMP traps by the same source IP in a given interval (see below).
  • snmp_storm_timeout:: The interval in seconds for protection against an SNMP Trap Storm. During this interval, the system will only process 'snmp_storm_protection' type traps from the same source (IP).


1.5 SNMP Trap stats

From 5.1 SP1 version, there is a new section which shows detailed, realtime stats of incoming SNMP Traps. It shows traps grouped by top 25 Source IP addresses and top 25 OIDs. This should help you to identify noisy sources and filter them using the filtering option. The stat system uses the last 30 days to get the stat information.

Captura de pantalla 2014-09-23 a la(s) 17.59.56.png

Captura de pantalla 2014-09-23 a la(s) 19.11.40.png

1.6 Customizing SNMP Traps

In order to give the operator a better understanding of the traps sent by the monitored devices, it's possible to either load the manufacturer's MIBs into Pandora FMS or to edit the traps to your liking.


Info.png

The functions mentioned so far are all features of the Pandora FMS Enterprise Version.

 


1.6.1 Renaming and Customizing Traps

The editing of traps is a process where the operator is allowed to customize the appearance of a trap within the console. If you take a look at the screen shot below, it might be the case that all of your traps are difficult to distinguish due to the received mess they contain as data (as shown there). If we could rename them, we could identify them as 'blue box' instead of a huge amount of encrypted data which is useless to the human eye.

To edit a trap, please go to 'Operation' and 'SNMP Console' and click on the OID field of the desired trap.

Cima2.png

In this moment, a page like the following appears:

Cima3.png

If the SNMP Console is going to find traps by the OID of ".1.3.6.1.4.1.2789.2005", it's going to display them as 'blue box sample', because they are going to be a lot more readable by the human eye in this way. The content (which is included the original OID) is not going to change.


Info.png

Please keep in mind that all the older traps are not going to change their appearance. This feature is only going to work along with new incoming traps in the system from the moment this feature was enabled.

 


New traps can also be created from scratch by the 'create trap' option. To do so, you're only required to go to the Trap Editor as shown on the screen shot below. We're also able to alter or to remove a trap definition in there.

Snmp trap editor.png

This is how a user-redefined trap would look:

Snmp custom view.png

1.6.2 Loading the MIBs of the Manufacturer

Info.png

The functions mentioned so far are all features of Pandora FMS Enterprise Version.

 


This option is only useful for uploading MIB traps and to provide more data for the Pandora FMS Translation Database. If a trap is received, it's going to be automatically translated by its description.

To upload the manufacturer's MIBs, please click on 'Examine', pick the file that should be with a '.txt' extension and click on 'Upload MIB'.

Cima6.png

Once it has been uploaded, the system is going to incorporate it to its trap library.

1.7 Associated Alerts to complex SNMP Traps

The previously described alerts are only going to be used where the trap is appropriately defined. It's always the same and it doesn't bear any relevant data to recover.

In certain situations, we however might find a trap which exhibits the following structure:

OID: .1.3.6.1.4.1.2789.2005
Value: 666
Custom data: 1.3.6.1.4.1.2789.2005.1 = STRING: "ID-00342" .1.3.6.1.4.1.2789.2005.2 = STRING: "Automated check"  
.1.3.6.1.4.1.2789.2005.3 = STRING: "NIC Offline" .1.3.6.1.4.1.2789.2005.4 = STRING: "4897584AH/345"

This is a 'complex' trap which contains complex data alongside an OID and a value, based on many other OIDs and values. In its complex part, a trap can contain a completely randomized structure which is based on OIDs or value pairs (e.g. counters, numerical, alphanumerical, dates, etc.).

Within the trap console, this trap would appear as in the picture below:

Compex trap def3.png

If you carefully read the extended info (Custom data), there might be some pieces of useful data. In the first field, bearing the OID ending of '2005.1' looks like an identifier. The third field, bearing the OID ending of '2005.3' looks like an error message. Fields 2 and 4 don't look pretty useful, since they seem to be of unknown code to us.

Let's assume for a moment we could create an event from a trap, moving specific parts from the trap data into the text. Let's suppose we intend to build an event which contains the following information:

Chassis Alert: <error message> in device <identifier>

The challenge is to make an alert which produces a 'match' in those fields, obtaining the piece of data and using it to create a message in the alert later. We're able to perform this task using Pandora FMS and advanced regular expressions and selectors. You can find more info about regular expressions here.

The selectors are using brackets (), allowing us to 'copy' information by utilizing a search expression.

We use the regular expression to obtain the identifier. The error message would be the following:

Identifier

.*.1.3.6.1.4.1.2789.2005.1 \= STRING\: \"([0-9\-\_A-Za-z]+)\"

Error message

.*.1.3.6.1.4.1.2789.2005.3 \= STRING\: \"([\sA-Za-z]+)\".*

Once we've obtained the data fields, we're required to use them in the alert. For this purpose, the special macros '_snmp_f1_', '_snmp_f2_' and '_snmp_f3_' are used. Using these macros doesn't make any sense outside any SNMP trap alert.

To build the message, we're going to use the following string:

Chassis Alert: _snmp_f2_ in device _snmp_f1_

The picture below shows how the complete alert is created.

Compex trap def1.png

This is how it would look from the perspective of the event viewer.

Compex trap def2.png

In order to successfully create this type of alert, an extensive knowledge of regular expressions is required, since a simple blank space, a symbol or another character in the wrong location could make it work inappropriately. Please keep in mind that SNMP alerts imply the use of regular expressions. An easier way to establish an alert by regular expressions would be the following:

Compex trap def4.png

This alert would be 'compatible' to the previous one in such a way that two different events could be displayed for the same trap:

Compex trap def5.png


1.7.1 An Additional Example

This additional example uses an email alert to send information about the interface name each time you receive a specific trap. You're going to produce an email containing a device name, IP and interface name as received in the trap.

This is a trap, received from a switch:

Another trap alert sample.png

This is the received mail.

Trap email.png


This is the trap definition.

Another snmp trap sample.png

1.8 TRAP Association to the Rest of Pandora FMS Alerts and SNMP Agent Trap Forwarding

The alerts defined on traps are completely independent from Pandora's Alert Engine, so correlations of them trigger an alert if the temperature reaches 29 degrees and the trap for secondary power supply is on. This type of alerts cannot be displayed since they're -possibly- not associated to any modules of Pandora FMS. Therefore, the trap console monitoring cannot be related to elements such as reports or maps.



Special ModuleModule SNMPTrap,with the trap sent from the SNMP console: Snmptrap agent.png

In order to achieve this, we've created a method called 'Agent SNMP Trap Forwarding'. This server-wide option forwards the trap to a special agent's module named 'SNMPTrap' as a text string, but only if the trap's originating IP address is defined as the agent's IP. Whenever this occurs, the trap arrives as a text line to the agent within that module, which is one that's only defined on arrival of the first trap.

Text alerts can be specified within that module, these being completely standard, just as any other module. This enables a customization of SNMP Monitoring in order for certain traps from certain origins to be treated as yet another type of module, which are thereby integrated into the rest of the monitoring, including Alert Correlation.



SNMPTrap data sample': Snmptrapforward2.png

This is an Enterprise feature which could be activated on the main setup screen as shown below.



Configuration option to activate the trap forward to the agents:
Snmptrap agent forwardsetup.png

If this setting is changed, the Pandora FMS Server must be restarted to enable it.

Another solution is to mount an alert on the trap to activate an agent's module. If the trap has a '1' in a certain log file, there is an agent reading that file, ready to run it when it finds this '1'. In this way, the module is going to be triggered if the desired trap arrives and the correlation can be established, based on the arrived trap.

1.9 Trap Filtering on the Server

Some systems receive a high number of traps. We're only interested in monitoring a tiny percentage of them. From Pandora FMS versions 3.2 and above, it's possible to filter the traps that the server obtains in order to avoid straining the application unnecessarily.

In order to define different filters, please go to 'Administration' -> 'Manage SNMP Console' and 'SNMP Filters'. One trap which is going to run in conjunction with any of them - just the ones for the server are going to get ruled out automatically.

Snmp filter editor.png

The filter is applied as a regular expression over the trap's corresponding entry within the SNMP log (it's '/var/log/pandora/pandora_snmptrap.log' by default), which comes with the following format:

SNMPv1 Trap:

%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n

SNMPv2 Trap:

%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

The accepted values are the following:

  • %y: The current year.
  • %m: The current month (numerical).
  • %l: The current month's day.
  • %h: the current hour.
  • %j: The current minute.
  • %k: The current second.
  • %a: The originating address (v1 traps only).
  • %b: The originating address (source address of the packet).
  • %N: The OID.
  • %w: The trap type (numerical).
  • %W: The trap description.
  • %q: The trap sub-type (numerical).
  • %v: T variable list in tab separated format (custom OID).

If the trap is of the SNMPv2 type, the trap OID is contained under the '%v' value as a second parameter. The third and the following parameters are an actual part of the variable's list (custom OID).

To e.g. filter all traps sent by the host, bearing the IP of '192.168.1.1', we could define the following filter:

Snmp filter example.png

1.10 External SNMP Trap Handler

The SNMP console is made for the sole purpose of obtaining traps. It only processes TRAPs as individual items. One trap is able to contain a lot of information. Sometimes, you have a situation when the only monitoring we're able to conduct is based on traps. For doing so, we might choose to post-process the obtained information of one trap by an external script working as a plug in.

To process the data of one trap in detail, you may send all contained information to a script as an alert result. I have used the trap shown below for the example. It's the trap view as it would be in Pandora's SNMP console log:

2010-08-26 12:01:46 pandora 10.201.246.2 .1.3.6.1.4.1.1722 .1.3.6.1.4.1.1722.2.10.1.1.1 233 .1.3.6.1.4.1.1722.2.10.1.1.3 = STRING: AIX_Software_Failure .1.3.6.1.4.1.1722.2.10.1.1.2 = STRING: 08 25 2010 08:23:43:697685 .1.3.6.1.4.1.1722.2.10.1.1.8 = STRING: 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED. .1.3.6.1.4.1.1722.2.10.1.1.6 = STRING: 8 .1.3.6.1.4.1.1722.2.10.1.1.11 = STRING: An application may not work properly .1.3.6.1.4.1.1722.2.10.1.1.10 = STRING: An application may not work properly .1.3.6.1.4.1.1722.2.10.1.1.12 = INTEGER: 4 .1.3.6.1.6.3.1.1.4.3.0 = OID: .1.3.6.1.4.1.1722

Snmp plugin1.png Snmp plugin2.png

On the screen shots below, you can see how a special alert is created. It executes a script which contains the complete content of the trap (_data_) and shows how an SNMP type of alert is created. In such a case, it has been mapped for the specific OID (.1.3.6.1.4.1.1722.2.10.1.1.1), although it could have been more general, e.g. (.1.3.6.1.4.1.1722) to call the script when there would be any type of traps like these (.1.3.6.1.4.1.1722, I suppose it would be part of the AIX specific MIB).

A script which processes this data is executed. It also 'analyzes' the trap to write data directly to Pandora FMS by creating an XML file, moving it to '/var/spool/pandora/data_in' as data, as if it had come from an agent. A basic script for this case would be one to generate complex information. We already possess enough information regarding this trap, which constitutes the following:

  • The originating IP
  • The main event (cold start)
  • The secondary events (descriptives): AIX_Software_Failure, 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly

When designing a script which 'parses' each of these data, e.g. 'miscript.pl' and storing it under '/var/spool/pandora/data_in' in the XML file along with a generic name and one random number, e.g. 'snmp_gateway.31415.data'.

The generated XML is supposed to look like the one on the picture below.


<?xml version='1.0' encoding='ISO-8859-1'?>
<agent_data description='' group='' os_name='aix' os_version='' interval='300' version='3.1(Build 100608)' timestamp='2010/08/26 12:20:26' agent_name='10.201.246.2'>
  <module>
    <name><![CDATA[Critical_Event]]></name>
    <description><![CDATA[]]></description>
    <type>async_proc</type>
    <data><![CDATA[1]]></data>
  </module>
<module>
    <name><![CDATA[events]]></name>
    <description><![CDATA[]]></description>
    <type>generic_string</type>
    <datalist>
      <data><value><![CDATA[AIX_Software_Failure]]></value></data>
      <data><value><![CDATA[A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC.]]></value></data>
      <data><value><![CDATA[Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.]]></value></data>
    </datalist>
  </module>
</agent_data>

The application is endless, but any script should be customized and it would have a very dynamic structure. Under a lot of systems, the information which gets received doesn't solely consist of text. It's also numerical and therefore able to feed numerical information to the modules in order to represent e.g. graphics, etc. Please keep in mind that all data is always of the asynchronous type.

1.10.1 Practical Example: ESX Monitoring using Traps

One of the most problematic things to monitor is the monitoring of distributed infrastructure. It gets even harder if each version changes its implementation to gather information (like VMware or ESX). In this chapter, we're going to explain how to monitor ESX systems by using an external SNMP Trap Handler.

ESX traps consist of the following data:


.1.3.6.1.4.1.6876.4.3.301 = STRING: "host" .1.3.6.1.4.1.6876.4.3.302 = STRING: "c7000-06-01.tsm.inet" .1.3.6.1.4.1.6876.4.3.303 = "" .1.3.6.1.4.1.6876.4.3.304 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.305 = STRING: "Yellow" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host cpu usage - Metric Usage = 1%"

.1.3.6.1.4.1.6876.4.3.301 = STRING: "host" .1.3.6.1.4.1.6876.4.3.302 = STRING: "dl360-00.tsm.inet" .1.3.6.1.4.1.6876.4.3.303 = "" .1.3.6.1.4.1.6876.4.3.304 = STRING: "Yellow".1.3.6.1.4.1.6876.4.3.305 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host memory usage - Metric Usage = 84%"

.1.3.6.1.4.1.6876.4.3.301 = STRING: "host" .1.3.6.1.4.1.6876.4.3.302 = "" .1.3.6.1.4.1.6876.4.3.303 = "" .1.3.6.1.4.1.6876.4.3.304 = STRING: "Red" .1.3.6.1.4.1.6876.4.3.305 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Datastore usage on disk - Metric Storage space actually used = 55%"

As you can see, traps could be used to gather information from the CPU, the hard drive or the memory. The general idea behind the trap handler is to write a small script which is able to 'understand' the trap and to create an XML file which emulates a software agent. It's recommended to write a trap handler for each technology. The process of doing so is pretty common and explained in four steps:

  1. Please create the handler script. You may base your work on the script which is provided below.
  2. Please create an alert command.
  3. Please create an alert action using the previous command. You can add some custom options for each 'destination' agent you want (if you possess several farms of ESX, you're probably going to like to host the data on different agents).
  4. Please create an SNMP trap-alert which maps the enterprise OID (information trap contains for all kind of this specific technology) and / or the source trap IP address.

Now we're going to show how to create the trap-handler script.

1.10.1.1 Step 1 - The Trap Handler Script (esx_trap_manager.pl)

#!/usr/bin/perl
# (c) Sancho Lerena 2010 <slerena@artica.es>
# Specific Pandora FMS trap collector for ESX 

use POSIX qw(setsid strftime);

sub show_help {
	print "\nSpecific Pandora FMS trap collector for ESX\n";
	print "(c) Sancho Lerena 2010 <slerena@artica.es>\n";
	print "Usage:\n\n";
	print "   esx_trap_manager.pl <destination_agent_name> <TRAP DATA>\n\n";
	exit;
}

sub writexml {
	my ($hostname, $xmlmessage ) = @_;
	my $file = "/var/spool/pandora/data_in/$hostname.".rand(1000).".data";

	open (FILE, ">> $file") or die "[FATAL] Cannot write to XML '$file'";
	print FILE $xmlmessage;
	close (FILE);
}

if ($#ARGV == -1){
	show_help();
}

$chunk = "";

# The first parameter is always the destination host for the virtual server.
$target_host = $ARGV[0];

foreach $argnum (1 .. $#ARGV) {
	if ($chunk ne ""){
		$chunk .= " ";
	}
	$chunk .= $ARGV[$argnum];
}

my $hostname = "";
my $now = strftime ("%Y-%m-%d %H:%M:%S", localtime());
my $xmldata = "<agent_data agent_name='$target_host' timestamp='$now' version='1.0' os='Other' os_version='ESX_Collectordime ' interval='9999999999'>";

if ($chunk =~ m/.1.3.6.1.4.1.6876.4.3.302 \= STRING\: ([A-Za-z0-9\-\.]*)\s\.1/){
	$hostname = "_".$1;
}

if ($chunk =~ m/Host cpu usage \- Metric Usage \= ([0-9]*)\z/){
	$value = $1;
	$module_name = "CPU_OCUPADA$hostname";
}

if ($chunk =~ m/Host memory usage \- Metric Usage = ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "MEMORIA_OCUPADA$hostname";
}

if ($chunk =~ m/Datastore usage on disk \- Metric Storage space actually used \= ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "DISCO_OCUPADO$hostname";
}

$xmldata .= "<module><name>$module_name</name><type>async_data</type><data>$value</data></module>\n";

$xmldata .= "</agent_data>\n";
writexml ($target_host, $xmldata);


1.10.1.2 Step 2 - Creating the Alert Command

In this example, I've put the command script in '/tmp' to put in a safer place, and to be sure it's executable (chmod 755):

Trap handler step2.png

1.10.1.3 Step 3 - Creating the Alert Action

We're going to create a specific action to send all information to specific trap agents. In this case, all information is going to be sent to an agent named 'WINN1247VSR'. The above mentioned command accepts the name of the agent as a parameter which is going to receive all the information (ESX Virtual Center), and 'chunks off' " the data from the trap, which can be unlimited and also includes all the information you're sending to the trap.

Trap handler step3.png

1.10.1.4 Step 4 - Creating the SNMP Alert

You may set alert traps by using the action you've just created.

Trap handler step4.png

In order to process all the traps the of ESX type by using the specific OID '.1.3.6.1.4.1.6876.4.3.301' to map the ESX type traps. You also have the option of filtering by source IP for each virtual center by filtering for the originating IP address (which is contained in the trap).

1.10.1.5 Data Visualization

This is an example of how the information is going to look. You may manage them as standard modules by this data.

Trap handler step5.png

1.11 SNMP Trap Forwarding (Pandora FMS versions 5 and above)

With Pandora FMS, it's also possible to forward SNMP traps to an external host by enabling the token named snmp_forward_trap within the server configuration file.

1.11.1 Configuration Example to forward Traps by SNMP v1

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 1
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.11.2 Configuration Example to forward Traps by SNMP v2c

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 2c
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.11.3 Configuration example to forward Traps by SNMP v3

This example is particularly challenging because of the implied knowledge regarding SNMP v3 traps. We are considering the remote SNMP agent to be defined under snmp_forward_ip. It contains the following entry in its file under '/etc/snmp/snmptrapd.conf':

createUser -e 0x0102030405 myuser MD5 mypassword DES myotherpassword

The Pandora server configuration file should look like this:

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 3
snmp_forward_secName myuser
snmp_forward_engineid 0x0102030405
snmp_forward_authProtocol MD5
snmp_forward_authPassword mypassword
snmp_forward_privProtocol DES
snmp_forward_privPassword myotherpassword
snmp_forward_secLevel authNoPriv

If you'd like to learn more about it, you can find more info here.


1.12 Independent Management of the snmptrapd daemon ( > v5.1 )

If, for some reason, you would like to manage the snmptrapd daemon independently from Pandora FMS (stop or start it independently from the main Pandora FMS daemon) then you need to take into account several things:

1. The snmpconsole parameter must be active for the pandora fms server.

2. The logs configured in the pandora fms server must be the same as the ones in the independent call to snmptrapd.

3. The call to snmptrap must be in a specific format the standard system call can't be used. That call must be like the following (the parameter: -A is very important!):

/usr/sbin/snmptrapd -A -t -On -n -a -Lf /var/log/pandora/pandora_snmptrap.log -p /var/run/pandora_snmptrapd.pid --format1=SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n --format2=SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

4. The snmptrapd token must be configured into the pandora fms configuration file:

snmp_trapd manual

5. To enable this functionality you need to complete the following procedure::

  • Change the configuration in /etc/pandora/pandora_server.conf
  • Stop the Pandora FMS server.
  • Check that the snmptrapd process isn't running (if it is running you can wait till the process stops or kill it)
  • Start snmptrapd manually (with the format indicated above).
  • Start the Pandora FMS Server.

1.12.1 Management of the trap log file

The snmptrapd process can be stopped and started without having to stop or start the pandora server process if the pandora_snmptrap.log.index and pandora_snmptrap.log haven't been modified. If those files are modified it is necessary to restart the pandora server. If you need to rotate externally the traps log files then you should restart the pandora server, after deleting the two files previously mentioned.

|Go back to Pandora FMS documentation index]]