Pandora:Documentation en:Alerts

From Pandora FMS Wiki

Jump to: navigation, search

Go back to Pandora FMS documentation index

Contents

Alerts

Introduction

An alert is Pandora FMS's reaction to a module's value being 'out of range'. Such a reaction is configurable and results in sending an e-mail or an SMS to the administrator, sends an SNMP trap, records the incident within the system's log, etc. An alert is basically any script-triggered action, configured in the operating system, where the Pandora FMS Server, which processes the module, is executed. There are three alert types: Individual Alerts, Event Alerts and SNMP Trap Alerts. In this chapter, we're going to talk about the Alert System in general and its individual alerts.

Introduction to the current Alert System

People usually complain about the complexity of defining alerts within Pandora FMS. Then (until version 2.0), alerts were much simpler to configure. For each alert, the condition and the behavior if the action wasn't executed, was defined for each case. It was a much more 'intuitive' thing (it also held fields like the 'threshold alert' which caused lot of headaches for more than one customer). It was very simple - but ... was it worth it?

One of our most helpful users in this case (he had lots of agents installed and was also able to manage Pandora FMS very well), told us that creating an alert within 2000 modules would be very time-consuming, especially if you have to modify the very same things within all of them. Due to these and quite a few other problems, we had to modify the alert system to become modular and for being able to separate the definition of the alert's firing condition (e.g. to alter a template) from the action to execute if it's fired (alert action) and from the command which is executed in the action (alert command). Now, the combination of an alert template and a module triggers the alert.

If I have e.g. a 1000 systems with a module called 'Host alive' and all of them have an associated alert template called 'Host down', an alert called 'Call the Operator' is going to be executed by default. If I want to e.g. change the minimum number of fired alerts before notifying the operator, I'm only required to make a change in the template's definition - I'm no longer forced to do it separately in all of the 1000 alerts to modify their conditions.

Several of our customers only manage a few dozen machines, but there are a lot of users with hundreds, even thousands of systems monitored by Pandora FMS, and we're making it possible to manage any and all types of environments by it.

The Alert Structure


Esquema-alert-structure.png


An alert consists of the following:

  • Commands
  • Actions
  • Templates

A command defines the operation to perform if an alert is fired. Some examples of a command could be e.g. to write into a log, send an email or SMS, execute a script or a program, etc.

An action links a command to a template and allows you to customize the command execution by using three generic parameters: 'Field 1', 'Field 2' and 'Field 3'. These parameters allow you to customize the command execution because they are passed as input parameters to the executed command.

Within the template, you're defining the alert's generic parameters like firing conditions, firing actions and the alert recovery.

  • Firing Conditions: The conditions for when the alert is going to be fired, e.g. if the data is above a threshold or if the status is in 'critical' range, etc.
  • Firing Actions: The configuration for the action which is going to be performed if the alert is fired.
  • Alert Recovery: The configuration for actions performed if the system is recovered after the alert was fired.

The Alert System's Information Flow

When you're defining actions and templates, you have some generic fields called 'Field1', 'Field2' and 'Field3' at your disposal. These are the values passed as input parameters to the executed command. The values of these parameters are subsequently propagated from the template to the action and to the command from there. The value propagation from the template to the action is only going to be performed if the field defined in the action hasn't got any value - otherwise, any resident value is going to be used.


Esquema-parameters-carrying.png


This is an example for how template values are overwritten by the action's values:


Alertas esquema6.png


We can e.g. create a template that fires an alert and sends an email, containing the following fields:

  • Template:
    • Field 1: myemail@domain.com
    • Field 2: [Alert] The alert was fired.
    • Field 3: The alert was fired!!! SOS!!!
  • Action:
    • Field 1: myboss@domain.com
    • Field 2: <left blank>
    • Field 3: <left blank>

The values which are going to be passed to the command are:

  • Command:
    • Field 1: myboss@domain.com
    • Field 2: [Alert] The alert was fired
    • Field 3: The alert was fired!!! SOS!!!

The Alert Command (Pandora Versions 5 and above only)

Pandora FMS's reaction to a value like 'out of range' can consist of the following types: A record in a system log, the sending of an e-mail or SMS or the execution of any processable script which is hosted on it.

The different reactions Pandora FMS can be adopted to are configured at the option Command in the Alerts menu.



Susi2.png



In this section, you're able to modify or add your own alert commands.

Command Creation for an Alert

New alert commands are created by clicking on the menu Alerts -> Commands and Create button.

Once you've clicked on 'Create', a window as the one shown below is going to appear.



Susi3 5.png



Next, the following fields are introduced:

Name: The command's name. It's important to be descriptive but short, e.g. 'Log' or 'Communications'.

Command: The Command to be executed as reaction to a module being 'out of range'. You may use macros to replace the preconfigured parameters within the alert definition. The available macros are the following:

  • _field1_ - _field10_: Ten fields to customize the macro.
  • _agent_: The agent's complete name.
  • _agentdescription_: The description of the agent who fired the alert.
  • _agentcustomfield_n_: The agent's custom field number n (eg. _agentcustomfield_9_).
  • _agentcustomid_: The agent's custom ID.
  • _agentgroup_: The agent's group name.
  • _agentstatus_: The current status of the agent.
  • _address_: The address of the agent which fired the alert.
  • _timestamp_: A standard representation of date and time. It gets replaced automatically in the moment of alert execution.
  • _timezone_: The timezone's name that _timestamp_ represents (in Pandora FMS versions >=5.0 SP1).
  • _data_: The values of the data which have triggered the alert.
  • _alert_description_: The alert's description.
  • _alert_threshold_: The alert's threshold.
  • _alert_times_fired_: The number of times the alert has been fired.
  • _module_: The module's name.
  • _modulecustomid_: The module's custom ID.
  • _modulegroup_: The module's group name.
  • _moduledescription_: The description of the module which fired the alert.
  • _modulestatus_: The module's status.
  • _moduletags_: The tags associated to the module.
  • _alert_name_: The alert's name.
  • _alert_priority_: The numerical value for alert priority.
  • _event_text_severity_: (Event alerts only) The severity of the text event which fires the alert, e.g. 'maintenance', 'informational', 'normal', 'minor', 'warning', 'major' and 'critical'.
  • _event_id_: (event alerts only) the numerical ID for the Pandora FMS event, useful to correlate data with external sources and/or pass ID for other processes, later you can validate this event with the API/CLI.
  • _event_description_: (event alerts only) the textual description of the Pandora FMS event.
  • _id_agent_: The ID of the agent. It's useful to e.g. build a direct URL to redirect to a Pandora FMS console's web page.
  • _id_alert_: The unique numerical ID of the alert which is used to correlate to a third party software.
  • _policy_: The name of the policy the module belongs to (if it applies).
  • _interval_:: The execution interval of the module.
  • _plugin_param1_ - _plugin_param10_ (could be more): The value of the corresponding plug-in parameters' field.
  • _plugin_param1_desc_ - _plugin_param10_desc_ (could be more): The description of the corresponding plug-in parameters' field.
  • _groupcontact_: The group's contact information. It gets configured when the group is created.
  • _groupcustomid_: The group's custom ID.
  • _groupother_: All other information about the group. It gets configured when the group is created.
  • _email_tag_: The emails associated to module tags.
  • _phone_tag_: The phone associated to the to module tags.
  • _alert_critical_instructions_: Instruction contained in the module for critical status.
  • _alert_warning_instructions_: Instruction contained in the module for warning status.


When it comes to creating the commands for the alerts, please keep in mind that such commands are executed by the Pandora FMS Server which processes the module of the processed agent, e.g. a Data or Network Server. Alerts will also be executed by the privileges of the user which is executing the Pandora FMS Server. It's recommended to test whether the command's execution is successful or not and if it produces the desired result (e.g. sending an e-mail, generating an entry within the log file, etc.) on the command-line interface in the moment of the command's definition.

Description

A thorough description of the alert command for information purposes.

Description of the fields and possible values

For each field:

  • Description: It would be the tab near the text box in the configuration form of the command action.
  • Possible values: A collection of the possible values for that field.

If the field is configured, will be a select instead a text box. The select needs a tag (the visible value) for each value (the sent value).

This is the supported syntax:

value1,tag1;value2,tag2;value3,tag3

For example:

1,Number one;2,Number two;3,Number three;4,Number four



Possible values 1.png





Possible values 2.png



Info.png

Since the 6.0 version, it will be possible to show a HTML editor in a command field in the creation or edition of an alert action if that command field has as value the special token _html_editor_

 


Once it's created, please click on the 'Create' button.



Susi4 5.png



Editing an Alert Command

You may edit the newly created alert commands by clicking on Alerts -> Commands.



Susi5.jpg



To edit an alert command, please click on the command's name.



Susi6 5.png



Once the chosen alert has been modified, please click on the 'Update' button.


Template warning.png

The alerts named eMail, Internal Audit and Pandora FMS Event cannot be modified.

 


How to Delete an Alert Command

In order to delete an alert, please click on the gray trash icon (which is located on the right hand side of the alert) as shown below.



Susi7.png




Predefined Commands

There are some predefined commands which could be adjusted if the system doesn't have the internal commands for executing such alerts. The development team has tested these alerts by the distributions named 'Red Hat', 'CentOs', 'Debian' and 'Ubuntu Server'.


eMail: Sends an email from the Pandora FMS Server and uses the Perl 'sendmail' command to do so. Pandora FMS utilizes the system-specific tools to conduct almost all alerts. In this case, it's going to be necessary to check whether or not the 'libmail-sendmail-perl' (an 'xprobe2' package) is already installed on your system.

Since the 6.0 version, this action sends the emails in HTML, which allow the creation of more visually attractive templates. It should be taken into consideration that the receiver of the email should have access to the resources used in the template (images, fonts, etc.).

Internal Audit:
This is just an 'internal' alert which generates a small entry within the internal Auditing System of Pandora FMS. It's kept in the Pandora FMS Database and could be reviewed by the console's event viewer.

Pandora FMS Event:
This alert creates a special event within the Pandora FMS Event Manager.

Pandora FMS Alertlog:
This is a default alert to write alerts in a standard ASCII plain-text log file located under '/var/log/pandora/pandora_alert.log'.

SNMP Trap:
It sends an SNMP trap if an alert occurs.

Syslog:
It sends an alert to the system registry and uses the system command named 'logger' to do so.

Sound Alert:
It plays back a sound if an alert is received.

Jabber Alert:
Sends a jabber alert to a chat room on a predefined server (please configure the file named '.sendxmpprc' first). It uses 'field3' for the text message, 'field1' for the user's alias-for-source message, and 'field2' for the chat-room's name.

SMS Text:
Sends an SMS to a specific cellphone. You're required to define an alert and a gateway for sending configured and accessible SMS from Pandora FMS before being able to do so. It's also possible to install one using 'Gnokii' to send SMS directly by using a Nokia telephone with an USB wire. Further information on the detailed procedure is going to be described below.

Validate Event:
It validates all events in relation to a module. The agent's and module's name will be given.

Examples of Commands

Integrating Alerts by the Jabber Instant Messenger

It's very easy to set up Pandora FMS to send alerts by using a Jabber Server. Jabber can be utilized as a system to get real time alerts as well as a history log, allowing a group of people to receive those alerts simultaneously.

Installing Jabber Services

Procedure for the Client:

  1. Please install a Jabber client like Pidgin.
  2. Register an account under 'Pidgin' by clicking on the 'Accounts' tab to configure the account.
  3. Login to that account.

Procedure for the Pandora FMS Server:

  1. Please install the package named 'sendxmpp'. It's a dependency for the Pandora FMS Server in order to send messages to Jabber services.
  2. Create a file named '.sendxmpprc' within your '/home' folder.
  3. Edit that file and insert the following text:
  useraccount@jabber.org password
  1. Please change the file permissions for '.sendxmpprc':
  chmod 0600 .sendxmpprc

By the example below, you're now able to send private messages using the command line.

  $ echo "Hello" | sendxmpp -s pandora useracount@jabber.org 

To register the alert within the Pandora FMS Web Console and to add a new command and configure its variables, you're required to do the following:

  • Field_1: The Jabber address.
  • Field_2: The text you intend to send.

The alert is going to be defined as follows:

  echo _field2_ | sendxmpp -s pandora _field1_
Additional Examples of Jabber Usage

To send a message to a chat room, please enter the following command:

  $ echo "Dinner Time" | sendxmpp -r TheCook --chatroom test2@conference.jabber.org

To send the log entries to a Jabber destination in real-time, please enter the following command:

  $ tail -f /var/log/syslog | sendxmpp -i sysadmin@myjabberserver.com


Template warning.png

Be careful not to flood public Jabber Servers by your messages or you're very likely to get banned by them.

 


Sending Emails by the 'Expect' Script

Sometimes it's necessary to use an authenticated SMTP to send emails. It's a probably easier and more versatile method to use a simple 'expect' script instead of configuring 'sendmail' to use an authenticated SMTP. This is an example using 'expect' to send emails by using an Exchange Server:

First, you're required to create a file called '/etc/snmp' containing the following script:

#!/usr/bin/expect -f
set arg1 [lindex $argv 0] 
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1 
spawn telnet myserver.com 25 
expect "220"
send "ehlo mymachine.mydomain.com\r"
expect "250"
send "AUTH login\r"
expect "334"
send "2342348werhkwjernsdf78sdf3w4rwe32wer=\r"
expect "334"
send "YRejewrhneruT==\r"
expect "235"
send "MAIL FROM: myuser@domain.com\r"
expect "Sender OK"
send "RCPT TO: $arg1\r"
expect "250"
send "data\r"
expect "354"
send "Subject: $arg2\r"
send "$arg3 \r\r"
send ".\r"
expect "delivery"
send "quit"
quit

To edit the file permissions to allow the execution, please enter the following command:

chmod 700 /root/smtp 

Before trying to use it, please make sure that '/usr/bin/expect' is working appropriately.

Before being able to utilize this in conjunction with Pandora FMS, you're also required to create a new command (or to modify an already existing email alert-sending command) and to specify the following fields within the Pandora FMS Alert Command definition in the field named 'Command'. It's going to write the following:

/root/smtp _field1_ _field2_ _field3_

The script is allowed to be located in any place on the system.


Info.png

Please keep in mind that the alert script is launched by the server which is going to process the data. If the payload is consisted of network data, the Network Server is going to process it. If it's an XML data file sent by an agent, it's the Data Server which is going to launch it.

 


If you have several physical servers, it's possible that you're required to copy the same script to the same location, along with the same permissions and the same owner on all the systems you have a Pandora FMS Server running and want to execute this alert on. Please keep in mind that the Pandora FMS Network Servers are required to be executed as 'root' (e.g. for being able to execute ICMP latency tests). However, the Data Server isn't required to be executed as 'root' - it may be started by any user without special privileges.

The alert is going to be executed by the user who's executing the Pandora FMS Server process.

Sending SMS by 'Gnokii'

There's also the option of using 'Gnokii'. Do do so, it's required to use a Nokia cellphone or one compatible with Gnokii (please feel free to check the compatible hardware list on the Gnokii Project Page. You're also required to have a USB data cable connected the cellphone and a connection to the Pandora FMS Server you intend to send the SMS Alerts from.

Info.png

Gnokii supports a large variety of Nokia cellphones and some models by other manufacturers.

 


By using Gnokii, you may also send SMS directly from the command line. This is a very easy and quick way to send any SMS directly from a Pandora FMS Server, thereby avoiding the use of gateways sending SMS by using the internet (which is not very useful if the network is down) or GSM hardware solutions for sending messages which are very expensive in some countries.


Info.png

An alternative to the use of Gnokii is the Gammu Project.

 



This is an example of sending an SMS from the command line by using Gnokii:

echo "PANDORA: Server XXXX is down at XXXXX" | gnokii --sendsms 555123123

Gnokii is unable to send an SMS which bears attached images, but it's able to send a URL via HTTP or WAP (Wireless Application Protocol). If a message is received, it could look like the one you're going to see if you enter the command shown below:

echo "Image capture sample" | gnokii --sendsms 555123123 -w http://artica.homelinux.com/capture.jpg

It's also able to send one image's URL or one that leads to a 'light version' of the console in order to provide console access for the cellphone, facilitating the reception and analysis of emergency data for the user.

The Artica Development Team has tested it. They've sent SMS alerts from a Nokia 6030 cellphone in a moment an internet connection wasn't available. The Nokia 6030 cellphone uses the module's 6510 definition within the 'gnokiirc' file. It takes about four seconds to send an SMS.

It's also possible to install a much more versatile sending gateway by using GAMMU instead of Gnokii.

Executing a Remote Command on another System (UNIX)

Sometimes, it's pretty interesting to execute the command on another system. Please use the SSH command to do so. The system in which the command is going to be executed should be a UNIX system. It's also required to have the SSH daemon installed, started and accessible.

To avoid storing the access password on the machine which executes the command within the Pandora Console, it's recommended to copy the server's public key to where you intend to execute the remote command on the Pandora FMS Server.

Once you have done this, please execute the following command:

ssh user@hostname [_field1_]

By using '_Field1_' as a variable, you may use any command you want.

Alert Actions (all Pandora FMS versions including 5.0)

Actions are alert components in which a command is linked to generic variables like 'Field 1', 'Field 2', ... 'Field 10', etc. (which are described in the previous section in detail). These actions are going to be used in the alert templates which are the ones that associate a data condition to a specific action later.

Creating an Action

New actions are created by clicking on Alerts -> Action and Create.



Accion1.jpg



Once you have clicked on 'Create', you're going to see the following window:



Accion2.jpg



An explanation of the fields you're going to see is shown below:

  • Name: The name of the action.
  • Group: The group of the action.
  • Command: The command which is going to be used in case of a fired alert. You may choose between numerous predefined commands under Pandora FMS.
  • Threshold: The action's execution threshold.
  • Command Preview: The command which is going to be executed on the system is going to appear here automatically. This field is not editable.
  • Field 1-10: The values of the macros from '_field1_ through '_field10_' are defined here. They are intended to be used in conjunction with the command if necessary.

Once you have filled out the fields, please click on the 'Create' button.



Boton1.jpg



To edit the newly created actions, please click on Alerts and Actions.

Editing an Action



Caca1.png



To edit the action, please click on the action's name.



Sugus.png



Once you've completed the changes, please update them by clicking on the 'Update' button.

Deleting an Action

To delete an action, please click on the gray trash icon which is located on the action's right side.



Sipo.jpg



Alert Templates

Alert templates are alerts in which all parameters are already predefined. They only require their assigned agent and the module that is used to activate the command or the response if a value is 'out of range'. The templates were created to render the administrator's management job a little easier, so they could be assigned to the required agents more quickly if they're already predefined.


Creating an Alert Template

You may create new templates by clicking on 'Administration' -> 'Manage Alerts' -> 'Templates' and the 'Create' buttons.
Pandora 6.0: You may create new templates by clicking on Alerts -> Templates and the 'Create' buttons.


Planti.jpg


Once you've clicked on the 'Create' button, a window like the one shown below is going to appear:


Sabo.jpg


This is a description for the fields you're going to see there:

  • Name: The name of the template.
  • Description: It describes the template function. It's useful to distinguish the template from others within the alert's general view.
  • Priority: The field which provides information about the alert. It's useful when searching for alerts.

You may choose between the following priorities:

  • 'Maintenance'
  • 'Informational'
  • 'Normal'
  • 'Warning'
  • 'Critical'
  • 'Condition Type:' The field where the type of condition which is going to be applied on the alert is defined. The required combos will be added according to the defined type.

Explanation for the fields:

  • Regular Expression: The used regular expression. The alert is going to be fired if the module's value performs a defined condition, expressed by using a regular expression. This is the used firing condition for string and text data. All other conditions are intended for status and any other types of numerical data.


Regular.jpg


By choosing the 'regular expression' condition, the possibility to select the trigger box appears if the value is matched. If you select it, the alert is going to be fired if the value matches. If not, the alert is going to be fired if the value doesn't match.

  • Max and Min: The used maximum and a minimum values.


Maximo.jpg


By choosing the 'Max and Min' condition, the possibility to select the trigger box appears if the value is matched. If you select it, the alert is going to be fired if the value is out of range of the predefined minimum and maximum values. If not, the alert is going to be fired if the value resides within the predefined minimum and maximum range.

  • Max: The used 'maximum' value. The alert is going to be fired if the module's value is higher than the defined 'maximum' value.


Minimo.jpg


  • Min: The used 'minimum' value. The alert is going to be fired if the module's value is lower than the defined 'minimum' value.


Minimo1.jpg


  • Equal to: The used 'equal' value. The alert is going to be fired if the module's value is equal to the one defined. It's intended to be used only for numerical values, e.g. '0' or '0.124'.


Equal.jpg


  • Not equal to: The used 'not equal to' value. The alert is going to be fired if the module's value is unequal to the one defined. It's intended to be used only for numerical values, e.g. '0' or '0.124'.


Notequal.jpg


  • Warning Status: The module's used 'warning' status. The alert is going to be fired if it's in 'warning' status.


Estupido.jpg


  • Critical Status: The module's used 'critical' status. The alert is going to be fired if it's in 'critical' status.


Critical.jpg


Once the appropriate fields have been filled out, please click on the 'Next' button, where you're going to see the following window:


Pincha.png


These are the explanations for the fields you're going to see there:

Days of Week: The days on which the alert could be fired at all.

Use special days list: It's used to enable or disable the use of the special days list, e.g. holidays and special working days.

Time From: The time from which the alert action is going to be executed.

Time To: The time until the alert action is going to be executed.

Time Threshold: It defines the time interval during which it's guaranteed that an alert is not going to be fired more times than the number of times defined under 'maximum number of alerts'. If the defined interval is exceeded, an alert is not going to recover if it reaches a specific value, except if the 'alert recover' value is activated. It's recovered immediately after receiving a specific value (regardless of the threshold) in this case.

Min. number of Alerts: The minimum number of times the data has to be 'out of range' to fire an alert. It's always counting from the number defined within the 'FlipFlop' parameter of the module. The default value is '0', which means the alert is going to be fired if the condition's first value is met. It's intended as a filter, which is necessary to eliminate any false positives.

Reset counter when alert is not continuously (>= 6.1)

Enable this option if you want to reset the counter for minimum number of alerts when the alert state is not continuously even if it's in the time threshold.

For example, when 'min. number of alerts' is set to 1 and this option is not checked, an alert is fired with the second time like below.

OK -> NG -> OK -> OK -> NG(an alert is fired at this time.) ->....

Contrary, when this option is checked, an alert is fired like below.

OK -> NG -> OK -> OK -> NG -> NG(an alert is fired at this time.) ->....

Max number of Alerts The maximum number of alerts which could be sent consecutively within the same time interval (time threshold).

Field 1: It defines the value for the '_field1_' variable. The list of macros (which are going to be described below) could be used here.

Field 2: It defines the value for the '_field2_' variable.

Field 3: It defines the value for the '_field3_' variable.

Default Action: The default action the template is going to have is defined in this combo. It's the action which is going to be automatically created if the template is assigned to the module. You may assign one or none to it, but you're unable to assign several default actions here.



Combo.jpg

Create template step3.jpg



This is a definition for the fields you're going to see there:

Alert Recovery: The Combo where you're able to define whether the alert recovery is enabled or not.

Field 2: Defines the value for the '_field2_' variable in the alert recovery.

Field 3: Defines the value for the '_field3_' variable in the alert recovery.

Once all appropriate fields have been filled out, please click on the 'Finish' button.

Replaceable Macros within Field 1 through Field 10

It's possible to use the following macros in all cases of the fields 'Field1', 'Field2' and 'Field3' (in the alert template, the command and the action). These are 'words' which are going to be replaced if executed by a value. That value is going to change by a value or agent which has fired the alert, etc. depending on the moment.

  • _field1_: The user-defined field 1.
  • _field2_: The user-defined field 2.
  • _field3_: The user-defined field 3.
  • _field4_: The user-defined field 4.
  • _field5_: The user-defined field 5.
  • _field6_: The user-defined field 6.
  • _field7_: The user-defined field 7.
  • _field8_: The user-defined field 8.
  • _field9_: The user-defined field 9.
  • _field10_: The user-defined field 10.
  • _agent_: The name of the agent which fired the alert.
  • _agentcustomfield_n_: Agent custom field number n (eg. _agentcustomfield_9_).
  • _agentcustomid_: Agent custom ID.
  • _agentdescription_: The description of the agent which fired the alert.
  • _agentgroup_: The agent's group name.
  • _agentstatus_: Current status of the agent.
  • _address_: The address of the agent which fired the alert.
  • _timestamp_: The time the alert was fired (yy-mm-dd hh:mm:ss).
  • _timezone_: The time-zone's name the monitoring server resides in (>=5.0SP1).
  • _data_: The module's data which caused the alert to fire.
  • _alert_description_: The alert's description.
  • _alert_threshold_: The alert's threshold.
  • _alert_times_fired_: The number of times the alert was fired.
  • _module_: The module's name.
  • _modulecustomid_: Module custom ID.
  • _modulegroup_: The module's group name.
  • _moduledescription_: The description of the module which fired the alert.
  • _modulestatus_: Status of the module.
  • _alert_name_: The alert's name.
  • _alert_priority_: The numerical alert priority.
  • _alert_text_severity_: Text alert severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
  • _event_text_severity_: (Only event alerts) Text event (who fire the alert) severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
  • _event_id_: (Only event alerts) Id of the event that fired the alert.
  • _event_description_: (Only event alerts) Textual description of the event that fired the alert.
  • _id_agent_: The agent's ID. It's useful to build direct URLs to redirect them to a console web page of Pandora FMS.
  • _id_group_: The agent group's ID. (>=5.1SP4)
  • _id_alert_: The alert's unique numerical ID. It's used to correlate on third party software.
  • _policy_: The policy's name the module belongs to (if any).
  • _interval_: The module's execution interval.
  • _target_ip_: The module's target IP address.
  • _target_port_: The module's target port number.
  • _plugin_parameters_: The module's plug-in parameters.
  • _groupcontact_: Group contact information. Configured when the group is created.
  • _groupcustomid_: Group custom ID.
  • _groupother_: Other information about the group. Configured when the group is created.
  • _name_tag_: Names of the tags associated to the module.
  • _email_tag_: Emails associated to the module tags.
  • _phone_tag_: Phone numbers associated to the module tags.
  • _moduletags_: URLs associated to the module tags.
  • _alert_critical_instructions_: Instructions for the CRITICAL status contained in the module.
  • _alert_warning_instructions_: Instructions for the WARNING status contained in the module.
  • _modulegraph_nh_: (>=6.0) (Only for alerts that use the command eMail) Returns an image codified in base64 of a module graph with a period of n hours (eg. _modulegraph_24h_). A correct setup of the connection between the server and the console's api is required. This setup is done into the server's configuration file.

Commands for the Replacement of Macros and '_field*_' Fields

After describing what commands, actions and templates are, you'd probably like to ask the question about the necessity of defining the fields 'Field1', 'Field2', 'Field3' etc. and how does it all make sense.

If an alert is fired, the 'field*' values are transferred from the action and from the template to the command. If the '_field1_' value differs from an empty string within the action, it's going to ignore the command which is transferred from the template, but it's not effecting anything. If the command's '_field1_' value differs from '_field1_', it's going to ignore any action's or template's 'field1'-parameter. Neither the action nor the template is going to be able to redefine it, because it has '_field1-' as a value. This means that it's ordering the command to insert whatever comes from the action or template into this field.

Within the action, the same thing happens but in a more subtle way. If this field is empty, anything that is transferred from the alert screen is going to be transferred to the command. If this field differs from an empty string, it's going to use the residing values in this field. The values which come from the template are going to be ignored.

They've been created this way to establish some 'fixed' parameters by a command or action without the possibility of losing its flexibility.

Complete Example of an Alert containing Replacement Macros

Let's suppose for a moment you intend to create a log entry in which every line appears in the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

To do so, you're required to change your configuration as shown below.

Command Configuration

echo _timestamp_ pandora _field2_ >> _field1_

Action Configuration

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template Configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovering section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

If an alert is fired, the following line is going to be added to the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status

In the moment of alert recovery, the following line is going to be added:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

Editing a Template

You may edit the newly created templates by clicking on the menus of 'Administration' -> 'Manage Alerts' and 'Templates'.
Pandora 6.0: You may edit the newly created templates by clicking on the menus of 'Alerts' and 'Templates'.



Plantilla.jpg



To edit a template, please click on the template's name.

Deleting a Template

To delete a template, please click on the gray trash icon which is located on the alert's right side.



Cruz.jpg



Assigning Alert Templates to Modules

Until now, we've defined the commands and actions as a given response from Pandora FMS for an 'out of range' value. By these templates, we're defining whether a value is 'out of range' or not and which circumstances / responses should be shown in order for Pandora FMS to work appropriately. In this section, we're going to describe the approach for relating the templates and actions to the Pandora FMS Agents and its modules. This operation is the one which triggers the responses of Pandora FMS if it encounters a value which is out of a predefined range.

You may assign the alerts in two ways: From 'Administration' -> 'Manage Agents' and 'Alerts' or under 'Administration' -> 'Policy' as we're going to explain within the chapter named Monitoring By Policies in detail.


Alert Management from an Alert's Sub Menu

Assigning Alerts from an Alert's Sub Menu

The alert assignment for modules is conducted by clicking on 'Administration' -> 'Manage Alerts' and 'Add Alert', subsequently filling out the appropriate fields.
Pandora 6.0: The alert assignment for modules is conducted by clicking on 'Alerts' -> 'List of Alerts' and 'Create', subsequently filling out the appropriate fields.



Pinar.jpg



This is a definition for the fields you're going to see there:

  • Group: You may choose the group the agent belongs to by a combo here.
  • Agent: The name of the agent to which the alert is going to be assigned to.
  • Module: The module which is used for firing the alert.
  • Template: You may choose the template to configure the alert by a combo here.
  • Actions:: It allows to choose between all preconfigured alerts. The selected action is added to the one defined within the template. You may choose more than one action.
  • Threshold: The alert action will not be executed more than once every 'action_threshold' seconds, regardless of the number of times the alert is fired.

Modifying Alerts from an Alert's Sub Menu

Once an alert has been created, it's only possible to modify the actions which have been added to the template's action.

It's also possible to delete the action that was selected in the moment you've created the alert by clicking the gray trash icon which is located on the right side of the action, or to add new actions by clicking on the 'Add' ('+') button.



Modifica.jpg



Deactivating Alerts from an Alert's Sub Menu

Once the alert has been created, it's possible to deactivate it by clicking on the light-bulb icon which is located on the right side of the alert's name.



Desha.jpg



The available alerts are rendered in blue while the ones which aren't are rendered in yellow.

Deleting Alerts from the Alert's Sub Menu

It's possible to delete any alert by clicking on the gray trash icon which is located on the right side of the alert.



Filter.jpg



Managing Alerts from within the Agent

Alert Assignment from within the Agent

Other option to add an alert is by doing so from within the agent. Please click on 'Administration' -> 'Manage Monitoring' and 'Manage Agents', where you're going to find all Pandora FMS Agents.



Tarantu.jpg



Please pick an agent and click on the 'Alerts' box.



Vicho.jpg



This is a definition for the fields you're going to find there:

  • Module: It's the module which is going to be used for firing the alert.
  • Template: You may select the template which is going to be used to configure the alert here.
  • Actions: It allows you to choose between all preconfigured actions. The chosen action is added to the one defined in the template. It's possible to select more than one action here.
  • Threshold: The alert action is not going to be executed more than once every 'action_threshold' seconds, regardless of the number of times the alert is fired.

Modifying Alerts from within the Agent

Once an alert has been created, it's only possible to modify the actions which have been added to the template's action.

It's also possible to delete the action which was selected in the moment you've created the alert by clicking on the gray trash icon which is located on the right side of the action, or to add new actions by clicking on the 'Add' button.



Gusano.jpg



Deactivating Alerts from within the Agent

Once an alert has been created, it's possible to deactivate it by clicking on the light bulb icon which is located on the right side of the alert's name.



Cuca.jpg



The available alerts are rendered in blue and the ones which aren't are rendered in yellow.

Deleting Alerts from within the Agent

It's possible to delete any alert by clicking on the gray trash icon which is located on the alert's right side.



Adorno.jpg



Defining an Alert

Let's refer to the previous case for a moment here. We're required to monitor a module which holds numerical values. In our case, a module which measures the system's CPU, in other cases it can be a temperature sensor which provides the value in degrees on the Celsius scale. Let's make sure that our module obtains its data appropriately:


Qgcpu1.png


On this picture, we see that we have a module called 'sys_cpu' holding a current value of '7'. In our case, we want the alert to be fired if it exceeds the value of '20'. To do so, we're going to configure the module to be in 'critical' state if it exceeds the value of '20'. To accomplish this, we're required to click on the wrench icon on the left to configure the monitor's performance as shown below.


Qgcpu2.png


To do so, we're going to change the 'critical' value as shown below.


Qgcpu3.png


In this moment, we're accepting and saving the changes. If the CPU module reaches the value of '20' or above, its state is going to change to 'critical' and rendered in red, as shown on the picture below.


Qgcpu4.png


We've already talked about the system 'knowing' how to distinguish between whether something is defined as 'OK' (green) or 'critical' (red). What we're required to do now is to get the system to send us an email if the module's status changes. To do so, we're going to use the Alert System of Pandora FMS.

The first thing we're required to do is to make sure there is a predefined command which exactly performs the desired task (which is to send emails). This example is an easy one, because there already is a predefined command to send us emails within Pandora FMS.

Configuring an Action

We have to create one action which could be described as 'Send an email to the operator'.
Please click on 'Administration' -> 'Alerts' and 'Actions' and click on the button to create a new action, as shown on the picture below.
Pandora 6.0: Please click on 'Alerts' -> 'Actions' and click on the button to create a new action, as shown on the picture below.


Qgcpu5.png


This action utilizes the command 'Send email' and it's really easy. You're only required to fill out one field (e.g. 'Field 1') and leave the other ones empty. This is one of the more confusing parts from the Pandora FMS Alert System:

What's the meaning of the fields named 'Field1', 'Field2' and 'Field3' ?
These fields are used to pass along the alert template's and the command's information to the command, so both the template and the command are able to provide different information to the executed command. In this case, the command only sets up 'Field 1'. We're going to leave 'Field 2' and 'Field 3' for the template, as we're going to see below.


'Field 1' is the one that we're going to use to define the operator's email address. In this case, it's 'sancho.lerena@notexist.com'.

Configuring an Alert Template

We're required to create an alert template which is recommended to be the most generic template possible (to be able to re-use it later). It's required to hold a message like "Alert! I have a module that's in 'critical' state!" and to send an email containing this message to the appropriate operator by default.
To do so, please click on 'Administration' -> 'Alerts' and 'Templates' and click on the button to create a new alert template as shown below.
Pandora 6.0: To do so, please click on 'Alerts' -> 'Templates' and click on the button to create a new alert template as shown below.


Qgcpu6.png

The field named 'Condition' is the one which defines the condition. In this case, it's the 'critical' status. If associated to a module, this template is going to be fired if the associated module is in 'critical' state. A short time ago, we've configured the module named 'cpu_user' in a such way that it changes to a 'critical' status if it exceeds the value of '20' or above.

The priority of 'critical' we've defined here is the alert priority. It has nothing to do with the module's 'critical' status. The alert criticalities are designed to visualize them later by different criticalities within other views such as the Event View.

Let's go to step 2 by clicking on the 'next' button.



Qgcpu7.png

Step 2 defines all configuration values which are considered 'refinements' from the firing condition's alert template. Some of them, like the first ones, are rather easy. They restrict the performance in the moment of alert firing to some specified days in a predefined hour range.

The most critical parameters are the following:

  • Time threshold: The default value is '1 Day'. If one module is down all the time, e.g. 'one day' and we've defined a value of '5 minutes' here, it means that it's going to send alerts every 5 minutes to us. If we've set it up to '24 hours', it's only going to send us the alert once in the moment it goes down. If the module is restored and falls down again, it's going to send us an alert again, but if it's still down from the second time, it's not going to send us more alerts until the 24 hours have passed.
  • Min. Number of Alerts: The minimum number of times the condition has to be met (in this case, the module is on 'critical' status) before Pandora FMS executes the actions associated to the alert template. It's a good way to avoid false positives, to get flooded by alerts or to prevent some element's erratic behavior from firing a lot of alerts. By entering a value of '1', the first alert is going to be ignored. If you enter a value of '0', it's going to fire the alert the first time the module shows any predefined erratic behavior.
  • Max. Number of Alerts: A value of '1' means the action will be executed once. If we e.g. enter a value of '10' here, it's going to execute the predefined action ten times. It's intended to be used as a way to limit the number of times an alert can be fired.

Now we're going to see the fields 'field1' - 'field10' again. We can see that 'field1' is empty, which is just the way we've defined it in the moment of configuring the action. The fields 2 and 3 are used for the action of sending an email and to define 'subject' and 'message text' where 'field 1' is used to define the receiver(s), separated by commas in case there are more than one. The other fields are going to be left blank. The template defines the subject and the alert message by using some macros. In our case, supposing that the agent where the module is located, would be named 'Farscape', we're going to receive a message like the one below:

To: sancho.lerena@notexist.com
Subject: [PANDORA] Farscape cpu_user is in CRITICAL status with value 20
email text:
This is an automated alert generated by Pandora FMS
Please contact your Pandora FMS System for more information. Please *DO NOT* reply to this email.

The default action is the one which we've defined before: All the alerts which use this template are going to use the predefined action by default, except if modified.

In case 3, we're going to see that the Alert System can be configured to notify us if the alert stops.


Qgcpu8.png

It's almost the same as above, but 'Field1' is not defined, because it's going to be used in the same way as defined within the action executed before (if the alert is fired). In this case, it's going to send an email which informs us of a recovered condition in the module named 'cpu_user'.

The alert recovery is optional. Please keep in mind that if there are defined fields within the alert recovery data ('Field2' and 'Field3'), these fields are going to ignore and overwrite the action's fields, because they have precedence to them. The only valid field which is impossible to be modified by them is 'Field 1'.

Associating an Alert to a Module

Now that we're already having all we need, we just have to associate the alert template to the module. To do so, we're required go to the 'Alert' tab within the agent where the module is located:


Qgcpu9.png

We've created an association between the module named 'cpu_user' and the 'critical condition' alert template. It's going to show the predefined action in this template ('Send email to XXX') by default.

Scaling Alerts

Alert scaling consists of the possibility of performing different actions, depending on the severity of the situation. This severity is determined by the number of times a value which is considered 'out of range' occurs. If an alert is e.g. fired if the CPU of a system is at 90% strain, it's possible to configure it to send an email in any case and an SMS if the value's 'out of range' condition has been met more than five times.

This type of alert scaling is basically accomplished by configuring more than one action within the alerts, especially by filling out the fields 'From' and 'To' extensively.



Ciempies.jpg



When an alert recovers, all the actions that have been executed up to that point will be executed again, not just the one that matches the 'From' , 'To' configuration.

Stand-By Alerts

Alerts can be defined as 'active', 'deactivated' or in 'standby'. The difference between 'deactivated' alerts and 'standby' is that 'deactivated' alerts aren't going to be fired at all. They're also not going to be shown in the alert's view. 'Standby' alerts will be shown in the views. They're also going to work - but only on the visualization level. They're going to show you whether they're fired or not, but they're neither going to perform the assigned actions nor generate events.

Standby alerts are useful to see what happens, but they're disabling the notifications and actions.

Cascade Protection

The Cascade Protection is a Pandora FMS feature which allows you to avoid a 'flooding' of alerts if a group of agents can't be reached due to a connection failure. These kinds of things tend to happen if an intermediate device such as a router or a switch is down and all the devices which come behind it simply cease to be reachable by Pandora FMS. It's probable the devices are working as they're supposed to, but if Pandora FMS can't reach them by the use of 'ping', it considers them to be 'down'.



Recursive cascade protection ilustration.png



The cascade protection is activated from the agent's configuration menu. You may activate and deactivate it by clicking on the check box named 'cascade protection'.



Down1.jpg



If the cascade protection is activated within an agent, the alerts with father CRITICAL state are checked either this would be simple or correlated. In this way if any father has a critical alert fired, the the alerts configured in the agent will not be fired. These alerts will be fired if the agent father does not have any module in a CRITICAL state or if they are fired by the father with an state lower than CRITICAL. It is understood that the agent will launch the alerts if the required conditions are fullfied.

So as the cascade protection works well, it is convenient to configure in all the fathers an alert with a CRITICAL state that check if the device is down. Besides, in order to avoid that an alert from an agent defined as a father and the other alerts have the CRITICAL state.

Examples

You're going to have the following monitoring types at your disposal:

  • Router: An ICMP and SNMP check utilizing a standard OID to obtain the ATM's port status along with a latency check, intended for your parent's or provider's router.
  • Web Server: You'll also going to have several internal checks performed by the Pandora FMS Agent, e.g. 'CPU Usage', 'MEM Usage' and 'Process Check' for your
    Apache Web Server along with a 4-step navigational HTTP latency check.
  • Database Server: In addition you're going to have several internal checks running on the Pandora FMS Agent at your disposal: 'CPU Usage', 'MEM Usage' and 'Process Check' for your database along with integrity checks for it. You're also able to check the remote connectivity to the database using a plug-in-defined test to login, to conduct a query and exit and a possibility to measure the response time of the answer.

You're also able to define several alerts. We suggest to define them in the following way:

  • Router:

SNMP Check / 'critical'-> Action: Send mail.

  • Web Server:

Process / 'critical' -> Action: Send mail. HTTP Latency / 'critical' -> Action: Send mail.

  • Database Server:

Process / 'critical' -> Action: Send mail. SQL Latency / 'critical' -> Action: Send mail.

We recommend to define the router as a parent device for the Database and Web Servers and to enable the Cascade Protection within both of their agents.

If e.g. the router connection is down, Pandora FMS receives information from the Web and Database Servers by using that connection within which you haven't activated the Cascade Protection, you're going to receive six alerts. Just try to imagine the effect if you e.g. have 200 Servers connected by this particular router. That's the reason for why it's sometimes called an 'Alert Storm'. In worst-case scenarios, this problem has the potential to kill your Mail and Monitoring Servers or your cellphone, because they're getting flooded by lots of alerts or SMS messages.

However, if you have the Cascade Protection enabled, you're only going to receive one alert, which e.g. says that the ATM interface on your router is down. You're still going to see the Web and Database Servers bearing a red status, but you won't receive tons of alert mails by them anymore.

List of special days (Version <= 6.0)

From versions 5.0 and above, Pandora FMS has a new 'Special Days' feature. It allows to define holidays and special working days for an alert template. The days defined within the Special Days list are treated as the same day of the week you've selected there.

Creating a Special Day

The new special days are created by clicking on 'Administration' -> 'Manage Alerts' -> 'Special Days List' and the 'Create' button.
Pandora 6.0: The new special days are created by clicking on 'Alerts' -> 'List of Special Days List' and the 'Create' button.



Creating special day1.png



Once you've clicked on 'Create', a window as the one shown below is going to appear.



Creating special day2.png



This is an explanation for the options you're going to encounter here:

  • Date: The special day's date. The data format is 'YYYY-MM-DD'. If you want to define the same day in every year, you may use wildcards like '*' for the 'YYYY' entry.
  • Same Day of the Week: Please select a day. The above date is treated the same as the selected day.
  • Description: The Special Day's description.

Let's assume for a moment that May 3, 2012 would be a holiday. If you define the date of '2012-05-03' as a 'Sunday', that day is treated in the same way as a Sunday would.

Once you have filled out the appropriate fields, please click on the 'Create' button.


Template warning.png

To enable the Special Days List, the option named 'Use Special Days List' should be preset within the Alert Template as explained as in Step 2.

 


Editing a Special Day

You may edit the Special Days created within the 'Special Days List' option by clicking on 'Administration' and 'Manage Alerts'.
Pandora 6.0: You may edit the Special Days created within the 'List of Special Days' option by clicking on 'Alerts'.



Editing special day1.png



To edit a special day, please click on its date.



Editing special day2.png



Once your changes are completed, please click on the 'Update' button.

Deleting a Special Day

In order to delete a Special Day, please click on the gray trash icon which is located on the far right side of the window.



Deleting special day.png



List of special days (Version >= 6.1)

From version 6.1 of Pandora FMS has special days feature with calendar style. It allows to define holidays and special working days for alert template. Days defined in special days list are treated as the same day of the week you selected.

Creating a special day

The new special days are created by clicking on 'Alerts' -> 'List of special days' and the Plus button in the calendar or 'Create' button at the bottom.



Creating special day61-1.png



Once clicked on Plus or Create, a screen as follows appears.



Creating special day2.png



Next are the fields that you should fill in:

  • Date: Special days date. The data format is YYYY-MM-DD. If you want to define same day every year, you can use '*' for YYYY.
  • Group: Select a group which is applied special days.
  • Same day of the week: Select a day. The above date is treated as the same as selected day.
  • Description: Special day description.

For example, assume that is a holiday, May 03, 2012. When you define '2012-05-03' with 'Sunday', that day is treated as the same as Sunday.

Once you have filled the fields, click on the Create button.

Note: To enable special days list, "Use special days list" should be set on the alert template (step 2).

Bulk creation of special days using iCalendar(.ics) file

Special days can also be created using iCalendar(.ics) file. An iCalendar file can be uploaded from the top of the window. Once uploaded, subsequent data from the current month will be registered.



Creating special day ics.png



Editing a special day

It is possible to edit the special days created from option Special days list at the Manage Alerts menu.



Editing special day61-1.png



To edit a special day, click on the wrench icon on the right side of the day.



Editing special day2.png



Once these changes have been done, click on the "Update" button.

Deleting a special day

To delete a special day, click on the gray trash icon on the right side of the day.



Deleting special day61.png



Complete Alert Examples

Sending SMS Alerts

In this example, we're going to see something we see very often: To send an SMS either if something happens or it's about to happen.

To accomplish this, we're going to use a script you may download from our Pandora FMS Module Library. This script uses a commercial Perl API to send the SMS by using a commercial HTTP gateway (for which you're required to create an account and to pay a small fee). This is very easy to do, because once you've set up the account and configured the script, it's ready to be put to use. You're just required to enter your user name and password to use it.

If you've already configured your SMS account and installed the script on the Pandora FMS Server, please enter the following command:

> sendsms 

You're required to enter three parameters: <source>, <destination> and 'complete message'.
Please keep in mind to encapsulate the message in single quotes (') and to enter the destination number
by using the international code format (e.g. 346276223 for Spanish phones).

After we've made sure the 'sendsms' command is ready to be used, the first thing we have to do is to define the 'alert' command. We're going to define the command within the Pandora FMS Administration Interface:



Smsalert sample1.png



Within this command, we're going to define '346666666666' as the source of the message. We could use an alphanumerical word here, but we're not going to do that, because some mobile phone providers can't handle alphanumeric IDs very well. 'Field 1' and 'Field 2' are going to be used to define the command's behavior. On the photo of the mobile phone which receives the SMS, we've used a string identifier named 'Aeryn'. 'Field 1' is the field in which the destination phone is defined, while 'Field 2' is going to be used for the text which is defined within the alert's action.

Now we're going to define the alert's action. It going to execute the predefined command and replaces 'Field 1' and 'Field 2' by custom values. In this specific case, the template's alert doesn't return any data within the SMS. All information is defined in the alert's action.



Smsalert sample3.png



The number in 'Field 1' is my phone number (and it's a bit obfuscated, because I don't want be called in middle of the night). The SMS text message is located in 'Field 2'. I'm going to use a few macros here - which will be replaced within the runtime in the moment the alert is produced.

Final Step: We're going to create an Alert Template (please skip this if you already have one). I intend to create a very simple alert template that's just going to fire if a module goes to 'critical' status. That alert will fire max. once per day - but if it recovers, it's going to fire again each time it recovers.



Smsalert sample5.png





Smsalert sample6.png



Please assign a module along with an alert template and an alert action:



Smsalert sample4.png



To get this alert fired, the module is required to be in 'critical' state. On the picture below, I'm going to review the module's configuration to see if their 'critical' thresholds are properly defined. If they weren't, the alert is never going to be fired because it's waiting for the moment to reach the 'critical' status. In my case, I've set it to the value of '20'. If a low value gets received, the module will go to a 'critical' state and the alert is going to be fired.



Smsalert sample7.png



All done. We may 'force' the alert in order to execute and test it. To force the alert, please go to the agent's alert view and click on the green, round-shaped icon:



Smsalert sample8.png



Just like on the picture below, an SMS should be appearing on my cellphone. I get an 'N/A' related to the data, because no actual data is received by the module if you force the firing of an alert.



Smsalert sample9.png



Using Alert Commands different from Email

The internal email is defined as a non-configurable command to Pandora FMS, because 'Field 1', 'Field 2' and 'Field 3' are fields which are clearly intended to be used for 'addressee', 'subject' and 'message text' - but what am I supposed to do if I intend to execute a user-defined action ?

We're now going to define a new command - something completely defined by us. Let's suppose that we intend to generate a log file entry for each alert we encounter. The format of this log file entry should be something like this:

DATE_HOUR - NAME_AGENT -NAME_MODULE -VALUE- PROBLEM DESCRIPTION

'Value' is going to be the module's value in this specific moment. There will be several log file entries, depending on the action which calls the command. The alert is going to define the description and the file to which the events are going to be added.


To accomplish this, we're required to create a command like this first:



Qgcpu11.png


Subsequently, we're defining an action:


Qgcpu12.png

If we take a look into the created log file, we're going to see the following:

2010-05-25 18:17:10 - farscape - cpu_user - 23.00 - Custom log alert #1

The alert was fired at '18:17:10' within the agent named 'Farscape', in the module named 'cpu_sys' containing the data of '23.00' and the description we've entered in the moment we've defined the action.

As for the command execution, the field order and the other things we're likely not to understand very well (e.g. how the command is executed), the easiest way to learn is to activate the Pandora Server's debug traces within the server's configuration file located at '/etc/pandora/pandora_server.conf'. Please restart the server by entering '/etc/init.d/pandora_server restart', look for the file named '/var/log/pandora/pandora_server.log' and look for the exact line which contains the execution of the user-defined alert command to see how the Pandora FMS Server is firing it in detail.

Complete Example of an Alert by Substitution Macros

Let's suppose for a moment you intend to generate a log entry in which each line is supposed to show its data the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command Configuration:

echo _timestamp_ pandora _field2_ >> _field1_

Action Configuration:

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template Configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovery section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

If you execute an alert, the following line is going to be written into the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status


The following line is going to be written into the log if the alert is recovered:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

Custom module alert macros

Any number of custom-made Module Macros may be added to an agent module.



Add module macros.png


These macros have the following characteristics:

  • Defined in the module configuration section
  • Store the information in database
  • Can have any name for example: _pepito_
  • Doesn't affect the agent configuration files(pandora_agent.conf)
  • Can only be used in the alert system.
  • Can't be added to the local components.
  • Can be added to modules in the policies.

These specific macros can be added by just expanding the module macros section.



Module macros.png


The macro values can be used as part of the fields in alert definitions. For Example:

To include a macro to the mail to xxx action and send an e-mail, when the alert fires, the field with the e-mail body must be configured in the following fashion:



Campos alertas.png


If a module is added without any defined custom macro then no information would be displayed for the value of the macro in the body of the e-mail when an alert fires.

Email configuration with a Gmail account

In order to configure Pandora FMS to send alerts via Gmail, Pandora and Postfix must be configured this way:

Pandora's Configuration

In order to properly configure your email with a Gmail account, all the fields must have the following comments in the Pandora FMS server configuration file (/etc/pandora/pandora_server.conf) except the mta_address field, which will be configured with the IP server or localhost (where the postfixserver is installed).

If Postfix is installed in the same server than Pandora FMS, the configuration in the pandora_server.conf would be like this:

mta_address localhost 
#mta_port 25
#mta_user myuser@mydomain.com
#mta_pass mypassword
#mta_auth LOGIN
#mta_from Pandora FMS <pandora@mydomain.com>


Now, I would like to show you briefly how to configure an alert in the Pandora FMS console.

Action Setup

To set the mail recipient, use the mail action to XXX so you can add an email recipient to which all the mail alerts will be sent.


GMAIL1.png

Alert setup

In this case, the module configuration has been generated in the module configuration> Alerts, a new alert with the module as the one that you can see in the screenshot below.


GMAIL2.png

Once the alert is fired, you can see how the alert reaches the e-mail picked in the action:


GMAIL3.png


GMAIL4.png

Postfix Setup

Assuming you already installed Postfix and everything works fine except sending to gmail smtps, here are the steps to follow:

1-- Edit the /etc/postfix/main.cf configuration file and add the following lines at the end of the file:

relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl/passwd
smtp_sasl_security_options = noanonymous
smtp_use_tls = yes
smtp_tls_CAfile = /etc/postfix/cacert.pem

2-- Create the /etc/postfix/sasl/passwd file with your gmail address and password (you must create the “sasl” directory and then create the passwd file in there).

To create the “sasl” directory:

mkdir /etc/postfix/sasl

To create the passwd file:

nano /etc/postfix/sasl/passwd

And paste the line below with your own gmail address and password inserted:

[smtp.gmail.com]:587 ACCOUNT@gmail.com:PASSWORD

Protect the password file accordingly:

chmod 600 /etc/postfix/sasl/passwd

This will allow only root users to access the file.

3-- Transform /etc/postfix/sasl/passwd into a hash type indexed file. This will create a lookup table via postmap:

postmap /etc/postfix/sasl/passwd

Issuing this command will create a passwd.db file in the /etc/postfix/sasl/ directory.

4-- Now to install the Gmail and Equifax certificates. Pre-built Pandora FMS ISO and VMware virtual image do not have these certificates by default. If you have the certificates installed, then you can skip this part.

To install the Gmail certificate, follow these steps:

Google’s SSL cert is signed by Equifax – so first we need to fetch that. Move to “tls” directory:

cd /etc/pki/tls/

We need to download Equifax certificate.

sudo wget -O Equifax_Secure_Certificate_Authority.pem https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.cer

Now let’s add the permissions to the downloaded file:

chmod 644 Equifax_Secure_Certificate_Authority.pem

We also need to request the signature for the certificate:

openssl x509 -in Equifax_Secure_Certificate_Authority.pem -fingerprint -subject -issuer -serial -hash -noout

Next we need need to install the GMail cert. The first thing we need is the c_rehash util, so lets install its package:

yum install openssl-perl

If you receive errors attempting to install openssl-perl, I took the following additional steps to resolve this problem:

 sudo su
 nano /etc/yum.repos.d/extra_repos.repo
 In the #percona repository I changed the baseurl line to:  http://repo.percona.com/centos/6/os/x86_64/
 ^O to write the edited file
 ^x to exit
 After returning to root terminal, enter "yum install openssl-perl" and accept the defaults

Next we need to actually acquire the certificate for GMail. So use openssl to do this:

openssl s_client -connect pop.gmail.com:995 -showcerts

The output should contain the required lines for the certificate and we need to copy them to /etc/pki/tls/gmail.pem file. For this, create the file:

nano /etc/pki/tls/gmail.pem

and paste these lines into the gmail.pem file:

-----BEGIN CERTIFICATE-----
MIIDWjCCAsOgAwIBAgIKYgy3qQADAAAJ5zANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0wOTA3MTcxNzE2NTVaFw0xMDA3MTcxNzI2NTVa
MGcxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRYwFAYDVQQDEw1wb3Au
Z21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDTHqjJfnRXdpmZ
4iP/WNCpvzX4N97bEZ3rvS4aDYey/DJetKZqp9DK1Ie4/C5j8M1aakwiTNA/eHS/
wNWVgQx8+HxproYKUeeYj3shYKEkHGfrRYBcyCxc7Gd6NSGaaYru3Z7nJ+STIPUJ
E1N35JAwcjjdITVI2O4LckAL4b7GkwIDAQABo4IBLDCCASgwHQYDVR0OBBYEFIln
0T5I8Mw6cqhtUS4pyMGYRxOTMB8GA1UdIwQYMBaAFL/AMOv1QxE+Z7qekfv8atrj
axIkMFsGA1UdHwRUMFIwUKBOoEyGSmh0dHA6Ly93d3cuZ3N0YXRpYy5jb20vR29v
Z2xlSW50ZXJuZXRBdXRob3JpdHkvR29vZ2xlSW50ZXJuZXRBdXRob3JpdHkuY3Js
MGYGCCsGAQUFBwEBBFowWDBWBggrBgEFBQcwAoZKaHR0cDovL3d3dy5nc3RhdGlj
LmNvbS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS9Hb29nbGVJbnRlcm5ldEF1dGhv
cml0eS5jcnQwIQYJKwYBBAGCNxQCBBQeEgBXAGUAYgBTAGUAcgB2AGUAcjANBgkq
hkiG9w0BAQUFAAOBgQCEGIebkDpktdjtzMiTTmEiN7e4vc73hEI4K0jYKyY0Wn5N
dc44AXTfIWOzsikwb886PCUSevGs9rcw2/kaHdPaBSuGrzSCf8ODQqTC3odry3lo
PtZGr6nf/81F5UW71+bE1iWOQlJ5/olWOr2SlqYla1iOmosEctD/GyoFnDh+BA==
-----END CERTIFICATE-----

Next we need to run the c_rehash util:

cd /etc/pki/tls

and

c_rehash .

Finally, we can test it with:

openssl s_client -connect pop.gmail.com:995 -CApath /etc/pki/tls

The important point is to Verify the return code:0 (ok), and the final OK Gpop ready. If you get them then you can connect to GMail.

Now let’s create the Equifax_secure_CA.pem file:

nano /etc/ssl/certs/Equifax_Secure_CA.pem

Paste the following certification lines:

-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIENd70zzANBgkqhkiG9w0BAQUFADBOMQswCQYDVQQGEwJV
UzEQMA4GA1UEChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2Vy
dGlmaWNhdGUgQXV0aG9yaXR5MB4XDTk4MDgyMjE2NDE1MVoXDTE4MDgyMjE2NDE1
MVowTjELMAkGA1UEBhMCVVMxEDAOBgNVBAoTB0VxdWlmYXgxLTArBgNVBAsTJEVx
dWlmYXggU2VjdXJlIENlcnRpZmljYXRlIEF1dGhvcml0eTCBnzANBgkqhkiG9w0B
AQEFAAOBjQAwgYkCgYEAwV2xWGcIYu6gmi0fCG2RFGiYCh7+2gRvE4RiIcPRfM6f
BeC4AfBONOziipUEZKzxa1NfBbPLZ4C/QgKO/t0BCezhABRP/PvwDN1Dulsr4R+A
cJkVV5MW8Q+XarfCaCMczE1ZMKxRHjuvK9buY0V7xdlfUNLjUA86iOe/FP3gx7kC
AwEAAaOCAQkwggEFMHAGA1UdHwRpMGcwZaBjoGGkXzBdMQswCQYDVQQGEwJVUzEQ
MA4GA1UEChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2VydGlm
aWNhdGUgQXV0aG9yaXR5MQ0wCwYDVQQDEwRDUkwxMBoGA1UdEAQTMBGBDzIwMTgw
ODIyMTY0MTUxWjALBgNVHQ8EBAMCAQYwHwYDVR0jBBgwFoAUSOZo+SvSspXXR9gj
IBBPM5iQn9QwHQYDVR0OBBYEFEjmaPkr0rKV10fYIyAQTzOYkJ/UMAwGA1UdEwQF
MAMBAf8wGgYJKoZIhvZ9B0EABA0wCxsFVjMuMGMDAgbAMA0GCSqGSIb3DQEBBQUA
A4GBAFjOKer89961zgK5F7WF0bnj4JXMJTENAKaSbn+2kmOeUJXRmm/kEd5jhW6Y
7qj/WsjTVbJmcVfewCHrPSqnI0kBBIZCe/zuf6IWUrVnZ9NA2zsmWLIodz2uFHdh
1voqZiegDfqnc1zqcPGUIWVEX/r87yloqaKHee9570+sB3c4
-----END CERTIFICATE-----

Save and exit.

In order to add the Equifax certificating authority (which certifies emails from Gmail) into the certificate file that postfix uses, run the following command in a root console:

cat /etc/ssl/certs/Equifax_Secure_CA.pem > /etc/postfix/cacert.pem

5 - Finally, restart postfix to apply the changes:

/etc/init.d/postfix restart

6 - You can verify the performance by opening two consoles. You should execute the following command in one console to monitor the behavior of the mail:

tail -f /var/log/mail.log

You can send an email through the other one:

echo "Hello" | mail your-email-address@gmail.com

You also may need to change the settings under your gmail account (under the “devices” tab) to receive the e-mail. You can also turn on access for less secure apps and read more about it from here: https://www.google.com/settings/security/lesssecureapps

If you have done everything right, something like that should appear in the other console:

Dec 18 18:33:40 OKComputer postfix/pickup[10945]: 75D4A243BD: uid=0 from=
Dec 18 18:33:40 OKComputer postfix/cleanup[10951]: 75D4A243BD: message-id=
Dec 18 18:33:40 OKComputer postfix/qmgr[10946]: 75D4A243BD: from=, size=403, nrcpt=1 (queue active)
Dec 18 18:33:44 OKComputer postfix/smtp[10953]: 75D4A243BD: to=prueba@gmail.com, relay=smtp.gmail.com[74.125.93.109]:587, delay=3.7,  delays=0.15/0.14/1.8/1.6, dsn=2.0.0, status=sent (250 2.0.0 OK 1324249500 eb5sm36008464qab.10)
Dec 18 18:33:44 OKComputer postfix/qmgr[10946]: 75D4A243BD: removed

If the result is similar, Pandora is properly configured and linked to the Postfix server, so it will send mails as expected.

Special Notes: Communicating with gmail can be tricky and I ran into a problem where the maillog indicated "Network is unreachable" - this required me to edit the connection protocol for Postfix to communication with the gmail smtp server, as follows:

 sudo nano /etc/postfix/main.cf
 Find the line that says, "inet_protocols = all" and change to "inet_protocols = ipv4"
 then
 sudo /etc/init.d/postfix restart to restart Postfix.

Go back to Pandora FMS documentation index