Pandora:Documentation en:Operations

From Pandora FMS Wiki

Jump to: navigation, search

Go back Pandora FMS 3.0 documentation index

Contents

Monitoring with Software Agents

Agent Configuration

Pandora FMS agents have a local configuration by default, so their maintenance will be managed from the monitored machine which edits their configuration file.

Remote Configuration

On the enterprise version, there is a remote Agent Configuration feature which allows their centralized configuration and file management from the server console.

The configuration consists of two files. Their file names are <md5>.conf and <md5>.md5, where <md5> is the agent's name hash code. Those files are stored in '/var/spool/pandora/data_in/conf' and '/var/spool/pandora/data_in/md5' folders.

Those files are composed of plain text and could be edited from the console, which regenerates them if there is any change. Communication is always in the direction from agent to server, which means the functionality is controlled by the agent.



Xml file transfer


In order to enable the remote configuration, you're required to edit the 'remote_config' parameter from the 'pandora_agent.conf' file, setting it to '1'. Once this change is done, the agent's configuration file will be ignored, because it's detecting a change in the configuration. It's going to download the new version from the server. This is a good way to force the server to get the agent's local configuration to delete both configuration files from the server.

Common Configuration Parameters

In the Pandora FMS Software Agents section you can find a complete explanation on Agent Configuration. In this section, the common parameters used to configure the Software Agents are going to be explained.

The most common parameters are:

  • server_ip: IP direction of Pandora FMS Server.
  • server_path: Path of the 'incoming' folder for the Pandora FMS server (it's '/var/spool/pandora/data_in' by default).
  • temporal: Software agent's temporal folder (it's '/var/spool/pandora/data_out' by default).
  • logfile: Software agent's log file (it's '/var/log/pandora/pandora_agent.log' by default).
  • interval: Agent's execution interval (it's '300' by default).
  • intensive_interval: Intensive module execution interval (it's '300' by default).
  • debug: Debug mode enable ('0' - it's disabled by default).
  • agent_name: Agent name (hostname is taken by default).
  • server_port: Tentacle server port (it's '41121' by default).
  • transfer_mode: File-transfer mode (it's 'tentacle' by default).
  • remote_config: Activation of remote configuration ('0' - it's disabled by default).

An example of the general parameters for a UNIX configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        /var/spool/pandora/data_out 
logfile         /var/log/pandora/pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1 

An example of the general parameters for a Windows configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        c:\program files\pandora_agent\temp
logfile         c:\program files\pandora_agent\pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1

Custom Fields

Custom fields are an easy way to personalized agent's information. You're able to create custom fields by klicking on 'Administration' -> 'Manage monitoring' -> 'Manage custom fields'.





If you want to create a custom field, you have to click on the 'Create field' button and fill in the fields as described below:





  • Name: Name of the custom field.
  • Display on front: With this field checked, the custom field will be displayed in front of the agent like on the screenshot below:




Monitoring with the Software Agent

The data collected by the software agents is kept in small information pieces called 'modules'. Each module only keeps one kind of data. Each module's value is the value of a supervised variable. Once the agent starts sending information, the data will start to consolidate in the database where you're able to access it.


Please check the Software Agents Installation Section to obtain more information about them.


The Pandora FMS Software Agents use the operating system's specific commands to obtain information. The Pandora FMS Data Server keeps and processes data generated by these commands which are sent to the server in form of an XML file. The information which gets returned by these commands is contained in what we call 'Modules'.



In the agent configuration file, the modules are defined by the following structure:

module_begin 
module_name cpu_user
module_type generic_data
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_description User CPU Usage (%)
module_end

The parameter which indicates how to get the module information is called 'module_exec' . This parameter specifies the command which the agent should execute in order to get the information from the system. Example:

module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'

The agent will execute the command as an operator to collect the information:

$> vmstat 1 2 | tail -1 | awk '{ print $13 }'

Then the agent gets the returned value by the command and attaches it to the XML file as module data. Of course the agent can execute any program or script which returns the monitoring value. The field 'module_exec' could be for example:

module_exec myScript.pl --h 127.0.0.1 -v cpu

Again the agent is going to execute the command and collects the returned value in the same way as an operator would type in the shell:

$> myScript.pl --h 127.0.0.1 -v cpu


When the software agent is executed for the first time, it sends an XML to the Pandora FMS data server which is received in the server's incoming directory through Tentacle, SSH or FTP. The data server checks this directory every X time and when it finds a file, it gets processed. On opening this data file (XML), it identifies the agent by its unique name. This is because each agent is required to have a completely unique name, where capital and small letters are distinguished by Pandora FMS. The server creates all agents automatically from which it receives data and which aren't already logged in on the DB by default. In the same way, if the agent has been added in 'learning mode', the modules which haven't been previously defined in the agent will be created automatically by the Server.


Image:template_warning.png

Configuration file encoding. It's very important and has to have exactly the same value as the one set in the encoding configuration parameter. If the encoding is properly set, you're going to avoid the reception of data, bearing wrong encoding characters.

 


Kinds of Modules

There are several kinds of modules, but they're mainly classified in two categories: Data which originated on software agents and data which originated on network modules, executed by a network server. The modules identified as 'generic' are the ones which originate on software agents and those identified as 'remote' are network modules.

generic_data

A kind of numerical data. It's pretty useful to keep numerical data (in whole numbers and floating point) obtained through a Pandora FMS agent module.

generic_data_inc

A kind of increasing numerical data. It keeps data which results in the difference between the last agent data and the current data. The Pandora FMS server calculates and keeps the rate by second automatically. All of the modules which end on 'inc' are of the incremental kind. This kind of data is used to count the 'number of times' of something, e.g. the entries in a log, bytes/sec, connections/sec, etc.

generic_proc

They are generally called "monitors", too. They belong to the boolean kind of data where a value of '0' means 'false' or 'bad value', and values higher than '0' mean 'right' or 'right value'. The 'generic proc' kind is also called 'monitor', because they can show if something is right or wrong without the need of interpreting or establishing alerts on it. They are displayed in the agent view as small lights. Red if it's '0', green if it's higher than'0'. All of modules which end on 'proc' are monitors.

generic_data_string

Kinds of alphanumeric data (text).

async_data

It's a kind of asynchronous numeric data. It's the same as 'generic_data' but for asynchronous data which are only updated if there is a change. The asynchronous kind of data don't have a defined periodicity when we can obtain data.

async_string

This is a kind of asynchronous alphanumeric data. It's the same as 'generic_string' but for asynchronous data which are only updated if there is a change. It's the kind of data that you're recommended to use if you want to monitor searches in logs or event viewers. We could have new data by any second or not having one at all in many days.

async_proc

It's a kind of asynchronous boolean data. It's the same as 'generic _proc' but for asynchronous data which are only updated if there is a change.

The software agent already comes preconfigured to send certain data from the system on which it's installed. These usually are (depending on the version):

  • System CPU
  • Available space on the harddrive
  • Free memory
  • Monitor of the programs and services states

Depending on the software agent, they are also going to look for one operating system or another. They also might have more modules or different things to check.

All these information is located in the file 'pandora_agent.conf'. This file is in the directory '/etc/pandora/' under GNU/Linux and, under the default Windows installation, the directory is 'C:\Archivos de Programa\pandora_agent\' or 'C:\Program Files\pandora_agent\' or similar.

We are now going to explain the kind of data for some of the modules:

Percentage of CPU usage under GNU/Linux

# CPU usage percentage (GNU/Linux)
module_begin
module_name cpu_user
module_type generic_data
module_interval 1
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_max 100
module_min 0
module_description User CPU Usage (%)
module_end

It's obvious the kind of module is from the 'generic_data' kind which executes a GNU/Linux console command to obtain the result ('module_exec'). The maximum is set to '100' and the minimum to '0'. The interval ('module_interval') represents the number of iterations between the execution of each module. If it's different from '1', the module will only start the execution of the agent the number of times which is set there. If e.g. the agent execution time is set on '300' and the module interval is '3', the module is going to be executed every 900 seconds.

Percentage of the CPU usage under Windows:

# CPU usage percentage (Windows)
module_begin
module_name CPUUse
module_type generic_data
module_cpuusage all
module_description CPU#0 usage
module_end

As you can easily see, the module is completely different from GNU/Linux under Windows. Under Windows it's an internal agent command where module_cpuusage all represents the CPU usage for all CPUs. Using 'module_cpuusage' is only going to calculate the CPU usage for CPU #0. The rest of the fields are optional.

To add one more module, please check the agent configuration and create a valid module block. Once you have done this, keep the agent configuration file and restart the agent, no matter if it's the UNIX daemon or the Windows service.

Module Creation Interface

The creation of local modules from the console is done using a form in the shape of a text box. This is the location to specify the configuration data, which will be placed into the configuration file of the software agent (besides the common configuration with the remote modules like thresholds, type and group).

Creation of a module with the remote config enabled on the agent:





In vicinity of this text box there are two buttons: One to create a basic configuration structure and the other to check that the data is correct. This check is intended for basic parameters, e.g. checking if it begins with 'module_begin', ends with 'module_end' and if it has a valid type and name. It's possible of other parameters to be wrong, but it won't be detected here.













The field name and the type combo are linked to the parameters 'module_name' and 'module_type' of the data configuration. If you change the module name on the name field, the configuration data name will be changed automatically and vice versa. If you select a type in the combo, the data configuration type will be changed and if a correct type was written in the configuration data, this type will be selected automatically in the combo.

When a module from a local component is changed, it might have macros. If it has macros, the configuration data will be hidden and a field for each macro is going to appear instead. You can find a detailed explanation of this feature in the following section:

Templates and components

Conditional Monitoring

The Pandora FMS Software Agent supports the execution of a script as a postcondition, depending on the module value. Also the software agent has a feature to evaluate a precondition before module execution. We're going to explain both features and provide examples in this section for them.

Post-Conditions

The parameter module_condition defines a postcondition in the moment of module execution. It means this parameters define the command which will be executed depending on the value returned by the module. The structure in the configuration file is:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

You can define multiple postconditions for the same module, e.g.:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition (90, 100) remove_processes.sh
module_condition < 20 add_processes.sh
module_end

Some examples:

Execution if the module data is less than '20' :

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

If the script named 'get_cpu_usage.pl' returns '18', the software agent is going to execute the script 'add_processes.sh', otherwise it won't.

Execution with two preconditions:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 10 start_new_server.sh
module_condition < 20 add_processes.sh
module_end

If the module returns '15', the software agent is going to only execute the script named 'add_processes.sh' - but if the value is '6', the script is going to execute both scripts named 'start_new_server.sh' and 'add_processes.sh'.

Pre-Conditions

The parameter 'module_precontition' defines a precondition to evaluate before a module execution. Depending on the result of this precondition, the software agent will execute the module or not. The structure of the configuration file is:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

You can e.g. define multiple preconditions for the same module:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

Some examples:

Module execution if the precondition is above '10' only:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

The software agent executes the script 'number_active_processes.sh', if the returned value is greater than '10'. If the returned value is lower than '10', the module will never be executed.

Module execution if only two of the preconditions are met::

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

We have two preconditions in this example. To execute the module (both of them in this case), all preconditions must be met. The module will only be executed if the script 'number_active_processes.sh' returns a value greater than '10' and when the script 'important_service_enabled.sh' returns a value of '1'.

Intensive Monitoring

(Only available in versions 5.0 or above)

Certain values are very important for some modules, while others are not. If when you are e.g. monitoring a system service, you want to be notified as soon as possible if the service goes down - but you don't need to be reminded the service is up every ten seconds.

This is where intensive modules are coming in. Intensive modules run with an interval of intensive_interval in seconds. When intensive conditions are met, they run the rest of the time with an interval of interval seconds like the rest of the modules.

If you want e.g. to check whether the SSHD system service is running every 10 seconds, but want to be notified every 10 minutes if the service is fine, you may configure the agent in the following way:

intensive_interval 10
interval 600
module_begin
module_name SSH Daemon
module_type generic_data
module exec ps aux | grep sshd | grep -v grep | wc -l
module_intensive_condition = 0
module_end

If the service goes down, you will be notified in the next 10 seconds. If the service is up, you will be notified in the next 10 minutes.

This can save a lot of space in the Pandora FMS Database.

Programmed Monitoring

The software agent supports the definition of programmed modules which are executed in the defined instants. The syntax is the same as crontab. For a detailed explanation, see http://en.wikipedia.org/wiki/Cron#Predefined_scheduling_definitions

The module structure is the following:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_end

In this example the module is executed once, every Monday from 1200 to 1500 hours.

If you want a module which is executed during all the interval, we have to set the option 'module_cron_interval' to '0' in this way:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_cron_interval 0
module_end

If we're required to execute the module every hour past ten minutes we can use this module definition:

module_begin module_name crontab module_type generic_data module_exec script.sh module_crontab 10 * * * * module_cron_interval 0 module_end

Specific Monitoring for Windows

The software agent for Windows has specific features to make monitoring a lot easier. These features are explained with some examples:

Monitoring Processes and Watchdog Process

Monitoring of Processes

The parameter module_proc verifies if a process with a preset name is running on this machine. The module definition is:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_description Process Command line
module_end

If the process name has blanks, so please don't use «" "». The process name is the same as shown in Windows Task Manager (taskmngr) including the .exe extension. It's very important to use the same upper and lower-case letters.

If you want the software agent to immediately notify you if a process is not working, you're required to add the parameter module_async yes. In this case, the module definition would be:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_async yes
module_description Process Command line
module_end
Watchdog Process

The watchdog feature on the Pandora Agent for Windows allows to respond immediately to the crash of a process and restarts it. It's important to keep in mind that the watchdog only works if the module is of the asynchronous type.

The definition of a module with watchdog enabled would be as follows:

module_begin
module_name Notepad
module_type generic_data
module_proc notepad.exe
module_description Notepad
module_async yes
module_watchdog yes
module_start_command c:\windows\notepad.exe
module_startdelay 3000
module_retrydelay 2000
module_retries 5
module_end

In the previous example, the watchdog is only going to activate the 'notepad.exe' if the process is not running. It's going to execute the command c:\windows\notepad.exe each time. Other preconfigured parameters for the watchdog execution are to try 5 times with a warm-up time (delay before trying) of 3 seconds and a time of 2 seconds between retries.

Service Monitoring and Watchdog Service

Service Monitoring

The parameter module_service verifies if a specified service is running on the machine. The definition of this module is as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_end

Remember to use «" "» if the name of the service has blanks in it's name. To find the service name, you can look at the Service Name field under the Windows Service Manager. It's important to keep in mind that all names are case sensitive.

If we want the software agent to notice us immediately when a service is down, we're required to add the parameter module_async yes. The module definition would be as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_async yes
module_end
Watchdog Service

There is a watchdog mode for services which allow you to detect and restart a downed service almost in real time. A module definition example using watchdog would be the following:

module_begin
module_name ServiceSched
module_type generic_proc
module_service Schedule
module_description Service Task scheduler
module_async yes
module_watchdog yes
module_end

The watchdog definition for services which have no need for any extra parameter because they are incorporated in the service definition.

Monitoring Basic Resources

This section describes how to monitor the basic variables of a Windows-based machine.

Monitoring the CPU

To monitor the CPU, you may use the parameter module_cpuusage which returns the CPU usage percentage.

Its possible to monitor the CPU based on its ID with a module definition like the following:

module_begin
module_name CPU_1
module_type generic_data
module_cpuusage 1
module_description CPU usage for CPU 1
module_end

You're also able to monitor the average CPU usage from all systems with the following module:

module_begin
module_name CPU Usage
module_type generic_data
module_cpuusage all
module_description CPU Usage for all system
module_end
Memory Monitoring

To monitor the memory, you can use two parameters: module_freememory which returns the amount of free memory in the system and module_freepercentmemory which returns the percentage of free memory.

An example for a module using the module_freememory parameter would be:

module_begin
module_name FreeMemory
module_type generic_data
module_freememory
module_description Non-used memory on system
module_end

An example for a module using the module_freepercentmemory parameter would be:

module_begin
module_name FreePercentMemory
module_type generic_data
module_freepercentmemory
module_end
Monitoring Harddrives

To monitor the harddrive space, you may use two parameters: module_freedisk which returns the amount of available space and module_freepercentdisk which returns the percentage of available space. Both parameters require the monitored unit as an input. Please don't forget the «":"» characters.

A module which e.g. utilizes the module_freedisk parameter is defined in this way:

module_begin
module_name FreeDisk
module_type generic_data
module_freedisk C:
module_end

An module which e.g. utilizes the module_freepercentdisk parameter is defined in this way:

module_begin
module_name FreePercentDisk
module_type generic_data
module_freepercentdisk C:
module_end

WMI Queries

The Pandora FMS Software Agent allows you to extract information by using WMI queries and ODBC connections - very common technologies used to store external or system-related information.

WMI Queries

The software agent allows you to execute any local WMI query you want using the module_wmiquery parameter. To do the query, you're required to set the parameter module_wmiquery by the query which is going to be performed and to set the parameter module_wmicolumn by the column which has the information to monitor.

We're able to get e.g. a list with the installed services:

module_begin
module_name Services
module_type generic_data_string
module_wmiquery Select Name from Win32_Service
module_wmicolumn Name
module_end

We're also able to get the CPU load using WMI:

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

Remote Checks with Software Agents

A remote check performed by the agent makes it e.g. easy to monitor complex networks with have special, security-related requirements. This section explains how to use this feature of the software agent:

ICMP Checks

ICMP or ping checks come in handy if the machine is connected to a network. Only one software agent is easily able to monitor the status of the whole network in this way.

UNIX

By using the UNIX software agent, you're able to utilize the system commands to create a module which performs the ping check. An example module definition would be:

module_begin
module_name Ping
module_type generic_proc
module_exec ping -c 1 192.168.100.54 >/dev/null 2>&1; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_end

In this example module, we're going to perform a ping check on the host '192.168.100.54'. If you want to check a different host, all you have to do is to change the IP.

Windows

The software agent for Windows platforms has the following parameters to configure the ping checks:

  • module_ping_count x: Number of ECHO_REQUEST packages to send (Default value is '1').
  • module_ping_timeout x: Timeout in miliseconds (Default value is '1000').
  • module_advanced_options: Advanced options for 'ping.exe'.

A module configuration example could be:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

In this example, we're going to perform the same check as in the previous one, but now we're going to use the Software Agent for Windows platforms.

TCP Checks

TCP checks are useful to verify if a port of a host happens to be open or not. It could be interesting in case you want to know e.g. if an application is connected to the network or not.

UNIX

We're able to perform the TCP checks by the following module by utilizing the software agent for UNIX platforms:

module_begin
module_name PortOpen
module_type generic_proc
module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_timeout 5
module_end

With this module, we're going to check if port 80 of the host '192.168.100.54' is open or not.


Windows

If we want to use the software agent for Windows, we have some parameters to configure for the TCP check. The parameters are:

  • module_tcpcheck: Host to be checked
  • module_port: Port to be checked
  • module_timeout: Timeout for the check

An module definition example is this:

module_begin
module_name TcpCheck
module_type generic_proc
module_tcpcheck 192.168.100.54
module_port 80
module_timeout 5
module_end

This module is the equivalent for the Windows software agent to perform the check on port 80 of the host '192.168.100.54'.

SNMP Checks

SNMP checks are commonly used to monitor network devices to check e.g. the interface status, inbound/outbound bytes, etc.

UNIX

If you are using the software agent for UNIX platforms, you may create the module using the command 'snmpget' like this:

module_begin
module_name SNMP get
module_type generic_data
module_exec snmpget 192.168.100.54 -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}'
module_end

This module returns the value for OID .1.3.6.1.2.1.2.2.1.1.148 on the host '192.168.100.54'.

Windows

For the Windows software agent, we have the following configuration parameters for the module:

  • module_snmpversion [1,2c,3]: SNMP version (Default value is '1').
  • module_snmp_community <community>: SNMP community (Default value is 'public').
  • module_snmp_agent <host>: The host to monitor.
  • module_snmp_oid <oid>: OID.
  • module_advanced_options: Advanced options for 'snmpget.exe'.

A module example could be:

module_begin
module_name SNMP get
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148
module_end

Using the UNIX software agents, this module is the Windows platform equivalent to do the last performed check.

Proxy Mode

Image:template_warning.png

To use the agent's proxy mode on Linux or UNIX systems, the agen't must -not- be executed by a root user ! You're required to perform a custom installation of the Pandora FMS agent to do so. You may look up all the details about custom installations in the section Custom Agent Installation.

 


Pandora FMS Software Agents have a Proxy Mode which allows them to act as proxies, redirecting the communication of several agents to the Pandora FMS Server. The software agent with an enabled proxy mode is able to perform monitoring tasks, too.





The Proxy Mode is very useful if you're dealing with a network in which only one machine can communicate with the Pandora FMS Server. In this situation, all software agents installed on machines without access to the Pandora FMS Server are going to send the XML files to the agent currently working as a proxy - which is going to redirect all files to Pandora FMS Server.

In addition to XML file redirection, the proxy mode supports Remote Configuration and File Collection features.

To enable the Proxy Mode in a software agent, you're required to configure the following parameters:

  • server_ip: IP of the Pandora FMS Server. If the proxy mode is enabled, the IP is not allowed to take the following values: '127.0.0.1', 'localhost' or '0.0.0.0'.
  • proxy_mode: If it's set to '1', the proxy mode is enabled. The default value is '0' (disabled).
  • proxy_max_connection: Maximum number of connections for the proxy. The default value is '10'.
  • proxy_timeout: Proxy timemout. The default value is '1' (in seconds).

An configuration example would be:

server_ip 192.168.100.230
proxy_mode 1
proxy_max_connection 20
proxy_timeout 3

To redirect a software agent's connection, you're only required to set the IP address of the proxy mode enabled software agent to the IP address of the Pandora FMS Server on agents with limited access, e.g.:

Our proxy mode enabled agent has the IP of '192.168.100.24'.

In the software agent which can't directly connect to the Pandora FMS Server, we configure the parameter server_ip in the following way:

server_ip 192.168.100.24


This configuration allows the access-limited software agent to use the other software agent with enabled proxy mode to communicate with the Pandora FMS Server.

Broker Mode

The software agent has a Broker Mode which allows one agent to monitor and manage the configuration as if there were several software agents installed:





The software agent with enabled Broker Mode has an auxiliary configuration file (defined for each broker) which is similar to software agent's main configuration file. In this way the software agent's behaviour is the same as if several software agents were installed in the same machine, it means, each broker works in the same way a software agent does. This feature is very useful to remotely manage several devices by installing only one software agent. It's also possible to monitor several devices with different configuration files but only one software agent in this way.

The main features of the Broker Mode are:

  • To send local data as another agent. Very useful to monitor different software instances as different agents.
  • To send the collected data from the remote checks to other machines as if a software agent had been installed onto them.

To create a broker, you're only required to add a line with the parameter broker_agent <broker_name>

broker_agent dev_1
broker_agent dev_2

Once the brokers are created, the configuration files 'dev_1.conf' and 'dev_2.conf' are going to be created with the same content as in the original software agent, but with a different agent name. By adding or deleting modules from configuration files 'dev_1.conf' and 'dev_2.conf', we can customize the checks performed by the brokers.

Inside Pandora FMS web console the brokers appear and will be managed as the other agents, which means that if we have a software agent installed with two brokers, we're going to see three different agents with their modules, configurations, etc. in the web console.

Examples of Use

Monitor a local Database as a different Agent

We want to monitor the basic parameters of a machine (CPU, memory and harddrive) and an installed database which that we want to monitor separately.

To perform this monitoring, we are going to use the following structure:

  • Installed Software Agent: monitoring CPU, memory and disk.
  • Broker for the Database: monitoring internal status of the database.

We're going to install the software agent on the machine to monitor the CPU, memory and harddrive. We're also adding the following line in the software agent configuration:

broker_agent DBApp

We're going to create a broker agent called 'DBApp' by adding this line. Because of it's creation, a configuration file named 'dbapp.conf' will appear. We're adding the modules to perform the checks for the database in this configuration file:

module_begin
module_name Num Users
module_type generic_data
module_exec get_db_users.pl
module_end

module_begin
module_name Num slows queries
module_type generic_data
module_exec get_db_slows_queries.pl
module_end

By doing this, you're going to see two agents in the Pandora web console: One bearing the name of the machine with the modules 'CPU', 'Memory' and harddrive and another one called 'DBApp' with the modules 'Num Users' and 'Num slows queries'.

Remotely Monitor Devices using Brokers

For this example, we've installed a software agent on a Windows machine, monitoring CPU, memory and harddrive. We're also required to monitor a router with the IP '192.168.100.54' without being allowed to install an agent on it. To solve the problem, we can use brokers.

We're going to create a broker using the following parameter:

broker_agent routerFloor5

By adding this line, we're going to create a broker agent by the name of 'routerFloor5'. The software agent was installed on a Windows machine, so we're able to monitor the router by using the ping and SNMP modules available for Windows software agents. To do that, we're modifying the file 'routerFloor5.conf' by adding the following lines:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

In this example, the web console of Pandora FMS shows two agents: One is the Windows machine with the modules 'CPU', 'Memory' and 'harddrive' and the other one is 'routerFloor5' bearing the modules named 'Ping', 'Eth 1 up' and 'Eth 2 up'.

Remote Monitoring of a Network with limited Communication Options

In some cases, it's required to monitor devices remotely where the Pandora FMS Remote Server can't access them directly.





In this example, we have to monitor some devices from one of the company sites from the headquarters remotely. The Pandora FMS Server is connected to the other company sites in the headquarters using a VPN. Due to some restrictions, the Pandora Remote Server can't access the machines directly. To monitor the company sites, we're going to use the Broker Mode which allows a software agent to send XML files to the Pandora Server as if there would be several different devices.

We add as many brokers as there are devices to be monitored. In the configuration file of the software agent, an example configuration could be:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

Once the brokers are created, we're able to customize the monitoring for each devices by modifying the configuration file of each broker. The configuration e.g. for the Windows machine called 'device_1' is:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

We're able to use the remote configuration feature by a configuration like this. We're also able to send monitoring information to the Pandora FMS Server, despite the restrictions in the communication between the different company sites.

Shared Monitoring Load by Brokers

The broker mode is very useful to share and distribute the monitoring load within several network points.





In this example, our architecture has several networks named from A to Z with a 1000 devices each. The capacity of the Pandora FMS Remote Server is about 2000 agents, because we've decided to use broker mode enabled software agents to share and distribute the load. These broker mode enabled software agents are going to remotely monitor all devices from the network and send the data in XML format to the Pandora FMS Central Server.

For each network, we have a broker mode enabled agent. On it, we're going to create as many brokers as the number of devices to be monitored. An example configuration for the software agent 'Broker_Agent_Net_A' could be the following:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

In addition for each broker, we're going to add the modules to monitor the devices. Example: The broker 'device_1' (which is a router) could have this configuration:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

Another example configuration for the broker 'device_2' which monitors a Windows machine with the following modules:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

Using broker mode enabled software agents, we're easily able to share the load to collect the data from thousands of devices.

Inventory using Software Agents

Pandora FMS Software Agents support inventory features for both hardware and software. The inventory system allows you to get a history of CPU, cards, memory, patches, software, etc, used in the company servers. Furthermore, it's possible to generate alerts if a change occurs in the inventory, e.g. if a disk was replaced or an application was deleted.

For further information on the subject, please have a look at the section Local Inventory by Software Agents.

How to ask an Agent for On-Demand Information

Before the publication of version 3.2, there wasn't any way for asking the remote software agent for information. You were compelled had to wait for the agent to reach its interval limit to send its information. The Windows Agent 3.0 has a not very well known feature called "UDP Server" which allows to receive communication data from outside to ask for information and to force the agent to refresh its cycle which forces it to send the information to the Server.

In 3.2 version, we now have implemented the same feature called 'REFRESH AGENT' which is available under the UNIX agent as well. We also have included a 'default' alert template and commands, designed for easy handling. You now can setup your agents (for Windows and UNIX) to receive orders from the console to report the data immediately, without having to wait for it's interval any more.

This feature is pretty simple. First, you're required to setup your agent (Windows or Linux) to accept outside connections on a specific UDP port, from a specific IP address (or 0.0.0.0 for anyone). Under Windows, you can also define other possible things the agent can execute as a result of a remote command. Under UNIX, the only supported operation (at this time) is "REFRESH AGENT". Klicking on it is going to result on an immediate agent execution which skips its interval.

This is an example of the UDP server settings under the 3.2 Unix software agent:

udp_server 1
udp_server_port 41122
udp_server_auth_address 0.0.0.0

You may enable the server by setting the value of '1' and disable it with '0' under the 'udp_server' option. Please set '0.0.0.0' as source IP address to enable any IP.

This is an example of the UDP server settings under the 3.x Windows software agent:

udp_server 1
udp_server_port 41122
udp_server_auth_address 192.168.1.23

As you can see, it's exactly the same. In this case, we're going to use the IP address '192.168.1.23' as the only authorized one to refresh this agent.

The Pandora FMS Server has a small script which sends the order to the agent. In our default command, its fully operational and ready to be used. This script is written in Perl which acts as a small client to communicate with the simple UDP server, embedded in the agent and sends commands passed to the command line.






To limit the possible values on a field, is possible to define a list of value/tag. If this list is defined, the field will be a selection combination.

The format will be the following:

value1,tag1;value2,tag2;value3,tag3





We also provide a default alert template to assign "manual" alerts to an agent which means an alert will never be fired automatically. You're required to use the 'manual alerts' to force a manual alert execution by using the circle-shaped button on the agent's main view.






We have created a default alert action called "Restart Agent", which is going to call up the remote agent's command. This action attaches the REFRESH AGENT command to the command, and utilizes the main IP address of the agent to reach, using the default port for the UDP server (41122):






Please follow these steps to enable the Software Agent's remote refresh option:

1. In the configuration file, set up the options for the software agent (UNIX or Windows). Please be mindful on the authorized IP address (is the Pandora FMS server behind a NAT ?), or just put '0.0.0.0' in the field to allow any IP address to force a refresh of the agent.

2. Restart the software agent.

3. You need to setup the IP address on your agent item in the Pandora FMS console. The alert action will utilize the IP address to connect to the running software agent on the remote system.

4. Set an alert to any of the modules of that agent (no matter which), using this screenshot as a example guide:






5. You're now ready to force a refresh for that agent using the main view, clicking on the green circle-shaped button on the left of the alert you've just defined:





If you want to instantly get the agent's information without having to wait for the agent interval, just click in the button and wait a few seconds. The agent will be contacted and forced to execute, the XML file are going to be transferred to your Pandora FMS server and processed, depending on your system load. It will be processed in approx. 1-5 seconds and displayed on the console.

Using Software Agent Plugins

Agent plugins are executed by the software agent and are able to report several information (modules) at once. Each plugin works in a different way and you should test how it works before using it. Default installation of Pandora FMS comes with a bunch of plugins. Of course, UNIX and Windows agents come with different plugins and some of them work very similar.

On Windows Systems

Under the 3.2 version, the windows agents come with the following plugins:

  • df.vbs: Reports the available harddrive space in bytes. It reports different modules for each harddisk that is found on the system. If you want to report the data of specific units only, just use the parameters after the 'module_plugin' call.
  • df_percent.vbs: Very similar to previous the one, but it's going to report the available space in percent. It's going to generate modules with names like 'DiskFree_C'.
  • logevent_log4x.vbs: Reads event-log entries and generates 'log4x' data.
  • ps.vbs: It requires process names and check if these processes are running. If you e.g. execute it with "iexplorer.exe mucommand.exe other.exe" it's going to check for three processes, returns different modules for each of them and reports if the proccesses are down or not.

Under Windows, all default plugins are coded in VBScript. You're going to need to use the correct interpreter for the VBScript Console (sometimes referred to as 'Windows Scripting Host') to run it.

These are some examples for the usage of the previous plugins:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\logevent_log4x.vbs" Aplicacion System 300

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" iexplore.exe myapp.exe

On Unix Systems

Under version 3.2, the generic UNIX agent comes with following plugins:

  • files_indir: This plugin receives a target directory, e.g. '/tmp' and returns two modules: One called 'FS_/tmp/' (boolean) which returns a value of '1' if it contains the same number of files as in the previous execution. The other module called 'NumFiles_FS_/tmp/' returns the number of files in that particular directory.
  • grep_log: It's a generic log parser and it takes tree arguments: <file_to_process>, <module_name> and <reg_exp>. It will generate information inside an 'async_string' module type called <module_name>, using all data which match the <reg_exp> regular expression. For more information on this plugin, please look at the example below.
  • pandora_df: It's very similar to the Windows plugin. It's going to report available space on all mounted partitions on the system. It also takes information from the NFS mounts. The information for all filesystems is returned, but one or more filesystems may be specified as plugin parameters by default.

These plugins are very similar to the Windows plugins. We're not required to use the full path to the plugins, because the 'module_plugin' directive looks for the plugin directory under the agent's home directory. We use the following syntax to execute them:

module_plugin grep_log /var/log/syslog Syslog .

module_plugin pandora_df tmpfs /dev/sda1

And some special plugins working under UNIX:

  • nagios_plugin_wrapper: This is not really a plugin, it's just a wrapper, used to execute a Nagios plugin and to process the output to generate a Pandora module. It grabs the standard output, puts the results into the module description and gets the errorlevel to process a 'module_proc' (boolean) module by its results. Just call up the nagios plugin as a parameter to 'nagios_plugin_wrapper' with all its needed parameters and it's going to generate a Pandora FMS module.
  • inventory: This is used in the inventory system. It will create an inventory XML with information about the system. It could be modified to gather different information, but by default, it only works under Linux and gets packages and services in the default runlevel and some other options. Please check out the inventory documentation for more information.
  • pandora_update: This is used to utilize the autoupdate feature on the software agents. Please check out the agent configuration section for more information.

Examples using Plugins

The plugins for software agents are able to return one type of data or a group of them. An example of a plugin which only returns one data type could be the plugin 'ps.vbs' for Windows. We're going to execute the plugin by the following line:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" IEXPLORE.EXE

The result will be a module which returns a value of '0' if the process is down and '1' if the process is running, e.g.:

<module>
    <name><![CDATA[IEXPLORE.EXE]]></name>
    <description><![CDATA[Process IEXPLORE.EXE status]]></description>
    <data><![CDATA[1]]></data>
</module>

An example of a plugin which returns several data could be the plugin 'df.vbs' for Windows. The line to execute the plugin could e.g. be:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

The plugin returns a module per found harrdrive. The result would be something like this:

<module>
    <name><![CDATA[C:]]></name>
    <description><![CDATA[Drive C: free space in MB]]></description>
    <data><![CDATA[8050]]></data>
</module>

<module>
    <name><![CDATA[D:]]></name>
    <description><![CDATA[Drive D: free space in MB]]></description>
    <data><![CDATA[900]]></data>
</module>

Local Plugins Editor

Since version 5 of Pandora FMS, it's possible to manage the software agent's plugins from the console without the requirement of editing the configuration file directly.

If an agent has an enabled remote configuration, it's going to have the plugins editor tab in the 'Manage' view, as you can see in the screenshot below:



image:plugin_editor_tab.png

The editor shows the plugins list and allows to delete, to add and to disable it. Disabling it under the plugins policy can be useful, because the plugins are still disabled if the policy is going to be applied again.





The managed plugins from this editor can be edited from the agent configuration file, too:





How to Create your own Agent Plugins

The plugins can be developed in any programming language, but they have to respect the following two restrictions:

  • Whatever you want to do: It has to be completely automated (no interactive processing from the user) and must be done from the command line (shell). You're allowed to use any kind of scripting or compiled language, but you have to provide a stand-alone executable with all it's dependencies (libraries, dll, etc.) in this case.
  • Plugin has to report the information to the standard output (just using 'echo', 'printf' or the equivalent in your language) and to use the XML syntax for the Pandora FMS agent information. This is an example of 'generic_data' (numerical information) in XML:
<module>
<name><![CDATA[Sample_Module]]></name>
<type><![CDATA[generic_data]]></type>
<data><![CDATA[47]]></data>
<description><![CDATA[47]]></description>
</module>

The <![CDATA[xxx]]> is used to 'encapsulate' data and to protect the XML files from non-valid characters like ',', '&' or '%'.

Please take a look at our Pandora FMS plugin library at http://pandorafms.org before trying to create your own plugin. If you want to create your own, please upload it to the Pandora FMS public library to allow others use your plugin when it's done.

Using Nagios Plugins from the Agent

Nagios has a lot of amazing plugins you're free to use in conjunction with Pandora FMS. One way is to use remote plugins with the Plugin Server, using the Nagios compatibility, but they only receive it's status. They're not going to use the descriptive output that some Nagios plugins have.

Using the wrapper to utilize Nagios plugins in the software agent will solve this problem. The wrapper comes with 3.2 Unix agent by default. An equivalent plugin for Pandora FMS Windows agents can be downloaded from our website at http://pandorafms.org resource library at [1]).

What does the plugin wrapper for nagios plugins do ?

It executes the nagios plugin, utilizes it's native parameters and converts the output into useful values for Pandora FMS. It has two kinds of information:

  • Status information: 'NORMAL' ('1'), 'CRITICAL' ('0'), 'WARNING' ('2'), 'UNKNOWN' ('3') and 'OTHER' ('4'). It's going to use a 'proc' module by default, so the values 'NORMAL' and 'CRITICAL' are working by default. If you want to have information on 'WARNING' and 'OTHER' values, you're required to setup the module thresholds manually.
  • Descriptive information: Usually string information. It's going to be put on the description field on the module. This is usually something like "OK: successfully logged in" or similar.

Example

You have a pop3 plugin (in '/tmp/check_pop3_login') with execution permissions, which checks if the pop3 account is working. By connecting to a remote host, send a user and password and see if everything is ok. This is a command line example:

/tmp/check_pop3_login  mail.artica.es sanler@artica.es mypass

It's going to return something like this:

OK: successfully logged in.

If it's -not- ok, will return something like this:

Critical: unable to log in.

Using the wrapper is simple. Before the call up, you're just required to put in the wrapper and the module name you want:

/etc/pandora/plugins/nagios_plugin_wrapper sancho_test /tmp/check_pop3_login  mail.artica.es sanler@artica.es mypass

It's going to generate a full XML for the agent plugin:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
<data>0</data>
<description><![CDATA[Critical: unable to log on]]></description>
</module>

Or:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
<data>1</data>
<description><![CDATA[OK: successfully logged in.]]></description>
</module>

The full entry in the 'pandora_agent.conf' will be something like this:

module_plugin nagios_plugin_wrapper POP3_artica.es /tmp/check_pop3_login mail.artica.es sanler@artica.es mypass

In the module, this will be seen like this on a 'fail' event:


file:Sample_plugin_wrapper.png

Monitoring with KeepAlive

There is a special module. It has a unique kind called 'keep_alive' and is very useful to receive any information if there is no contact to the agent. It comes in handy if an agent has stopped to send its information and notifies us.

If there is a module (local or remote) which gets information from the agent, the date of the last 'contact' is updated in a way there is always data. The agent is going to have its date of last contact updated. This can come in handy if the agent 'doesn't answer'. Any agent is considered 'dead' if it hasn't managed to get its data updated in the double of time of its interval, to be precise. If it has e.g. a 5 minutes interval and more than 10 minutes have been past since his last update, the agent is considered 'dead'.

In this moment, the KeepAlive module would appear. It would get fired and appears as a 'Critical' state on the monitor.

The configuration of this kind of modules is very easy. You're just required to create a new module kind called 'dataserver' e.g. in the following way:





Once created (if the agent has data in its interval) it will always be in 'NORMAL' (green) status.





If the agent stops sending data (in this example it has interval of 1 minute), it will pop up automatically and on 'CRITICAL' status (red):





Important: If we have a remote module, e.g. one ping apart from the data reported by the agent, the KeepAlive module will never pop up. We're continually updating the agent by the ping.

Furthermore, the KeepAlive module works the same as any other. It could be associated to an alert and used for lots of other elements (reports, maps, etc.).

Monitoring Command Snapshots

From the 4.0.3 version and above, this way to monitor allows administrators to use a special way to capture the output from commands which differs from the parsing of a single value or string. This module stores the information as text, but with the purpose of getting the exact output of the command, not as single data. It's going to show the same output format and contents like the one returned by the command.

An image always works better than words, so:



This is the 'netstat -an' command output, captured by Pandora FMS after clicking on the special icon for the command snapshot. It's only available if we have a multiline text and a non-split output.



These are the different "snapshots" across a certain time. Pandora is only going to show information if the agent's information is different (data type continues to be of the 'generic_data_string' type here). It's -not- recommended, but if you want to store each single data string, you may use 'async_string'. The behaviour of showing information only when changes occur is perfect for reporting data on problems and to compare other events to the information contained in the command snapshot pertaining that period.

To capture the command outputs in a snapshot, you're required to write a small plugin which sends all data with a 'module_plugin' syntax (including the XML tags) and executes the plugin as a standard plugin. This is an example which generates output for the 'netstat' command:

#!/bin/bash
echo "<module>"
echo "<name>netstat</name>"
echo "<type>generic_data_string</type>"
echo "<data><![CDATA["
netstat -an | grep LIST
echo "]]></data>"
echo "</module>"

Please save this script contents in a file on the agent (or remotely distribute it along with file collections), execute it and use the syntax below:

module_plugin <full path for file>

The agent is going to generate a command snapshot for almost any command. Just replace 'netstat' with your command. Some useful suggestions for UNIX systems are e.g.:

* top -b -n 1
* ps aux
* vmstat 1 5
* who 
* last -10

On Windows systems:

* tasklist
* netstat -an
* net start

Image:Info.png

Remember that you're always required to do this within a script which is generating the XML file. If you do that by calling up a 'module_exec', each line reported by the command will be interpreted as a line with different 'data blocks' - and you cannot see it as a command snapshot anymore.

 


Go back to Pandora FMS documentation index