Pandora: Documentation en: Operations

From Pandora FMS Wiki
Jump to: navigation, search

Go back Pandora FMS 3.0 documentation index

Contents

1 Monitoring with Software Agents

1.1 Name of the Agents

From Pandora FMS version 7 onwards, the agents have an alias and a name (or unique identifier). An agent configured by default will generate a name (or identifier) based on a pseudo-random hexadecimal string, and an alias (or visible name) based on the hostname of the machine.

In previous versions there was only the "name" of the machine, and the previous system is totally compatible with the most modern versions of Pandora FMS, only that if in the same Pandora FMS installation there are two agents with the same identifier (or names) the data of both agents will be mixed and/or will be overwritten. That's why we introduced from version 7 onwards the possibility that agents with different names could have the same alias.

The following configuration tokens are used to change this behavior:

pandora_agent
pandora_alias

By default, the configuration file does not use either one of them, so it gets the hostname of the machine as an alias and a very large random hexadecimal number as a name or identifier. The agent name is no longer visible (except in the agent detail view) and CANNOT be changed. The Agent Alias can be changed at any time, without having to worry about the configuration of the software agent, since the agent's unique identifier is the agent's "name".


1.2 Agent Configuration

All the configuration and monitoring parameters of the software agents can be found in their configuration file pandora_agent.conf. This is stored locally in the machine where the software agent is installed, so any modification we want to make in the agent must be reflected in this file. You have a detailed description of all agent configuration tokens in the chapter "PandoraFMS Agent Configuration" [1] while here we will only focus on the advanced uses of some of them.

1.2.1 Remote Configuration

On the Enterprise version, there is a remote Agent Configuration feature which allows centralized configuration and file management from the server console. This allows centralized management of all our software agents without the need to physically access the systems where they are installed.

The configuration consists of two files. Their file names are <md5>.conf and <md5>.md5, where <md5> is the agent's name hash code. Those files are stored in '/var/spool/pandora/data_in/conf' and '/var/spool/pandora/data_in/md5' folders respectively.

These files stored in the server are modified by Pandora FMS console when the agent configuration is edited remotely.


Sw-agent.png


To enable the remote configuration, you only need to enable the corresponding parameter in the agent's local configuration file first. From this moment on, all the changes must be done from Pandora FMS console:

remote_config 1

Info.png

Once the agent's remote configuration is enabled, any changes made locally in the configuration file will be overwritten by the configuration stored in the console. If you want to prevent this from happening, stop the agent, modify the configuration file, disable the remote configuration and launch the agent again.

 


1.3 Common Configuration Parameters

In the Pandora FMS Software Agents section you can find a complete explanation on Agent Configuration. In this section, the common parameters used to configure the Software Agents are going to be explained.

The most common parameters are:

  • server_ip: IP address of Pandora FMS Server.
  • server_path: Path of the 'incoming' folder for the Pandora FMS server (it's '/var/spool/pandora/data_in' by default).
  • temporal: Software agent's temporal folder (it's '/var/spool/pandora/data_out' by default).
  • logfile: Software agent's log file (it's '/var/log/pandora/pandora_agent.log' by default).
  • interval: Agent's execution interval (it's '300' by default).
  • intensive_interval: Intensive module execution interval (it's '300' by default).
  • debug: Debug mode enable ('0' - it's disabled by default).
  • agent_name: Agent name (hostname is taken by default).
  • remote_config: Activation of remote configuration ('0' - it's disabled by default).

An example of the general parameters for a UNIX configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        /var/spool/pandora/data_out 
logfile         /var/log/pandora/pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1 

An example of the general parameters for a Windows configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        c:\program files\pandora_agent\temp
logfile         c:\program files\pandora_agent\pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1

1.4 Custom Fields

Custom fields are an easy way to personalize agent information. Create custom fields by clicking on 'Resources' -> 'Custom fields'.


Custom fields1.png



Custom fields2.png


  • Name: Name of the custom field.
  • Display on front: With this field checked, the custom field will be displayed in front of the agent like on the screenshot below:


Customfields.JPG


These custom fields can also be passed from the agent configuration file, using the following configuration token:

custom_field1_name Model
custom_field1_value i386

This example allows you to use a custom agent field defined in the agent's. conf.

1.5 Monitoring with the Software Agent

Software agents are running on the systems from which they collect information. Each of the checks they make on the system, such as CPU usage, free memory or disk space, corresponds to a module. Therefore, each of these modules collects a single data in each execution.

There are also some agent-specific directives that serve to collect certain data directly from the operating system (e.g. CPU usage, memory, events, etc.). With these agent-specific directives, you don't need to execute commands. Please check the Software Agents Installation Section to obtain more information about them.


Agent-monitoring.png


Pandora FMS software agents use the commands of the operating system in which they are installed to obtain the information for each one of their modules. The Pandora FMS data server processes and stores in the database all the information generated by the software agents, which send you their data in an XML file.


Esquema-3.png
Logical schema of an agent/physical agent


In the agent configuration file, the modules are defined by the following structure:

module_begin 
module_name cpu_user
module_type generic_data
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_description User CPU Usage (%)
module_end

A real case can be:

module_begin 
module_name Files data_in
module_type generic_data
module_exec ls /var/spool/pandora/data_in | wc -l
module_description Number of files incoming dir
module_end

The line module_exec contains the command that will be executed to collect the information. The value returned by that execution will be the data obtained by the module and will be shown in the monitoring. Another command to get information using module_exec would be:

module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'

To collect the information the agent will execute the command in the shell as if it was done by an operator, so it is always advisable to manually launch the command and analyze the output:

$> vmstat 1 2 | tail -1 | awk '{ print $13 }'

The value returned by the execution will be stored in XML as module data. You can indicate anything in the line module_exec as long as the output is compatible with the values accepted by Pandora (numerical, alphanumeric or boolean), so it is possible to indicate custom scripts:

module_exec myScript.pl --h 127.0.0.1 -v cpu

Again the agent is going to execute the command and collects the returned value in the same way as an operator would type in the shell:

$> myScript.pl --h 127.0.0.1 -v cpu

When the software agent is executed for the first time, it sends an XML to the Pandora FMS which creates the agent automatically with all its modules.

1.5.1 Types of Modules

The following are the types of modules possible in software agents depending on the type of data returned:

  • generic_data: Numerical and floating point data.
  • generic_data_inc: A kind of increasing numerical data. Stores the difference between the previous and current data divided by the elapsed time in seconds, showing the rate per second. This type of data is used to count "number of times per second" of something, such as log entries/sec, receivedbytes/sec , incoming connections/sec , etc.
  • generic_data_inc_abs: Type of absolute increasing numerical data. It stores the difference between the previous and current data, without dividing it between the elapsed seconds, so the value will correspond to the total increase between the two executions, and not to the increase per second. This type of data is used to count the number of times something happens, such as log entries, total received bytes, number of incoming connections, and so on.
  • generic_proc: Boolean type of data, where a value of 0 means False or incorrect, and values above zero mean True or correct. The generic_proc types have the critical (0) and correct (1 or higher) states preconfigured.
  • generic_data_string: Kinds of alphanumeric data (text).
  • async_data: It's a kind of asynchronous numeric data. It's the same as 'generic_data' but for asynchronous data which is only updated if there is a change. The asynchronous kind of data don't have a defined periodicity when we can obtain data.
  • async_string: This is a kind of asynchronous alphanumeric data. It's the same as 'generic_string' but for asynchronous data which are only updated if there is a change. It's the kind of data that you're recommended to use if you want to monitor searches in logs or event viewers. We could have new data by any second or not having one at all in many days.
  • async_proc: It's a kind of asynchronous boolean data. It's the same as 'generic _proc' but for asynchronous data which are only updated if there is a change.


  • Image module: They are based on a text string type module (generic_data_string or async_string). If the data in the module is a base64 image, in other words, part of the string contains "data:image", it will be identified as an image and, on the views that it appears, it will enable a link to open a window to display the image. Also on its historical data the strings that build/generate the images will be saved and displayed.

The software agent already comes preconfigured to send certain data from the system on which it's installed. These usually are (depending on the version):

  • System CPU
  • Available space on the hard-drive
  • Free memory
  • Monitor of the programs and services states

1.5.2 Intervals in local modules

The local modules (or software agent modules) all have the interval of their agent as a "base". In other words, if an agent has an interval of 5 minutes (300 seconds), all modules will default to an interval of 5 minutes. You can make a module have a GREATER interval, but you can never make it less than the agent's base interval, since the interval of a module is defined as a multiplier of the agent's interval.

If an agent has an interval of 300 and a module of that agent has the following configuration:

module_interval 2

The interval module will be 300x2. The module_interval parameter only supports whole numbers greater than zero.

1.5.3 Module Creation Interface

(Enterprise only)

In the Enterprise version it is possible to create and manage local modules of the software agents if they have the parameter remote_config 1. If you do not have the Enterprise version, all these operations must be done directly on the configuration file of the software agent, locally in the system where the agent is installed.

The creation of local modules from the console is done using a text box form. This is the location to specify the configuration data, which will be placed into the configuration file of the software agent (besides the common configuration with the remote modules like thresholds, type and group).

Creation of a module with the remote config enabled on the agent:

Local module editor.png

In this text box there are two buttons, one to create a basic configuration structure and the other to check that the data is correct. This check is intended for basic parameters, e.g. checking if it begins with 'module_begin', ends with 'module_end' and if it has a valid type and name. Other parameters may be wrong, but that won't be detected here.

The field name and the type combo are linked to the parameters 'module_name' and 'module_type' of the data configuration. If you change the module name on the name field, the configuration data name will be changed automatically and vice versa. If you select a type in the combo, the data configuration type will be changed and if a correct type was written in the configuration data, this type will be selected automatically in the combo.

When a module from a local component is changed, it might have macros. If it has macros, the configuration data will be hidden and a field for each macro is going to appear instead. You can find a detailed explanation of this feature in the following section:

Templates and components

1.5.4 Conditional Monitoring

1.5.4.1 Post-Conditions

Pandora FMS software agent supports the execution of commands and scripts as post-conditions. This means that actions could be performed depending on the value obtained in the execution of the module.

With the parameter module_condition we will indicate a value or range of values and the execution to be carried out in case the module is between these values:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

You can define multiple postconditions for the same module, e.g.:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition (90, 100) remove_processes.sh
module_condition < 20 add_processes.sh
module_end

Some examples:

Execution if the module data is less than '20' :

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

If the script named 'get_cpu_usage.pl' returns '18', the software agent is going to execute the script 'add_processes.sh', otherwise it won't.

Execution with two postconditions:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 10 start_new_server.sh
module_condition < 20 add_processes.sh
module_end

If the module returns '15', the software agent is going to only execute the script named 'add_processes.sh' - but if the value is '6', the script is going to execute both scripts named 'start_new_server.sh' and 'add_processes.sh'.

1.5.4.2 Pre-Conditions

The parameter module_precondition defines a precondition to evaluate before a module execution. Depending on the result of this precondition, the software agent will execute the module or not. The structure of the configuration file is:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

You can define multiple preconditions for the same module:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

In the example above we have two preconditions. For the module to be executed all preconditions must be met, in this case two, so the module will only be executed when the number_active_processes.sh script returns a number greater than 10 and the important_service_enabled.sh script returns 1.

Module execution if the precondition is above '10' only:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

In the example above, the software agent executes the script 'number_active_processes.sh', if the returned value is greater than '10'. If the returned value is lower than '10', the module won't be executed.

1.5.5 Intensive Monitoring

There are certain modules that are of special importance, such as critical running processes or services. Intensive monitoring is available to enable more controlled monitoring of these cases.

It consists of warning in a shorter interval that a problem has appeared without reducing the agent's general interval.

The software agent presents these two configuration parameters:

  • "Interval": agent sampling time in seconds. This is the general range for all local modules. Required parameter.
  • Intensive_interval: time in which we will be notified of a problem in the especially critical modules. Optional parameter.

At module level, the parameter module_intensive_condition will be used to determine under which condition the status of the module will be notified in the time defined by the intensive_interval.

  • module_intensive_condition = 0: if the module obtains as a result the value indicated in this parameter (in this case 0), it will be notified in the intensive interval defined in the agent.

If you want to check whether the SSHD system service is running every 10 seconds, but want to be notified every 10 minutes if the service is fine, you may configure the agent in the following way:

intensive_interval 10
interval 300
module_begin
module_name SSH Daemon
module_type generic_data
module exec ps aux | grep sshd | grep -v grep | wc -l
module_intensive_condition = 0
module_end

If the service goes down, you will be notified in the next 10 seconds. If the service is up, you will be notified in the next 5 minutes, like normally.

1.5.6 Programmed Monitoring

The software agent supports the definition of programmed modules which are executed in the defined instances. The syntax is the same as crontab. The module structure is the following:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_end

In this example the module is executed every Monday from 12 to 15.

If we're required to execute the module every hour and ten minutes, we can use this module definition:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab 10 * * * *
module_end

Info.png

Note that if you use an interval that causes the module not to report data, this module will go into "unknown" status. Use asynchronous modules for these cases.

 


1.6 Specific Monitoring for Windows

The software agent for Windows has specific features to make monitoring a lot easier. These features are explained with some examples:

1.6.1 Monitoring Processes and Watchdog Process

1.6.1.1 Monitoring of Processes

The parameter module_proc verifies if a process with a preset name is running on this machine. The module definition is:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_description Process Command line
module_end

If the process name has blanks, don't use «" "». The process name is the same as shown in Windows Task Manager (taskmngr) including the .exe extension. It's very important to use the same upper and lower-case letters.

If you want the software agent to immediately notify you if a process is not working add the parameter module_async yes. In this case, the module definition would be:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_async yes
module_description Process Command line
module_end

1.6.1.2 Watchdog Process

The watchdog feature on the Pandora Agent for Windows allows immediate response to the crash of a process and restarts it. It's important to keep in mind that the watchdog only works if the module is of the asynchronous type.

The definition of a module with watchdog enabled would be as follows:

module_begin
module_name Notepad
module_type generic_data
module_proc notepad.exe
module_description Notepad
module_async yes
module_watchdog yes
module_user_session yes
module_start_command c:\windows\notepad.exe
module_startdelay 3000
module_retrydelay 2000
module_retries 5
module_end

In the previous example, the watchdog will activate each time the notepad.exe process is deactivated and the command c: \windows\notepad. exe will be executed. In addition, if we look at the other watchdog configuration parameters, we will see that the process reactivation will be attempted 5 times with an initial wait time of 3 seconds and a wait time between retries of 2 seconds. In this example, the notepad.exe process will be launched in the user's session.

1.6.2 Service Monitoring and Service Watchdog

1.6.2.1 Service Monitoring

The parameter module_service verifies if a specified service is running on the machine. The definition of this module is as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_end

If the service name has blanks, don't use «" "». To find the service name, you can look at the Service Name field under the Windows Service Manager. It's very important to use the same upper and lower-case letters.

If you want the software agent to warn you immediately when a service is down, add the parameter module_async yes. The module definition would be as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_async yes
module_end

1.6.2.2 Service Watchdog

There is a watchdog mode for services which allows you to detect and restart a downed service almost in real time. A module definition example using watchdog would be the following:

module_begin
module_name ServiceSched
module_type generic_proc
module_service Schedule
module_description Service Task scheduler
module_async yes
module_watchdog yes
module_end

The watchdog definition for services has no need for any extra parameters because they are incorporated in the service definition.

1.6.3 Monitoring Basic Resources

This section describes how to monitor the basic variables of a Windows-based machine.

1.6.3.1 Monitoring the CPU

To monitor the CPU, you may use the parameter module_cpuusage which returns the CPU usage percentage.

Its possible to monitor the CPU based on its ID with a module definition like the following:

module_begin
module_name CPU_1
module_type generic_data
module_cpuusage 1
module_description CPU usage for CPU 1
module_end

You're also able to monitor the average CPU usage from all systems with the following module:

module_begin
module_name CPU Usage
module_type generic_data
module_cpuusage all
module_description CPU Usage for all system
module_end

1.6.3.2 Memory Monitoring

To monitor the memory, you can use two parameters: module_freememory which returns the amount of free memory in the system and module_freepercentmemory which returns the percentage of free memory.

An example for a module using the module_freememory parameter would be:

module_begin
module_name FreeMemory
module_type generic_data
module_freememory
module_description Non-used memory on system
module_end

An example for a module using the module_freepercentmemory parameter would be:

module_begin
module_name FreePercentMemory
module_type generic_data
module_freepercentmemory
module_end

1.6.3.3 Monitoring Hard drives

To monitor the hard drive space, you may use two parameters: module_freedisk which returns the amount of available space and module_freepercentdisk which returns the percentage of available space. Both parameters require the monitored unit as an input. Please don't forget the character « : ».

A module which utilizes the module_freedisk parameter is defined in this way:

module_begin
module_name FreeDisk
module_type generic_data
module_freedisk C:
module_end

An module which utilizes the module_freepercentdisk parameter is defined in this way:

module_begin
module_name FreePercentDisk
module_type generic_data
module_freepercentdisk C:
module_end

1.6.4 WMI Queries

The Pandora FMS Software Agent allows you to extract information by using WMI queries and ODBC connections - very common technologies used to store external or system-related information.

The software agent allows you to execute any local WMI query you want using the module_wmiquery parameter. To perform the query, set the parameter module_wmiquery by the query which is going to be performed and to set the parameter module_wmicolumn by the column which has the information to monitor.

We're able to get a list with the installed services:

module_begin
module_name Services
module_type generic_data_string
module_wmiquery Select Name from Win32_Service
module_wmicolumn Name
module_end

We're also able to get the CPU load using WMI:

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

1.7 Remote Checks with Software Agents

A remote check performed by the agent makes it easy to monitor complex networks that have special, security-related requirements.

This way of working is usually used when you want to launch remote checks on systems that the main Pandora FMS server does not have access to, for which we can install a software agent, run remote checks from that point and distribute them in broker agents.

This section explains how to use this feature of the software agent

1.7.1 ICMP Checks

ICMP or ping checks are very useful to know if a machine is connected to a network or not. In this way, a single software agent could easily monitor the status of all machines.

UNIX

By using the UNIX software agent, you're able to utilize the system commands to create a module which performs the ping check. An example module definition would be:

module_begin
module_name Ping
module_type generic_proc
module_exec ping -c 1 192.168.100.54 >/dev/null 2>&1; if [ $? -eq 0 ]; then echo 1; else echo 0; fi
module_end

In this example module, we're going to perform a ping check on the host '192.168.100.54'. If you want to check a different host, all you have to do is to change the IP.

Windows

The software agent for Windows platforms has the following parameters to configure the ping checks:

  • module_ping_count x: Number of ECHO_REQUEST packages to send (Default value is '1').
  • module_ping_timeout x: Timeout in seconds (Default value is '1').
  • module_advanced_options: Advanced options for 'ping.exe'.

A module configuration example could be:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 5
module_end

In this example, we're going to perform the same check as in the previous one, but now we're going to use the Software Agent for Windows platforms.

1.7.2 TCP Checks

TCP checks are useful to verify if a port of a host happens to be open or not. Interesting in case you want to know if an application is connected to the network or not.

UNIX

We're able to perform the TCP checks by the following module by utilizing the software agent for UNIX platforms:

module_begin
module_name PortOpen
module_type generic_proc
module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_timeout 5
module_end

With this module, we're going to check if port 80 of the host '192.168.100.54' is open or not.


Windows

If we want to use the software agent for Windows, we have some parameters to configure for the TCP check. The parameters are:

  • module_tcpcheck: Host to be checked
  • module_port: Port to be checked
  • module_timeout: Timeout for the check

A module definition example is this:

module_begin
module_name TcpCheck
module_type generic_proc
module_tcpcheck 192.168.100.54
module_port 80
module_timeout 5
module_end

This module is the equivalent for the Windows software agent to perform the check on port 80 of the host '192.168.100.54'.

1.7.3 SNMP Checks

SNMP checks are commonly used to monitor network devices to check the interface status, inbound/outbound bytes, etc.

UNIX

If you are using the software agent for UNIX platforms, you may create the module using the command 'snmpget' like this:

module_begin
module_name SNMP get
module_type generic_data
module_exec snmpget 192.168.100.54 -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}'
module_end

This module returns the value for OID .1.3.6.1.2.1.2.2.1.1.148 on the host '192.168.100.54'.

Windows

For the Windows software agent, we have the following configuration parameters for the module:

  • module_snmpversion [1,2c,3]: SNMP version (Default value is '1').
  • module_snmp_community <community>: SNMP community (Default value is 'public').
  • module_snmp_agent <host>: The host to monitor.
  • module_snmp_oid <oid>: OID.
  • module_advanced_options: Advanced options for 'snmpget.exe'.

A module example could be:

module_begin
module_name SNMP get
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148
module_end

Using the UNIX software agents, this module is the Windows platform equivalent to do the last performed check.

1.8 Proxy Mode

Template warning.png

To use the agent's proxy mode on Linux or UNIX systems, the agent must -not- be executed by a root user ! You're required to perform a custom installation of the Pandora FMS agent to do so. You may look up all the details about custom installations in the section Custom Agent Installation.

 


Pandora FMS Software Agents have a Proxy Mode which allows them to act as proxies, redirecting the communication of several agents to the Pandora FMS Server. The software agent with an enabled proxy mode is able to perform monitoring tasks, too.


Proxy-mode.png


The Proxy Mode is very useful if you're dealing with a network in which only one machine can communicate with the Pandora FMS Server. In this situation, all software agents installed on machines without access to the Pandora FMS Server are going to send the XML files to the agent currently working as a proxy - which is going to redirect all files to Pandora FMS Server.

In addition to XML file redirection, the proxy mode supports Remote Configuration and File Collection features.

With all these features the Proxy Mode offers a transparent operation of software agents in networks with limited connectivity.

To enable the Proxy Mode in a software agent, you're required to configure the following parameters:

  • server_ip: IP of the Pandora FMS Server. If the proxy mode is enabled, the IP is not allowed to take the following values: '127.0.0.1', 'localhost' or '0.0.0.0'.
  • proxy_mode: If it's set to '1', the proxy mode is enabled. The default value is '0' (disabled).
  • proxy_max_connection: Maximum number of connections for the proxy. The default value is '10'.
  • proxy_timeout: Proxy timemout. The default value is '1' (in seconds).

A configuration example:

server_ip 192.168.100.230
proxy_mode 1
proxy_max_connection 20
proxy_timeout 3

To redirect the connection of a software agent we will only have to put as Pandora's server address the one of the agent with the Proxy Mode activated. For example:

Our proxy mode enabled agent has the IP of 192.168.100.24

In the software agent which can't directly connect to the Pandora FMS Server, we configure the parameter server_ip in the following way:

server_ip 192.168.100.24

With this configuration the software agent with limited communication will use the software agent in Proxy Mode to communicate with Pandora's server, keeping all its functionalities such as remote configuration, policies or file collections.

1.9 Broker Mode

The software agent has a Broker Mode which allows one agent to monitor and manage the configuration as if there were several software agents installed:


Modo-broker.png


When the broker mode is activated in a software agent, a new configuration file is created. From that moment on, the original software agent and the new broker will be managed separately with their independent configuration files, as if they were two completely separate software agents on the same machine.

The main features of the Broker Mode are:

  • To send local data as another agent. Very useful to monitor different software instances as different agents.
  • To send the collected data from the remote checks to other machines as if a software agent had been installed on them.

To create a broker, you're only required to add a line with the parameter broker_agent <broker_name>. It is possible to create as many broker agents as we want, just by adding the corresponding broker_agent lines, as follows:

broker_agent dev_1
broker_agent dev_2

Once the brokers are created, the configuration files 'dev_1.conf' and 'dev_2.conf' are going to be created with the same content as in the original software agent, but with a different agent name. By adding or deleting modules from configuration files 'dev_1.conf' and 'dev_2.conf', we can customize the checks performed by the brokers.

On the Pandora FMS web console the brokers appear and will be managed like other agents, which means that if we have a software agent installed with two brokers, we're going to see three different agents with their modules, configurations, etc. on the web console.

NOTE: broker agent instances cannot use file collections. If you want to use collections, you can "execute" files from collections from an instance, but it must be distributed by the "main" agent, not in an instance.

NOTE: modules that keep data in memory between executions (module_logevent and module_regexp on Windows) will not work when broker agents are enabled.

1.9.1 Examples of Use

1.9.1.1 Monitor a local Database as a different Agent

We want to monitor the basic parameters of a machine (CPU, memory and hard drive) and an installed database that we want to monitor separately.

To perform this monitoring, we are going to use the following structure:

  • Installed Software Agent: monitoring CPU, memory and disk.
  • Broker for the Database: monitoring internal status of the database.

We're going to install the software agent on the machine to monitor the CPU, memory and hard drive. We're also adding the following line in the software agent configuration:

broker_agent DBApp

We're going to create a broker agent called 'DBApp' by adding this line, so a configuration file named 'dbapp.conf' will appear. We're adding the modules to perform the checks for the database in this configuration file:

module_begin
module_name Num Users
module_type generic_data
module_exec get_db_users.pl
module_end

module_begin
module_name Num slows queries
module_type generic_data
module_exec get_db_slows_queries.pl
module_end

By doing this, you're going to see two agents in the Pandora web console: One bearing the name of the machine with the modules 'CPU', 'Memory' and hard drive and another one called 'DBApp' with the modules 'Num Users' and 'Num slows queries'.

1.9.1.2 Remotely Monitor Devices using Brokers

For this example, we've installed a software agent on a Windows machine, monitoring CPU, memory and hard drive. We're also required to monitor a router with the IP '192.168.100.54' without being allowed to install an agent on it. To solve the problem, we can use brokers.

We're going to create a broker using the following parameter:

broker_agent routerFloor5

By adding this line, we're going to create a broker agent by the name of 'routerFloor5'. The software agent was installed on a Windows machine, so we're able to monitor the router by using the ping and SNMP modules available for Windows software agents. To do that, we'll modify the file 'routerFloor5.conf' by adding the following lines:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

In this example, the web console of Pandora FMS shows two agents: One is the Windows machine with the modules 'CPU', 'Memory' and 'hard drive' and the other one is 'routerFloor5' bearing the modules named 'Ping', 'Eth 1 up' and 'Eth 2 up'.

1.9.1.3 Remote Monitoring of a Network with limited Communication Options

In some cases, you need to monitor devices remotely where the Pandora FMS Remote Server can't access them directly.


Broker example no access.png


In this example, we have to monitor some devices from one of the company sites from the headquarters remotely. The Pandora FMS Server is connected to the other company sites in the headquarters using a VPN. Due to some restrictions, the Pandora Remote Server can't access the machines directly. To monitor the company sites, we're going to use Broker Mode which allows a software agent to send XML files to the Pandora Server as if there were several different devices.

We add as many brokers as there are devices to be monitored. In the configuration file of the software agent, an example configuration could be:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

Once the brokers are created, we're able to customize the monitoring for each device by modifying the configuration file of each broker. The configuration e.g. for the Windows machine called 'device_1' is:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

With this configuration we are able to make a remote configuration and send the files to Pandora FMS server in spite of the communication restrictions between the company's headquarters.

1.9.1.4 Shared Monitoring Load by Brokers

Broker mode is very useful for sharing and distributing the monitoring load within several network points.


Broker scalation example.png


In this example, our architecture has several networks named from A to Z with a 1000 devices each. The capacity of the Pandora FMS Remote Server is about 2000 agents, because we've decided to use broker mode enabled software agents to share and distribute the load. These broker mode enabled software agents are going to remotely monitor all devices from the network and send the data in XML format to the Pandora FMS Central Server.

For each network, we have a broker mode enabled agent. On it, we're going to create as many brokers as the number of devices to be monitored. An example configuration for the software agent 'Broker_Agent_Net_A' could be the following:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

In addition for each broker, we're going to add the modules to monitor the devices. Example: The broker 'device_1' (which is a router) could have this configuration:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

Another example configuration for the broker device_2 which monitors a Windows machine with the following modules:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

Using broker mode enabled software agents, we're easily able to share the load to collect the data from thousands of devices.

1.10 Inventory using Software Agents

Pandora FMS Software Agents supports inventory features for both hardware and software. The inventory system allows you to get a history of CPU, cards, memory, patches, software, etc, used in the company servers. Furthermore, it's possible to generate alerts if a change occurs in the inventory, e.g. if a disk was replaced or an application was deleted.

For further information on the subject, please have a look at the section Local Inventory by Software Agents.

1.11 UDP remote commands

1.11.1 How to ask an Agent for On-Demand Information

Pandora FMS software agent includes the UDP Server functionality, which allows to remotely indicate actions to an agent, such as restarting or executing a command.

The basic configuration parameters of the UDP Server in the software agents are:

  • udp_server: enables (1) or disables (0) this functionality.
  • udp_server_port: listening port of the UDP server in the software agent.
  • udp_server_auth_address: IP address where the UDP server accepts requests. You can set it to 0.0.0.0.0 to accept from all origins.

Configuration example :

udp_server 1
udp_server_port 41122
udp_server_auth_address 0.0.0.0

Now, to force the restart of the agent we must use the script udp_client. pl, present in the Pandora FMS server, and normally located in /usr/share/pandora_server/util. We can run it from the command line or use it in an alert, making use of the command that comes pre-configured in the "Remote agent control" console.

There is also a default alert action called Restart agent, which uses this script. It uses the action REFRESH AGENT on the script udp_client. pl to restart the agent if it has the UDP server listening.


Agent restart action.png


Please follow these steps to enable the Software Agent's remote refresh option:

1. In the configuration file, set up the options for the software agent (UNIX or Windows). Please be mindful on the authorized IP address (is the Pandora FMS server behind a NAT ?), or just put '0.0.0.0' in the field to allow any IP address to force a refresh of the agent.

2. Restart the software agent for the changes to apply.

3. Associate the Restart agent alert to the module of some agent (it is necessary that this agent has the IP address correctly configured).

4. Force the execution of the alert or force an incorrect state of the module to trigger the alert.

Now thanks to this action, it is possible to manually force the alert at any time to refresh the agent software remotely and get the information quickly.

1.11.2 Custom remote commands

Apart from the Refresh agent command, you can specify new and custom actions for the agents to perform under the Pandora server's orders.

If you're interested, you need to make a slight modification in pandora_server.conf, apart from enabling udp server, as shown before:

udp_server 1
udp_server_port 41122
udp_server_auth_address <server IP>

Then you have to add a line for each custom command you want to perform, following this syntax:

process_nameofthecommand_start <command>

For example, if you wanted to create a custom command to start sshd service, you would have to add a line such as this:

process_sshdproc_start /etc/init.d/sshd start

Then you have to create a new alert action at Pandora Console for each remote command you made. You can copy the "Remote agent control" action, which is already prepared to send udp commands. Set "START PROCESS sshdproc" on Field 1.


Udp process.JPG


Now you only need to set a new Manual alert with the new alert action on the agent whose sshd service you wanted to start. When the alert is forced, the order will be launched and the agent will start the service.

Info.png

Custom orders can also be created to execute scripts. This allows a huge variety of remote actions to be performed on a remote agent just by clicking a button.

 


1.12 Using plugins in software

They are characterized by performing complex advanced checks from the software agents, being able to return several modules as a result instead of a single value. Unlike the server plugins, which are executed by Pandora FMS server, the agent plugins return their data in an XML, reporting one or several modules at the same time.

1.12.1 Execution on Windows systems

In Windows, all the default plugins are programmed in VBScript, to run them you need to use the appropriate interpreter indicating the full path.

Here are some examples of how to use the default plugins included in the Windows agent:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\logevent_log4x.vbs" Application System 300
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" iexplore.exe myapp.exe

El agente Windows trae diversos plugins listos para utilizarse.

1.12.2 Execution on Unix systems

Unix plugins are by default in the directory "/plugin" of the agent directory, in /etc/pandora/plugins, so it would not be necessary to use the full path in its execution if they are in this directory.

Here are some examples of using plugins:

 module_plugin grep_log /var/log/syslog Syslog .
 module_plugin pandora_df tmpfs /dev/sda1

The Unix software agent comes with several plugins by default ready to function.

1.12.3 Examples of use

Plugins for the software agent can return a single data or a group of data. An example of a plugin that returns a data can be the Windows ps. vbs plugin, which simply checks if a process is running. With the next line we run the plugin:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" IEXPLORE.EXE

The result will be a module that returns 0 if the process is not active and 1 if it is active:

<module>
    <name><![CDATA[IEXPLORE.EXE]]></name>
    <description><![CDATA[Process IEXPLORE.EXE status]]></description>
    <![CDATA[1]]>
</module>

An example of a plugin that returns several data can be the Windows plugin df. vbs. The line to run the plugin would be:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

The plugin returns one module for each found disk, the result would be:

<module>
    <name><![CDATA[C:]]></name>
    <description><![CDATA[Drive C: free space in MB]]></description>
    <![CDATA[8050]]>
</module>

<module>
    <name><![CDATA[D:]]></name>
    <description><![CDATA[Drive D: free space in MB]]></description>
    <![CDATA[900]]>
</module>

1.12.4 Management of the agent plugins from the console

In the Enterprise version it is possible to manage the software agent plugins from the console without directly editing the configuration file.

If an agent has the remote configuration activated, the plugin editor tab is available in its administration view.



Plugin editor tab.png

This section shows the list of the active plugins in the agent, and allows you to delete, add or deactivate them. In the case of policy plugins, it may be useful to disable them because they will remain disabled when the policy is re-applied.



Plugin editor.png

The plugins managed in this editor can also be edited from the agent configuration file.



Plugin editor conf.png

1.12.5 How to create your own agent plugins

Plugins can be created in any programming language. They only have to respect these restrictions:

  • Regardless of what we want to do, it must be automatic (without user interaction), and must be done from the shell. You can use any type of scripting language or compiled language, but in that case you must also distribute, in addition to the executable, all libraries (or DLL) that are necessary for the plugin to run.
  • The plugin must return the information through the standard output (simply using echo, printf or the equivalent in the language that will be used for the plugin), and must use the XML format of Pandora FMS agents to return the information.

This is an example of a numerical module in XML:

<module>
<name><![CDATA[Sample_Module]]></name>
<type><![CDATA[generic_data]]></type>
<![CDATA[47]]>
<description><![CDATA[47]]></description>
</module>

The data contained in the <!CDATA[xxx]]> are used to protect XML from certain information that may contain "difficut" characters for XML, such as <, >, & or %.

Before trying to create a plugin, visit the Pandora FMS plugin library at https://library.pandorafms.com, and if you decide to create your own plugin, send it to the public library, so that others can use it.

Template warning.png

Make sure you finish the output of your plugin (if it is a script) with an error level 0 or the agent will interpret that the plugin has had an error and wasn't able to run.

 


1.12.5.1 Shellscript (Linux/Unix) plugin example

#!/bin/bash
# Detect if local Mysql is without password
# First, do we have a running MySQL?
CHECK_MYSQL=`netstat -an | grep LISTEN | grep ":3306 "`
if [ ! -z "$CHECK_MYSQL" ]
then

        CHECK_MYSQL_ROOT=`echo "select 1234" | mysql -u root 2> /dev/null | grep 1234`
        if [ -z "$CHECK_MYSQL_ROOT" ]
        then
        echo "<module>"
        echo "<type>generic_proc</type>"
        echo "<name>mysql_without_pass</name>"
        echo "<data>1</data>"
        echo "<description>MySQL have a password</description>"
        echo "</module>"
        else
        echo "<module>"
        echo "<type>generic_proc</type>"
        echo "<name>mysql_without_pass</name>"
        echo "<data>0</data>"
        echo "<description>MySQL do not have a password</description>"
        echo "</module>"
        fi
fi

exit 0

1.12.5.2 VBScript (Windows) plugin example

' df.vbs
' Returns free space for avaible drives.
' --------------------------------------

Option Explicit
On Error Resume Next

' Variables
Dim objWMIService, objItem, colItems, argc, argv, i

' Parse command line parameters
argc = Wscript.Arguments.Count
Set argv = CreateObject("Scripting.Dictionary")
For i = 0 To argc - 1
    argv.Add Wscript.Arguments(i), i
Next

' Get drive information
Set objWMIService = GetObject ("winmgmts:\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery ("Select * from Win32_LogicalDisk")

For Each objItem in colItems
	If argc = 0 Or argv.Exists(objItem.Name) Then
		If objItem.FreeSpace <> "" Then
			Wscript.StdOut.WriteLine "<module>"
			Wscript.StdOut.WriteLine "    <name><![CDATA[" & objItem.Name & "]]></name>"
			Wscript.StdOut.WriteLine "    <description><![CDATA[Drive " & objItem.Name & " free space in MB]]></description>"
			Wscript.StdOut.WriteLine "    <data><![CDATA[" & Int(objItem.FreeSpace /1048576) & "]]></data>"
			Wscript.StdOut.WriteLine "</module>"
            Wscript.StdOut.flush
		End If
	End If
Next

1.12.6 Using Nagios plugins from the agent

Nagios has a large number of plugins that you can use with Pandora FMS. One way to do this is to use remote plugins with the Plugin Server, using Nagios compatibility. But in this way, you will only get the statuses, since it does not use the descriptive output that some plugins for Nagios have.

Using the wrapper to use Nagios plugins in the software agent will solve this problem. The wrapper comes by default with the Unix 3.2 agent. An equivalent plugin for Pandora FMS Windows agents can be downloaded from the Pandora FMS resource library, at [2]).

What does the plugin wrapper do for Nagios plugins?

It executes the Nagios plugin, using its original parameters and converting the output into useful data for Pandora FMS. It has two types of information:

  • Status information: NORMAL (1), CRITICAL (0), WARNING (2), UNKNOWN () and others (4). By default, they will use a proc module, so the NORMAL and CRITICAL values are working "by default"; if you want to have information about WARNING and other values, you must configure the module thresholds manually.
  • Descriptive information: generally string information. It will be placed in the module description field. Usually something like "OK: successfully logged in."

1.12.6.1 Example

If you have a pop3 plugin (in /tmp/check_pop3_login) with the run permissions, (which checks if the pop3 account is working only by connecting to a remote host, sending a user and password and displaying everything is correct), then you can run it from the command line:

/tmp/check_pop3_login  mail.artica.es [email protected] mypass

it will return something like :

OK: successfully logged in.

And if it's not right, he'll return something like this.:

Critical: unable to log on

Using the wrapper is easy. Just put the wrapper and the name you want into the module before executing the call:

/etc/pandora/plugins/nagios_plugin_wrapper sancho_test /tmp/check_pop3_login  mail.artica.es [email protected] mypass

This will generate a complete XML for the agent plugin.

<module>
<name>sancho_test</name>
<type>generic_proc</type>
0
<description><![CDATA[Critical: unable to log on]]></description>
</module>

Or:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
1
<description><![CDATA[OK: successfully logged in.]]></description>
</module>

The complete entry in pandora_agent. conf will be something like:

module_plugin nagios_plugin_wrapper POP3_artica.es /tmp/check_pop3_login mail.artica.es [email protected] mypass

This will look similar to this in the module (in the fail event):


Sample plugin wrapper.png

1.13 Monitoring with KeepAlive

There is a special module, has a unique type called "keep_alive" and it's used to give information in the absence of contact with the agent. It is useful to know when an agent has stopped sending information and alerting us of this fact.

When there is a module, remote or local, that obtains information from the agent, the date of last "contact" with the agent is updated, so that whenever there is data, even if it is only one module of the total, the agent will have updated its date of last contact, which is useful to know if the agent "does not respond". Specifically, an agent is considered "dead" when it has not updated its date in twice the time of its interval, that is, if it has an interval of 5 minutes, and more than 10 minutes ago there is no update, the agent is considered dead.

This is the case when the keepalive module comes into play, triggering and marking the monitor into Critical status.

The configuration of this type of modules is very easy, just create a new module type "KeepAlive".:



Keepalive.JPG


Once created, if the agent has data, within its interval, it will always be in the "NORMAL" status (green):

Keepalive1.png


If the agent stops sending data (in this example, it had an interval of 1 minute), then it will automatically jump and be set to CRITICAL (red) status:

Keepalive2.png


It should be noted that if we have a remote module, for example, a Ping, in addition to the data reported by the agent, the keepalive module would never be triggered, since we are constantly updating the agent through Ping.

The keepalive module also behaves like any other module, it can be associated with an alert and can be used for any other element: reports, maps, etc.

Info.png

The keepalive module can be created only from the console (even if we don't have remote configuration enabled) and leaves no trace in the pandora_agent.con file.f

 


1.14 Command snapshots monitoring

Commands that have extensive outputs, such as top or netstat can be captured completely by a module and displayed in full. The module must be configured as a text type.

In the next screenshot, the result of one of these modules is shown, in this case the output of netstat -an:


Snapshot 1.png


Template warning.png

In order for this to work like this, it is necessary to configure properly both the Pandora FMS console (setup) and the agent that collects this information, making sure that it is untreated text.

 


To configure the console properly, make sure that in the main setup section we have the "Command line snapshot" property activated:

Command line snapshot setup.png

1.15 Image monitoring and visualization

This method allows you to define string type modules (generic_data_string or async_string) that contain images in text format with a base64 encoding, being able to display that image instead of a specific result. This is stored as text information, and displayed in a different way, not as simple data, but by reconstructing an image.

This is how you can see the output of a text string with the content "data: image" (image in base64), captured by Pandora FMS, when clicking on the special icon for image capture:


Snapshot text 1.png


To capture these images, just write a plugin that sends all the data, generating the necessary XML tags, and running the plugin as such, with the directive module_plugin. Let's see the following example plugin, which generates the output of an image, as you have just seen in the previous capture:

#!/bin/bash
echo "<module>"
echo "<name>Líder actual de la liga</name>"
echo "<type>async_string</type>"
echo "<data><![CDATA[data:image/jpeg;base64,/9j/4AAQSkZ....]]></data>"

The previous data would be generated by a device/application giving images in base64.

echo "</module>"

Save that content to a file in the agent (or distribute it with file collections) and run it as follows:

module_plugin <complete path to the file>

1.16 Password protected groups

By default, when an agent sends data for the first time to the Pandora FMS server, it is automatically added to the group that has been defined in the agent configuration file. This means that, in practice, anyone can add an agent to a group if they know the group name. This could be a problem if several clients share their Pandora FMS instance or if you want to control what is in each group.

We can optionally configure a password for a group from the pandora FMS Console. An agent will not be added to a group unless the correct password has been specified in the agent configuration file.

1.16.1 Example

To set a password for a group, navigate to the group editor and click on edit, enter the group password and save your changes:

Passgr1.JPG

To add a new agent to this group, edit your configuration file and add the following configuration option:

 group_password <password>

Don't forget to restart the agent to make the changes effective. The agent should be created correctly in the Pandora FMS console.


Go back to Pandora FMS documentation index