Pandora:Documentation en:Server Management

From Pandora FMS Wiki

Jump to: navigation, search

Go back to Pandora FMS documentation index

Pandora FMS Server maintenance

Database management

Pandora FMS infrastructure does not need external maintenance, but it is very important to purge old data, and maintain the database compacted. There is an essential tool for the proper functioning of Pandora FMS. This tool must be launched only once by night. If you have multiple physical servers, start it from one of them. This tool must be launched from a system where there is a Pandora FMS server, if you have two systems and one has the server and another one the console, run it from the server where server is the Pandora FMS. This tool is at:

/usr/share/pandora_server/util/pandora_db.pl

This tool, hereinafter pandora_db.pl is included in the package Pandora FMS server.

This tool performs all maintenance database automatically and is essential for the proper functioning of Pandora FMS so be sure it works properly. Its functions are:

  • Delete old data.
  • Compact existing data, interpolated at various intervals, so that graphics are the same but the space needed for storage is much lower (this is one reason why Pandora FMS is capable of processing such information).
  • Check the consistency of the database for non-existing modules, or modules which are not used because they can not be initiated (these modules appear in print as uninitialized modules ).
  • Eliminates the daily information contact the agent. Pandora FMS does not need more than historical 24hr contact agent, and if it builds up, slows down in the access to the database.
  • The enterprise version, moving all the old data to the standby database as historic.

This task should be performed every night, and it is very important to do it, taking time to understand and fixing the cron task. The installation should have been programmed correctly, but you should check that this has been done. This chapter explains how program it manually, so you can verify whether the installation on your system is working properly.

To install this tool on standard Linux systems, we recommend the following procedure:

  • Create a new file called /etc/cron.daily/pandora_db that contains following lines:
#!/bin/bash
/usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf
  • Alter permissions of file:
 chmod 750 /etc/cron.daily/pandora_db
  • Change ownership
 chmod root:root /etc/cron.daily/pandora_db
  • Reload cron configuration
/etc/init.d/cron reload

From now on, every night it will run the maintenance tool database Pandora FMS, ensuring that the database is always in optimum condition.

Finally, to ensure that you have left everything up correctly, or to ensure that the system, after installing the tool has been programmed correctly, manually run it once:

/etc/cron.daily/pandora_db

It should show a message like this (perhaps with less information, depending on their level of verbosity in the configuration file of Pandora FMS):

Pandora FMS DB Tool 3.0-dev PS090930 Copyright (c) 2004-2008 Artica ST
This program is Free Software, licensed under the terms of GPL License v2
You can download latest versions and documentation at http://www.pandorafms.org
Pandora DB now initialized and running (PURGE=60 days, COMPACT=15 days, STEP=1) ... 

Starting at 2009/10/10 02:02:18
[PURGE] Deleting old event data (More than 60 days)... 
[PURGE] Deleting old data... 
[PURGE] Delete old data (string) ... 
[PURGE] Delete pending deleted modules (data table)...
[PURGE] Delete pending deleted modules (data string table)...
[PURGE] Delete pending deleted modules (data inc  table)...
[PURGE] Delete pending deleted modules (status, module table)...
[PURGE] Delete old session data 
[PURGE] Delete old data from SNMP Traps 
[PURGE] Deleting old access data (More than 24hr) 
[CHECKDB] Deleting non-init data... 
[CHECKDB] Checking database consistency (Missing status)... 
[CHECKDB] Checking database consistency (Missing module)... 
[CHECKDB] Deleting non-existing module 1189 in state table 
[CHECKDB] Deleting non-existing module 1190 in state table 
[COMPACT] Compacting data until 2009092502:02:18
Ending at 2009/10/10 02:02:31

Manual Execution of Maintenance Tool

It is possible to execute manually the maintenance tool once the script has been created. Its use is very easy. From a shell console, execute:


Template warning.png

In installed systems this could take hours. It is recommended to leave the process in second level

 


/usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf

To execute manually the maintenance tool and leave it in second level, execute:

nohup /usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf

The process will take some time until it load throughly in second level. After, you could close the shell console window without problems, while the process will continue executing.


Info.png

In some installations the tool directory could change. The most common one is:

/usr/share/pandora_server/util/

In Pandora FMS previous versions, it could be find at:

/usr/share/pandora/util/

 


It is very important that you make sure to use the current version of the tool, and not the one from a previous version. If you execute the program without arguments, it will show the tool version at the head of the message.

Database Backup

A simple command mysqldump, will do a dump of the database contents.To restore data it will be necessary an empty database with the same name that the original one (usually Pandora).

Doing the Backup

mysqldump -u root -p pandora > /backup/pandoradb_backup.sql

Restore the Backup

mysql -u root -p
create database pandora;
use pandora;
source /backup/pandoradb_backup.sql

Probably, it will be also necessary to create new permissions to the console user:

grant all privileges on pandora.* to pandora@localhost identified by 'mypassword';

If you want to do a complete backup of the system, do not forget to do a backup of the whole directory /etc/pandora, to keep the information of the configuration of the local agents and the servers.

It is important to emphasize that this ONLY do a backup/restoration of the database files.

Backup and Complete Recovery of Pandora FMS

There is an script in the Pandora FMS server distribution that is useful to do a backup and a complete restoration of all Pandora FMS. This script is intended to do copies and restoration in systems where the server and the console are located in the same machine. If in your environment there are several components, then you should use the tool with the most adequate parameters for its use or modify them so they could be adapted to their circumstances.

In order it could do its tasks, this script needs to be executed as root.

This script is located at:

/usr/share/pandora_server/util/pandora_backup.sh 

If we execute it without parameters, it will give us some help:

Pandora FMS Command line backup tool. http://www.pandorafms.org
(c) 2009 Sancho Lerena <slerena@gmail.com>, Artica Soluciones Tecnologicas

Syntax:
		-c Path to Pandora FMS console, p.e: /srv/www/htdocs/pandora_console
		-d Destination path for backup file. p.e: /tmp
		-s Source filename for backup restore. p.e: /tmp/pandorafms
		-f Restore also files
		-q Quiet. No output message (used for scripts/cron)
 		-b No database backup/restore


Please BE SURE TO USE RESTORE (-s) option. This will OVERWRITE ALL your
PandoraFMS install, including files, configuration and data. Please backup first!

This script is designed to do security copies and restoration of the following components:

  • Server Configuration File(s).
  • Files waiting for execution, and also agent remote configuration files.
  • Complete DB.
  • Complete WEB Console.

Origin and Destination Options of the Copy

This script obtains the credentials to have access to the DB directly from the WEB console configuration. Is because of this that you should go, with the -cparameter the complete path to the WEB console. This same parameter is used also to show it where it will find the WEB console to do its backup.

The backup destination is specified with the -d parameter. In this path, it will leave the backup file compressed, with a name similar to pandorafms_backup_xxxxxxx.tar.gz.

The source origin of the restoration is the complete name and path of the backup generated file by this same tool.


File Restoration, not only Data Restoration

The -f option also allows to restore the files (overwriting the current ones) of a security copy, not restoring the data from the database. As overwriting the current configuration files could have serious consequences, it is necessary to use -f if we want to do a backup recovery and we want to it restore all the Pandora files (Console and Server).

File Restoration, without Data

Same as with the previous option, we could restore only the files, without dumping the data. To do it, use the -boption.

Data Restoration, without Files

It is the default option. For doing this, you will not have to use neither the -b option nor the -f option.

Examples of Use

Create backup

Execute as root:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/pandora3 -d /tmp/ 

It will return something similar to this:

Backup completed and placed in /tmp//pandorafms_backup_2009-10-10-01-35-31.tar.gz

This means that the backup is at /tmp//pandorafms_backup_2009-10-10-01-35-31.tar.gz

Restoring Backup

To restore the backup in an automatic way, you are supposed to have a console with the authentication credentials on the DDBB correctly defined.

Execute as root:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/pandora3_broken/ -s /tmp/pandorafms_backup_2009-10-10-01-35-31.tar.gz 

It will give back something similar to:

Detected Pandora FMS backup at /tmp/pandorafms_backup_2009-10-10-01-35-31.tar.gz, please wait...
Dropping current database
Restoring backup database
Restoring files and configuration
Done. Backup in /tmp/pandorafms_backup_2009-10-10-01-35-31.tar.gz restored

Restoring Backup in case of disaster

If you have lost the Pandora FMS console but you have a backup generated by this tool,then, first you will have to restore the console directory. For it, decompress manually its backup:

cd /tmp
tar xvzf pandorafms_backup_2009-10-10-0

This will unpack your WEB console complete directory in /tmp. In the case of the generated backup of the previous example, it creates a directory named:

/tmp/var/www/pandora3/

Copy the content of all this directory to to your web publication directory, that could change depending on the distribution you use:

cp -R var/www/pandora3 /var/www

Then restore the backup as usual.

Manual startup/shutdown for Pandora FMS servers

To start and / or stop the server manually Pandora FMS is running the following in a console shell':

Stop daemon:

/etc/init.d/pandora_server stop

Start daemon:

/etc/init.d/pandora_server start

Restart daemon:

/etc/init.d/pandora_server restart

Watchdog implementation for Pandora FMS

In the repository there is a small script that is used as a "watchdog" (Watchdog). This script performs a monitoring Pandora (who monitors who monitored?). Thus we can perform a recovery operation (trying to lift Pandora), and if that fails, we can tell the event.

pandora_watchdog.sh

#!/bin/bash
# Copyright (c) 2005-2012 Artica ST
# Author: Sancho Lerena <slerena@artica.es> 2009
# Licence: GPL2
#
# daemon_watchdog
#
# Generic watchdog to detect if a daemon is running. If cannot restart, execute
# a custom-user defined command to notify daemon is down and continues in
# standby (without notifying / checking) until daemon is alive again.

# Default configuration is for Pandora FMS Server daemon

# =====================================================================
# Configuration begins here. Please use "" if data contain blank spaces

export DAEMON_WATCHDOG=pandora_watchdog.sh
# DAEMON_WATCHDOG: Name of this script. Used to check if its running already

export DAEMON_CHECK="/usr/bin/pandora_server /etc/pandora/pandora_server.conf"
# DAEMON_CHECK: Daemon monitored, please use full path and parameters like
#               are shown doing a ps aux of ps -Alf

export DAEMON_RESTART="/etc/init.d/pandora_server restart"
# DAEMON_RESTART: Command to try to restart the daemon

export DAEMON_DEADWAIT=90
# DAEMON_DEADWAIT: Time this script checks after detect that
#                  daemon is down before to consider is really down. 

export DAEMON_ALERT="/usr/bin/pandora_alert"
# DAEMON_ALERT: Command/Script executed if after detecting daemon is down,
#               and waiting DAEMON_DEADWAIT, and daemon continues down. 

export DAEMON_LOOP=7
# DAEMON_LOOP: Interval within daemon_wathdog checks if daemon is alive.
#              DO NOT use values under 3-5 seconds or could be CPU consuming.
#              NEVER NEVER NEVER use 0 value or gets 100% CPU!.

# Configuration stop here
# =====================================================================

# Check if another instance of this script

RUNNING_CHECK=`ps aux | grep "$DAEMON_WATCHDOG" | grep -v grep |wc -l`
if [ $RUNNING_CHECK -gt 2 ]
then
        echo "Aborting, seems that there are more '$DAEMON_WATCHDOG' running in this system"
        logger $DAEMON_WATCHDOG aborted execution because another watchdog seems to be running
        exit -1
fi


# This value always must be 0 at start. Do not alter
export DAEMON_STANDBY=0

# This function replace pidof, not working in the same way in different linux distros
function pidof_daemon () (
        # This sets COLUMNS to XXX chars, because if command is run
        # in a "strech" term, ps aux don't report more than COLUMNS
        # characters and this will not work.
        COLUMNS=300
        DAEMON_PID=`ps aux | grep "$DAEMON_CHECK" | grep -v grep | tail -1 | awk '{ print $2 }'`
        echo $DAEMON_PID
)

# Main script

if [ ! -f `echo $DAEMON_CHECK | awk '{ print $1 }'` ]
then
        echo "Daemon you want to check is not present in the system. Aborting watchdog"
        exit
fi

while [ 1 ]
do

        DAEMON_PID=`pidof_daemon`
        if [ -z "$DAEMON_PID" ]
        then

echo "Checkpoint #1 $DAEMON_PID "
 
                if [ $DAEMON_STANDBY == 0 ]
                then

                        # Daemon down, first detection
                        # Restart it ! 

                        logger $DAEMON_WATCHDOG restarting $DAEMON_CHECK
                        $DAEMON_RESTART 2> /dev/null > /dev/null

                        # Just WAIT another DAEMON_DEADWAIT before consider it DEAD

echo "Going to DAEMON_DEADEWAIT"
 
                        sleep $DAEMON_DEADWAIT
                        DAEMON_PID=`pidof_daemon` 

                        if [ -z "$DAEMON_PID" ]
                        then

                                # Is dead and can't be restarted properly. Execute alert

echo "I cannot startup again the process"

                                logger $DAEMON_WATCHDOG $DAEMON_CHECK is dead, alerting !
                                $DAEMON_ALERT  2> /dev/null > /dev/null 

                                # Watchdog process puts in STANDBY mode until process get alive again
                                logger $DAEMON_WATCHDOG "Entering in Stabdby mode"

                                DAEMON_STANDBY=1
                       fi
               fi
       else

echo "Checkpoint #1B $DAEMON_PID "

                DAEMON_STANDBY=0
        fi

        sleep $DAEMON_LOOP
done

/usr/bin/pandora_alert

This is the script that acts when the watchdog process cannot start the process that monitors (pandora). In our case, besides alert by SMS, it disables Tentacle.

There will be given rights with chmod 750 / usr / bin / pandora_alert


#!/bin/bash
sendsms +34458474843 "Pandora FMS Server stopped and can't be started"
/etc/init.d/tentacle_serverd stop


Watchdog Startup

If you have copied pandora_watchdog.sh to /usr/bin, the manual way to start the wathdog will be:

nohup /usr/bin/pandora_watchdog.sh &

Remarks

Having a watchdog running on the system can cause unpleasant consequences if we do not consider that there is a watchdog. If for example, we want to make a Pandora to disconnect maintenance, the watchdog will automatically rise again, so we will go "crazy" if we do not stop watchdog first.

History database

A history database is a database where old module data is moved to make the main Pandora FMS database more responsive for everyday operations. That data will still be available seamlessly to the Pandora FMS console when viewing reports, module charts etc.

Setting up a history database

To configure a history database follow these simple steps:

  • Create the new history database.
  • Create the necessary tables in the new database. You can use the pandoradb.sql script provided with the Pandora FMS console:
cat pandoradb.sql | mysql -u user -p -D history_db
  • In your Pandora FMS console navigate to Setup->History database and enter the host, port, database name, user and password of the new database.



History db conf.png



Data older than Days days will be moved to the history database in blocks of Step rows, waiting Delay seconds between one block and the next to avoid overload.

Info.png

This is an Enterprise feature

 


Setting up history database purge and compactation

History database is supposed to contains "all history data", but if you want to delete old data, or compact them, you will need to execute pandora_db script with a "fake" configuration data, to let it think is the "normal" behaviour. To do this, you need to enter some data in your History Database.

First, you need to "recreate" a functional table values in tconfig, with useful values to be used by pandora_db tool. Use this SQL queries ON YOUR history database to create a minimal configuration for the behaviour of the pandora_db running against history database. You fist, need to connect your database, using mysql CLI tool.

This is an example, replace values as you needed (but let history_db_enabled to zero):

INSERT INTO `tconfig` VALUES (1,'days_purge','180');
INSERT INTO `tconfig` VALUES (2,'history_db_enabled','0');
INSERT INTO `tconfig` VALUES (3,'days_compact','120');
INSERT INTO `tconfig` VALUES (4,'step_compact','1');

This means data in history is stored for 6 months (starting from NOW), and compacted from 4th month. If you have 1 month of data in your "main" database, you will have a total of 6 months of data, because the last month is coming from the main database, and the other five from the history database. You can put any value here, there is no limit of data storage in history database. Just remember, history database MUST BE IN A DIFFERENT PHYSICAL SERVER, not in the same host/system where is the main database and/or running Pandora FMS server or console.

Second, you need to create an aditional pandora_server.conf file, use this "small version" to create your own (replace the values for your history database values), name it /etc/pandora/pandora_server_history_db.conf :

dbengine mysql
dbname pandora4_history
dbuser pandora4_history
dbpass 1234
dbhost 192.168.50.23
log_file /var/log/pandora/pandora_db_history.log


Now you can execute pandora_db tool against the "fake" configuration:

/usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server_history_db.conf

This process SHOULD NOT affect your main operation because is running against a different database in a different server. The only possible delay should be if someone is trying to render a big ammount of data from history, that will take more time than usual.


Go back to Pandora FMS documentation index