Pandora: Documentation en: Services
- 1 Service Monitoring
- 1.1 Introduction
- 1.2 Services under Pandora FMS
- 1.2.1 How Services work under Pandora FMS
- 1.2.2 Creating a New Service
- 1.2.3 Service Visualization
- 1.2.4 How to read the service values
- 1.2.5 Service grouping
- 1.2.6 Examples of services monitoring
- 1.3 Pandora Server
1 Service Monitoring
A service is a way to group your IT resources based on their functionalities.
A service could be your official website, your CRM system, your support application, or even your printers. Services are logical groups which can include hosts, routers, switches, firewalls, CRMs, ERPs, websites and of course, numerous other services.
In Pandora FMS, we represent the services as a grouping of monitored elements (modules, agents or other services) whose individual status affects in a certain way the global functionality of the service that is provided.
1.2 Services under Pandora FMS
1.2.1 How Services work under Pandora FMS
The basic monitoring in Pandora FMS consists of collecting metrics from different origins, representing them as monitors (modules).
The monitoring in services allows us to group these modules, so that, playing with certain margins based on the accumulation of failures, we can monitor groups of different types of elements and their relationship in a larger and general service.
In short, service monitoring allows us to check the status of a global service. We will be able to know if our service is being provided normally (green), degraded (yellow) or if we are not providing the service (red).
To better understand what service monitoring is all about, let's give a small example.
Suppose we want to monitor our web application, which we have balanced through a series of redundant elements. The infrastructure on which our application is based could consist of the following elements:
- Two routers in HA.
- Two switches in HA.
- 20 Apache Web Servers.
- Four Weblogic Appliance Servers.
- One MySQL Cluster consisting of two Storage and two SQL Processing Nodes.
Since our goal is to know if our web application is working correctly, that is, the final assessment by our customers is that the application works.
The need to monitor services as something "abstract" arises when faced with the following question:
What happens to my application if a non-critical element falls down?
For example, if one of the twenty Apache servers were to crash. In theory, we could not warn, because so much redundancy arises to have problematic situations covered. But then, which one to alert about? everyone? Just some? What's the rule for the warning?
We could think that Pandora should only warn us if a more critical element is dropped (for example a router) or if several Apache servers are dropped.
The monitoring through services in Pandora FMSfeature appears to solve all these doubts.
The services in Pandora FMS help us to:
- Limit the number of received alerts. We will receive alerts about situations that compromise the reliability of the services we provide.
- Track the compliance level.
- Simplify the visualization of the monitoring of our infrastructure.
To achieve this, we will have to monitor every element that could negatively affect our application.
Through the Pandora FMS console, we will have to define a service tree in which we will indicate both the elements that affect our application, as well as the degree to which they affect.
All the elements we add to the service trees will correspond to information that is already being monitored, whether in the form of modules, specific agents or other services.
To indicate the degree to which the status of each element affect the overall status, a system of sum of weights will be used, so that the most important (with more weight) will be more relevant to adjust the overall status of the complete service to an incorrect status before the less important elements (with less weight).
Let's look at all these ideas through a practical example:
- Switches and routers: 5 points each when in critical, and 3 points if in warning.
- WEB servers: 1.2 points for each one in critical, we do not contemplate the warning status.
- WebLogic Servers: 2 points each in critical.
- MySQL Cluster: 5 points for each node in critical and 3 points in warning.
|Element type||Weight asignment|
We set a warning threshold for service of 4, and a critical threshold of 6. In this way, and assuming that everything is going well, the service would be "OK" if all the monitored elements are OK or not important enough to cause deficiencies in the provision of our service.
Now let's suppose that one (1) Apache web server goes down:
- 1 x SApache server in CRITICAL x 1.2 pto = 1.2 because 1.2 < 4 (Warning), the service is stil in OK status.
The weight contribution will be:
2 x 0 (routers in OK) + 2 x 0 (switches in OK) + 19 x 0 (apache OK) + 1 x 1.2 (apache CRIT) + 4 x 0 (weblogic OK) + 1 x 0 (mysql OK) Total: 1.2 --> Our service will be in NORMAL
Let's see what happens if a WEB server and a Weblogic go down:
- 1 x Apache Server in CRITICAL x 1.2 pto = 1.2
- 1 x Weblogic Server in CRITICAL x 2 = 2
Total, 3,2 is still< 4 so the server remains in OK status, it is still working, it's not necessary to make a technical action immediately.
The weight contribution will be:
2 x 0 (routers in OK) + 2 x 0 (switches in OK) + 19 x 0 (apache OK) + 1 x 1.2 (apache CRIT) + 3 x 0 (weblogic OK) + 1 x 2 (weblogic CRIT) + 1 x 0 (mysql OK) Total: 3.2 --> Our service will be in NORMAL
Let's see what happens if two WEB servers and a WEblogic go down:
- 2 x Apache Server in CRITICAL x 1.2 pto = 2.4
- 1 x Weblogic Server in CRITICAL x 2 = 2
Total, 4.4 now it is > 4 and the service goes into WARNING status, our service has gone into a degraded status. It continues to work, and may not require immediate technical action, but it is clear that there has been a problem with our infrastructure.
2 x 0 (routers in OK) + 2 x 0 (switches in OK) + 18 x 0 (apache OK) + 2 x 1.2 (apache CRIT) + 3 x 0 (weblogic OK) + 1 x 2 (weblogic CRIT) + 1 x 0 (mysql OK) Total: 4.4 --> Our service will be in WARNING
Let's suppose that in addition to the above, a Router goes down:
- 2 x Apache Server in CRITICAL x 1.2 pto = 2.4
- 1 x Weblogic Server in CRITICAL x 2 = 2
- 1 x Router in CRITICAL x 5 = 5
Now we have 9.4 above the threshold set at 6 for CRITICAL, so, the service is in critical, it is not working Immediate technical action is imperative.
1 x 0 (routers in OK) + 1 x 5 (router in CRIT) + 2 x 0 (switches in OK) + 18 x 0 (apache OK) + 2 x 1.2 (apache CRIT) + 3 x 0 (weblogic OK) + 1 x 2 (weblogic CRIT) + 1 x 0 (mysql OK) Total: 9.4 --> Our service is in CRÍTICAL
Pandora FMS will alert the corresponding work team (operators, technicians, etc.).
Service monitoring is a feature only of the Enterprise version of Pandora FMS.
184.108.40.206 How the simple mode works
The weight system as detailed above may be too complex when the monitoring needs are basic. To deal with this situation, a new simple mode is available on the service configuration since the 5.1 version.
In this mode the only configuration needed is to select which elements are critical and which not. Only the critical elements will be taken into account when calculating the service status and only the critical status of the critical elements will have value. The service will go to warning when 50% of the critical elements reach critical status. When 50% of the critical elements in critical are surpassed, the service will go to critical.
Let's follow an example of a simple service:
- Router as critical element.
- Printer as non critical element.
- Apache server as critical element.
One day the elements report this status:
- Router on critical.
- Printer on critical.
- Apache server on warning.
The service status is warning, because the printer isn't a critical element and its status is not taken into account, as well as the Apache service status, which, even though a critical element, will only be taken into account in critical status. In this situation, one critical element is on critical status, 50% of the critical elements.
Another day the elements report this status:
- Router on critical.
- Printer on critical.
- Apache server on critical.
The service status is critical, since over 50% of the critical elements are on critical status.
Finally, another day the elements report this status:
- Router on normal.
- Printer on critical.
- Apache on normal.
The service status is normal, since less than 50% of the critical elements are on critical status. Only the printer is on critical status but, as we have seen, the non critical elements aren't taken into account when calculating the service status.
1.2.2 Creating a New Service
You need the Enterprise version and the PredictionServer component enabled to be able to use the services.
- Other Services
The service values are calculated using the Prediction Server which utilizes the default interval of the prediction modules.
Once you have all the devices monitored. Within each service, you can add all the modules, agents or sub-services you need to monitor the service. For example, if you want to monitor the Online Store service you need a module for content, a service that monitors the status of communications and so on. Through the following steps, you can see how to create a service with Pandora FMS.
To create a new service, please click on Services at the Topology Maps .
A list of all the available services will be shown. The next screenshot shows an empty service list.
To create a new service, just click on the 'Create' button and fill out the form as shown below.
The names of the form fields and their meaning are as follows:
- Name: The name of the service.
- Description: The description of the service, a long text that can be optional.
- Group: The group of the service. It's quite useful for organization purposes and to enforce the SLA (Service Level Agreement) restrictions.
- Mode: the mode in which the calculation of the weight of the elements will be performed.
- Manual: the weights should be entered manually into the service and their elements.
- Auto: implying the 'critical' threshold for the service to be '1' and the 'warning' threshold to be '0.5'. It's also assumed that you'll automatically assign weights of '0' for the 'OK' status, '0.5' for 'warning' and '1' for 'critical' each time you're creating an element for this service.
- Simple: there is no need to enter weights, only enable or disable a checkbox to indicate if the element is critical.
- Critical: The weight threshold to enter the 'critical' status. This field is disabled if the auto-calculate check is enabled. The default value is '1'.
- Warning: weight threshold for declaring service in warning status. This field is disabled when the automatic mode is selected and has the default value of '0.5'. Not visible when the simple mode is selected.
- Agent to store Data: the service stores the data in special data modules (specifically the prediction modules) and it is necessary to introduce an agent to be the container of these modules, as well as the alarms that you will later have to configure in this form. Note Please note that the interval at which all service module calculations will be performed depends on the agent interval configured as a container.
- Quiet:Activates the silent mode of the service, it will not generate alerts or events
- Cascade Protection:Activates cascade protection over the elements of the service. These will not generate alerts or events if they belong to a service (or subservice) that is in critical condition.
- Calculate continuous SLA for this service: Activates the creation of SLA and SLA value modules for the current service. If disabled, dynamically calculated SLA information is not available, and SLA compliance alerts for this service do not work. It is used for cases where the number of services needed is so high that it can affect performance.If this option is disabled, once the service has been created, the data history of these modules will be deleted and information will be lost.
- SLA Interval: The time range for performing the SLA constraint's calculation. The default value is '1 month'.
- SLA Limit: OK status threshold of the service considered an SLA as positive for the period of time you have set in the previous field.
- Warning Service Alert: alert template that the service will use to issue the alert when the service goes into warning status.
- Critical Service alert: alert template that the service will use to issue the alert when the service goes into critical status.
- SLA Critical Service Alert: alert template that the service will use to issue the alert if the SLA restrictions aren't met.
Once the form has been filled in correctly, you will have an empty service which must be filled in with elements or service items as we will see below. In the service edit form, the' Config Elements' tab is selected.
You'll see a page like the one below where you can manage (modify, add new ones or delete) service elements.
Some important items on the services configuration page are:
- Type: a drop-down list that can show service, module or agent.
- Agent: The smart-search input control for the agent. It's only visible if the element type is either the 'agent' or the 'module' type.
- Module: The drop-down list along with the modules' agent under previously chosen via smart search. This control is only visible when editing or creating a service element for the 'module' type.
- Service: The drop-down list of the services to create an item. It's only visible if the item is of the 'create' or 'edit service' type. It's also important to keep in mind that the services which are going to appear in the drop-down list are not ancestors of the service. It's also necessary to show an appropriate tree-structure dependency between the services.
- Critical: A checkbox to select if the element is critical. Not visible unless the service is in simple mode.
- Weight on Critical: The weight of the element if it's in a 'critical' status. The default value is '1'. It's disabled if the service is in 'auto calculate' mode. Not visible if the service is in simple mode.
- Weight on Warning: The weight of the 'warning' status. The default value is '0.5'. It's disabled if the service is in 'auto calculate' mode. Not visible if the service is in simple mode.
- Weight on Unknown: The weight of the element if it's in unknown status. The default value is '0'. It's disabled if the service is in 'auto calculate' mode. Not visible if the service is in simple mode.
- Weight on OK: The weight of the element if it's in perfect conditions. The default value is '0'. It's disabled if the service is in 'auto calculate' mode. Not visible if the service is in simple mode.
Once you have created the service items on this page, we're looking at a list management similar to the one shown in the picture below:
In which, in the last column on the right, entitled "Actions", you have icons for:
- Edit: which is the icon represented by a wrench with an orange handle. Edit the element of the row corresponding to that icon.
- Delete: which is the icon represented by a red cross. When clicking on it, you will be asked in a modal window for confirmation to remove and delete the service element from the database.
220.127.116.11 Modules created when configuring a service
- SLA Value Service: The percentage value of the SLA compliance. (async_data).
- Service_SLA_Service: This shows if the SLA is being accomplishing or not. (async_proc).
- Service_Service: This module shows the sum of the weights of the service. (async_proc).
1.2.3 Service Visualization
18.104.22.168 Simple list-based View of all the Services
It is the operation list that shows all the services created, of course, it only shows those of the groups that the user that is using the Pandora console has access to.
To get to this view, just go to the Operation menu, open the Monitoring entry and within this is the Services section.
Each row represents a service, and the columns represent:
- Name: The name of the service.
- Description: The service description.
- Group: The icon of the group the service belongs to.
- Critical: The threshold value for the sums of weights to put the service into 'critical' status.
- Warning: The threshold value for the sums of weights to put the service into 'warning' status.
- Value: The current value for the sum of all weights for the service.
- Status: An icon which represents the status of the service.
Four possible status are represented:
- Red: The service is in 'critical' status because the value exceeded the critical threshold.
- Yellow: The service is in 'warning' status because the value exceeded the critical threshold.
- Green: The service is within the 'normal' range.
- Gray: The service is in 'unknown' status. This usually means the service has been recently created and doesn't contain any modules or the Prediction Server is down.
- SLA: The current value of the SLA Service. The values can be:
- OK: the SLA is met for the interval defined in the SLA service.
- INCORRECT: The SLA is not meant for the interval currently defined in the SLA Service.
- N/A: The SLA is in 'unknown' status because there is insufficient data to perform the calculation.
22.214.171.124.1 Table of all services
A table for quick display of all visible services and their current status.
126.96.36.199.2 List-based view of a Service and its Elements
This view is accessible by clicking on the name of a service in the list of all services, or through the magnifying glass icon tab in the service title header.
Pandora will show a page similar to the one shown in the following screenshot:
In the screenshot, we can distinguish two zones, the service with the same columns as in the previous view at the top. And the list of the elements that compose this service at the bottom.
The list of elements appears in table format, where the rows correspond to each element and the columns represent:
- Type: The icon which represents the type of an element. It's a Lego block for modules, some stacked Lego blocks for an agent and a Network Diagram Icon for the services.
- Name: The text which contains the name of the module, agent or service. They're also linked to the proper section.
- Description: A small free-text field intended for a short description.
- Weight critical: The value if the element is in 'critical' status.
- Weight warning: The value if the element is in 'warning' status.
- Weight normal: The value if the element is in 'normal' status.
- Data: The value of the element. It's able to adopt the following modes:
- Module: The value of the module.
- Agents: The text which displays the agent's status.
- Service: The sum of all elements' weights from the selected service.
- Status: The icon which represents the element's status by color.
Keep in mind that the service-elements calculation is performed by the Prediction Server. It's not real-time data you're looking at. There are some situations in which a module's agent is added to the service where its weight is not going to be updated until the calculation is performed by the Prediction Server again.
188.8.131.52.3 Service Map View
To access this view, you're required to click on the flap above the header in the service operation view, as you can see in the picture below.
This view will display the service in arborescent form as you can see in the following screenshot. In this way, it is possible to quickly see how modules, agents or sub-services influence the monitoring of the service. Even in the subservices you can see what influences them when calculating the status by the sum of the weights.
The possible nodes can be:
- Module Node: It's represented by the 'heartbeat' icon. This node is always final (leaf).
- Agent Node: It's represented by the 'CPU box' icon. This module is also always final (leaf).
- Service Node: It's represented by the 'crossed hammer and wrench' icon. This module is not a final node. It's required to contain additional nodes.
The node's colors and the arrow which connects them to the service depend on the node's status.
There are the following attributes within the node:
- Title: The name of the service's / agent's or module's node.
- Value list: This list refers to the possible numeric value calculated for that instance. It accepts any assigned integer value.
- Critical:: The weight if it reaches 'critical' status, except if it's the root-service node, which represents a threshold to reach the 'critical' status.
- Warning: The weight if it reaches 'warning' status, except if it's the root-service node, which represents the threshold to reach the 'warning' status.
- Normal: The weight if it reaches 'normal' status, except if it's the root-service node, in which case nothing is going to be displayed here.
- Unknown: The 'unknown' status, except if it's the root-service node, which represents a threshold to reach the 'unknown' status.
You may click on each node in the tree. The target link represents the operational view of the node itself.
When the service mode is simple, a red exclamation mark appears on the right side of the critical elements.
184.108.40.206.4 Services within the Visual Console
From Pandora FMS versions 5 and above, you may add services in the Visual Console like any other item on the map.
To create a service item on a map, the process is the same as for all other visual map items, but the options palette will be the same as in the screenshot.
It contains the following attributes:
- Label: The title which is going to be shown within the visual console's node.
- Service: The service that's going to be represented.
Note that a service item, unlike other items on the visual map, cannot be linked to other visual maps, and always the clickable link in the visual console is intended for the tree service map view described above.
1.2.4 How to read the service values
Planned shutdowns added before the stop date allow us to recalculate the value of the SLA reports. First, we need to activate it in the general setup. When it comes to an SLA service report, if there is a scheduled outage affecting one or more elements of the service, it is considered that the planned shutdown affects the entire service, because the system cannot evaluate the impact of a service component "inactive" in the whole service.
It is important to remember that this is at report level; service map, and the information presented in the visual console are not altered based on planned shutdowns added after the effective execution date. These service compliance percentages are calculated in real-time, based on the history data of the same service, it is very different than a report which can be "cooked" adding a "fake" planned downtime.
On the other hand, it is important to know how the compliance of a service is calculated:
Let's suppose we have a service defined by a 95% compliance in an interval of 1 hour (this is very short for the real world, but good for understanding the internal algorithm). We will use a table of values, where t is time, x is the % compliance (SLAs), and s is whether or not the service complies (1 complies, 0 fails). In 1 hour we should have exactly 12 values, assuming an interval of 5 minutes.
A similar case, where the service complies for the first 11 samples (first 55 minutes) and in the 60th minute, it fails, we would have these values:
t | s | x --------+-------+-------- 1 1 100 2 1 100 3 1 100 4 1 100 5 1 100 6 1 100 7 1 100 8 1 100 9 1 100 10 1 100 11 1 100 12 0 91,6
This case is easier to calculate. The % is calculated depending on the number of samples, for example in t3 there are a total of three samples that meet service, 100%, whereas t12, we have 12 and 11 valid samples: 11 / 12.
Assume you are in the middle of the series, and it is recovering slowly:
t | s | x --------+-------+-------- 1 1 100 2 1 100 3 1 100 4 1 100 5 1 100 6 0 83,3 7 1 85,7 8 1 87,5 9 1 88,8 10 1 90 11 1 90,9 12 1 91,6
So far all seems similar to the previous scenario, but let's see what happens if we go over time:
t | s | x --------+-------+-------- 13 1 91,6 14 1 91,6 15 1 91,6 16 1 91,6 17 1 91,6 18 1 100 19 1 100 ....
Now we see unintuitive behavior, because the volume of valid samples remains 11 for a window of time to get to t18, where the only invalid value is out of the window, so in t18 compliance becomes 100%. This step between 91.6 and 100 is explained by the size of the window. The larger the window is (usually SLA calculation interval is daily, weekly or monthly), the less abrupt will be the step.
1.2.5 Service grouping
Services are logical groups that conform part of a business structure. Due to that, it makes sense to group services, because in a lot of cases there can be dependences between them, conforming, for example, a global company service composed by some other particular services (webpage, communications, etc). To group the services, it's necessary to create the big general service and the smaller ones that will be aggregated to the global service, creating a logical tree structure.
The service groups can help us to: create visual maps, configure alerts, apply monitoring policies, etc. So we can create specific alerts when the company service is down due to the commercial department not being able to work, or the webpage being offline.
Next we have two examples to understand service grouping.
1.2.6 Examples of services monitoring
220.127.116.11 PandoraFMS service
In this case the service of PandoraFMS is being monitored. It is composed of the Apache service, MySQL, Pandora server and Tentacle server. Every one of these elements also constitutes a service with different components, creating a tree-type structure.
The general Pandora service will turn into critical status if it reaches the weight of 2, and warning status with 1. As you can see, the four components have different weights over the Pandora service:
- MySQL: critical for the Pandora service, individual weight of 2 if MySSQL is down. It will have weight 1 if it turns into warning status, already displaying yellow status on the Pandora service.
- Pandora Server: critical for the Pandora service, individual weight of 2 if Pandora Server is down. Individual weight of 1 if it is on warning status, displaying the warning status on the Pandora service for example if it reaches a heavy CPU load.
- Apache: it means a degradation of the global Pandora service, but not a complete interruption, so it will have an individual weight of 1 if it is down, showing the warning status on the Pandora service.
- Tentacle: same as the Apache, it means a degradation of the service, but not a total interruption, so it gets 1 of weight if down, and will display warning status.
In the following picture we can see the setup of the different weights for the elements over the Pandora general service:
18.104.22.168 Storage cluster, grouping of services
Services are logically arranged groups which are part of a company's business structure. Therefore, it may be necessary to create groups of services, because services alone sometimes don't have an appropriate context. To create service groups, you're required to add each service to an existing agent. In this case, a service is going to be a module of an agent. You're able to create a new logical structure (a group of services) by these groups.
On the following example we have an HA storage cluster. For this case there are two fileserver systems working in parallel, each one controlling the percentage and status of some different disks that provide service to specific departments, creating a tree-type structure with grouped services.
According to this structure, the critical threshold of the storage service will be reached only if both of the fileservers fail, this would totally deny the service, and if only one of the fileservers fail it would only suppose a degraded service. In the screenshot below we can appreciate the weight configuration of the two main elements of the storage service:
In the following image, we can see the content and weight configuration of the grouped service FS01. Here, the elements will have a specific weigh according to its criticalness, being:
- FS01 ALIVE: critical to the FS01 service, since it is the virtual IP assigned to the first disk cluster. Individual weigh 2, if it's down, the other elements would automatically be inoperative. In this case there is no warning threshold, since it is a yes/no based type of information.
- DHCPserver ping: critic to the FS01 service, we give it an individual weight of 2. In this case there is no warning threshold either.
- Disks we give them an individual weight of 1 in case they reach its own critical status, and 0.5 for their warning status. According to this, the FS01 service will only reach critical status if there are two disks on critical status o four in warning status.
1.3 Pandora Server
It's mandatory the Prediction Service is running appropriately and also to have the Enterprise Version of Pandora FMS installed.