Pandora: Documentation en: User Monitorization

From Pandora FMS Wiki
Jump to: navigation, search

Contents

1 Introduction

User experience monitoring consists of making recordings of automated tasks for both web browsing and interaction with the desktop and Windows system applications. These recordings can consist of a click of a mouse on a browser, write a text and perform a search in web browsing, such as being able to open an application on the desktop. This will allow us to perform the recorded functions automatically and run to get results, such as time spent on tasks. To do this we have automated UX monitoring and monitoring user web experience (WUX).

From Pandora FMS version 7.0NG.728, the UX and WUX monitors have been merged inside Pandora FMS console, having both functionalities in only one screen.

Wux ux unifcado.png

Next, we will see the differences between the WUX and UX systems and the description of both functionalities with their different possibilities.

1.1 Differences between UX and WUX monitoring

Both monitoring systems are used for the execution of automated web browsing tasks through Pandora Web Robot Daemon (PWRD) system.

The UX system carried out these monitoring by means of the software agent installed in a machine, while the WUX monitoring is based on a server integrated inside Pandora FMS.

Pandora Web Robot Daemon (PWRD) is a service that provides the necessary tools to automate web browsing sessions. For it uses a file that contains a list of the necessary actions to navigate through the user's web portal.

Additionally, UX monitoring allows the execution of automated tasks that interact with the desktop and Windows system applications. This type of tasks cannot be performed with WUX.

2 Automated UX monitoring

2.1 Introduction

Pandora UX is an automated task-execution system that subsequently reports the results of executions and chronometries to Pandora FMS, and also includes screenshots of possible errors, to provide more intuitive, visually-oriented feedback.

Pandora UX employs two default technologies for automatizing tasks:

  • Pandora Web Robot (PWR): which permits web navigation and replicates user actions. It checks contents, texts, etc. and is ideal for monitoring sites and apps not based on flash. Pros: quicker, can run in the background.
  • Pandora Desktop Robot (PDR): replicates user-interactions with Windows. Pandora UX launches prepared scripts using this tool, allowing for more advanced and specific monitoring. E.g., plugins can be deployed on hardware and connect to distinct networks to check that what users would see is functioning correctly. Its principal use is to monitor heavy desktop apps.

As can be seen, there is a big difference between the two automation methods. While the PWR method is based on the automated navigation of a user through a website performing the tasks determined in the recording, the PDR method can interact with different operating system tools as if a user were running them. In short, one is based on web browsing, and the other is based on running different desktop applications.

If your system already has automated task-execution Pandora UX can still be used to execute pre-existing scripts and also to monitor their execution.

It is important to specify the use of this type of monitoring. Both methods are based on the execution of the monitoring by means of plugin in the software agents installed in the corresponding machines.

2.2 WEB UX (PWR)

2.2.1 PWR deployment

Pandora UX set-up.

2.2.1.1 Selenium server deployment on Windows

Prerequisites:

C:\PWR
C:\probes

To download the PWR_Server.zip go to: https://pandorafms.com/library/pwr-server-for-ux-monitoring/

Create file distribution:

  • File decompression PWR_Server.zip a C:\PWR\
  • Export Firefox profile to C:\PWR\profile

Using a profile to carry out checks is optional, but recommendable in case of using proxies or password autofill. To create a Firefox profile follow the steps in the following screenshots:

Ux1.JPG
Ux2.JPG

Choose directory address:

Ux3.JPG

Start Firefox with the new profile to customize options such as proxies, popups, etc.:

Ux4.JPG

Install the service by executing BAT file service_installer.bat provided. For a correct operation of the service_installer.bat we will have to modify the content, putting the paths that exist in the file as we have them installed in our machine. For example, in the case of java, it can only serve as java if we have correctly configured the PATH of it, otherwise we will have to give it all the PATH inside the file. Finally initiate the service:

net start PWRSRV

The Selenium server is now running. It can also be manually executed (after interrupting the service) in order to debug. Use the command:

java -jar selenium-server-standalone-2.53.0.jar -firefoxProfileTemplate C:\selenium\firefox_profile -port 4444 -v

2.2.1.2 Selenium server deployment on Linux

Prerequisites:


To install the dependencies from the repositories:

yum install xorg-x11-server-Xvfb
yum install java


If it's not in the repositories, the rpm file can be found at the following link: ftp://rpmfind.net/linux/centos/6.9/os/x86_64/Packages/xorg-x11-server-Xvfb-1.17.4-16.el6.centos.x86_64.rpm

To install the rpm package manually:

yum install xorg-x11-server-Xvfb-1.17.4-16.el6.centos.x86_64.rpm

Once the prerequisites are installed, we continue the installation of the install_pwrd.sh. The default PATH is /usr/share/pandora_server/util/pwrd/install_pwrd.sh and it's execution is:

/usr/share/pandora_server/util/pwrd/install_pwrd.sh --install

Once installed will have to start the service:

/etc/init.d/pwrd start

The Selenium server can be booted up with the script:

#!/bin/sh
# Monitoring selenium process
if grep selenium |grep -v grep |grep -v selenium.sh |wc -l `" != "0" ; then
   exit
else
   if grep Xvfb |grep -v grep | wc -l `" != "0" ; then
      Xvfb :99 -ac &
      export DISPLAY=:99
   fi
   export DISPLAY=:99
   java -jar /usr/share/pandora_server/util/pwdr/selenium-server-standalone-2.53.1.jar  &
fi

Or manually with the following commands:


$ Xvjb :99 -ac &
-> Press Enter to continue
$ export DISPLAY=:99
$ java -jar /usr/share/pandora_server/util/pwdr/selenium-server-standalone-2.53.1.jar  -port 4444 &

2.2.2 Pandora Web Robot Daemon (PWRD)

Pandora Web Robot Daemon is a service, which provides tools to automate web browser sessions, using a file containing a list of necessary actions in order to navigate the user's web portal.

It comes integrated with the Pandora server and we can find it in the folder /usr/share/pandora_server/util/pwrd when installing the Enterprise server (Linux) or in the module library (Windows)

It contains:

  • Firefox 43 Binary for Windows.
  • Pre-built profile to perform web browser sessions.
  • Server for session automation (Selenium server).

2.2.3 Recording a PWR session

Before UX monitoring can take place, you need to record a session. Depending on the type of tech you use, various recording options are available.

To record the navigation with PWR, you need the PWR recorder available at plugin module library:

https://pandorafms.com/library/pandora-ux-and-wux-pwr-recorder/

The package contains:


Start PWR_recorder.exe:


PWR recorder.PNG


The recorder will start automatically Firefox with a prebuilt environment:


PWR recorder main.png



After start Selenium IDE, we'll be able to record our web navigation test:

Ux10.JPG

Now you can navigate the website you want to monitor and register the different actions associated with each step on the recording. To stop recording use the following button, located on the upper-right of the screen:

Ux11.JPG

Once the actions are completed the page can be checked, e.g. verification of a text to ensure the correct page has loaded. Right click on a section of text in the browser window while recording and select verifyText:

Ux13.JPG

Another step will appear on the recording indicating the text to be checked:

Ux14.JPG

The entire sequence can be reproduced by clicking Play entire test suite and checking that it finishes correctly:

Ux15.JPG

Once the whole browsing sequence has been checked and verified save it (File -> Save Test Case) in order to execute it on Pandora UX. The resulting file is an HTML document that Pandora UX can then interpret.

2.2.3.1 Recording a transaction session with Pandora UX PWR

Pandora UX in PWR mode (Pandora Web Robot) allows browser monitoring to be divided among multiple modules, that represent each of the steps taken.

To insert a new control point and generate the phase modules (to this point) right-click on the point where the phase is to start.

Ux16.JPG

Use the following as the New Comment:

phase_start:name_of_phase

The phase includes the timings and results of all the commands up to the following comment:

phase_end:name_of_phase

All commands executed between a phase_start and a phase_end are included in that phase.

2.2.4 Executing PWR sessions

2.2.4.1 Standard execute

To launch prerecorded PWR sessions call the pandora_ux_x64.exe executable. It can be downloaded in: https://pandorafms.com/library/user-experience-pandora-ux/

Select PWR work mode and the file containing the session guidelines. Any error screenshots will be saved in the route indicated on the -folder parameter, optional. You can also user the parameter -retries to reexecute the script after a failure execution. The execution in Windows is:

pandora_ux_x64.exe -exe PWR -script tests\std.html -retries 3

The following modules are returned:

  • UX_Status_project_name: whether or not the sequence was successful
  • UX_Time_project_name: time taken to complete the sequence
  • UX_Snapshot_project_name: screenshot of the last point before any error occurred

Example of correct execution:

<module>
   <name><![CDATA[UX_Status_std.html]]></name>
   <type>generic_proc</type>
   <![CDATA[1]]>
   <description><![CDATA[Test OK]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
</module>
<module>
   <name><![CDATA[UX_Time_std.html]]></name>
   <type>generic_data</type>
   <![CDATA[16.317]]>
   <description><![CDATA[Test OK]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_std.html</module_parent>
</module>

Example of incorrect execution:

<module>
   <name><![CDATA[UX_Status_std.html]]></name>
   <type>generic_proc</type>
   <![CDATA[0]]>
   <description><![CDATA[Failed to execute verifyText]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
</module>
<module>
   <name><![CDATA[UX_Time_std.html]]></name>
   <type>generic_data</type>
   <![CDATA[15.463]]>
   <description><![CDATA[Failed to execute verifyText]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_std.html</module_parent>
</module>

<module>
   <name><![CDATA[UX_Snapshot_std.html]]></name>
   <type>async_string</type>
   <![CDATA[data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAA…JRU5ErkJggg==]]>
   <description><![CDATA[Image (last error)]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_std.html</module_parent>
</module>

Provided everything is OK the execution line can be added to the Pandora agent (already installed on the machine) which is going to carry out the checks. It should look something like this: In Windows:

module_plugin C:\Users\artica\Documents\Producto\UX-Trans\UX\pandora_ux_x64 -exe PWR -script C:\Users\artica\Documents\Producto\UX-Trans\PWR\sesion.html -folder C:\probes

In Linux:

module_plugin /usr/share/pandora_server/util/pwrd/firefox/pandora_ux.64 -exe PWR -script /usr/share/pandora_server/util/pwrd/firefox/PruebaUX.html -pwr_port 4444	

Once the agent is reporting to the Pandora FMS server the corresponding modules will appear. Enable hierarchy mode to see the relation among them:

Ux33.JPG

As previously indicated, the result of the execution (successful or not) can be seen on the UX_Status_session.html module, the time taken (in secs) on the UX_Time_session.html module , and another with the screenshot of the last error on UX_Snapshot_session.html which, in this case, is empty. Use these modules to create alerts and display the individual graphs from each module.

Imagen nueva ux.png

2.2.4.2 Transaction-based execution

If there is a Pandora UX PWR transaction recording as outlined in previous sections, it will be the system itself that generates the necessary modules to identify each of the phases indicated, which means the execution won't vary with respect to the previous case. Indicate the corresponding html file, which, in this case, contains the different phases. E.g. in Windows:

pandora_ux_x64 -exe PWR -script tests\std.html –folder C:\probes

The following modules will be returned:

  • UX_Time_project_name.phase_order
  • UX_Status_project_name.phase_order

If any phase returns an error, the following module is created:

  • UX_Snapshot_project_name.phase_order

A screenshot with the moment the error took place will be displayed.

Global status modules are returned with the following labels:

  • UX_Global_Time_project_name
  • UX_Global_Status_project_name
  • UX_Global_Snapshot_project_name

A screenshot showing the moment the error took place will be displayed.

The agent's execution line is the same to the previous example, but with an html file containing the phases. Something like this:

When the data reaches the Pandora FMS server it is displayed as modules. Enable hierarchy mode in the modules list to see the information clearly:

Ux34.JPG

Access the agent's UX section to see extra details on the transaction and its phases:

The following example represents a browsing process, plus the purchase of various articles, performed on our website, and divided into five phases to exactly measure the time taken and to locate areas that could be improved or where bottlenecks are being produced:

Ux44.JPG

2.2.4.3 Data retrieval

Pandora_UX is able to retrieve data from web content while testing the user experience test.

To use this feature, you must add some directives as comments in the user experience test recorder in Selenium IDE.


PWR recorder getValue.png


In the attached exampled, both values will be retrieved after test the navigation, and will be added to Pandora as modules (both getValue directives).

The stepts to make the plugin retrieve values are following:

  • Add a new comment to our navigation test in the Selenium IDE.
  • Fill the comment with the directive.

Data retrieval directive:

getValue;module name;module type;Regular expression for data capture

What does each field represented by a dot and comma mean?

  • getValue: UX system directive
  • Module name: module name as it will appear in Pandora
  • Module type: datatype to use, any from Pandora datatypes is allowed (generic_proc,generic_data,generic_data_string,async_proc,async_data,async_string...)
  • Perl regular expression: Must be a perl regex, specifing data to be retrieved between parenthesis

For example, in following text:

 <p>The temperature of the solar plate is: <span class="temperature">54º C</span></p>

If we want to retrieve the temperature of the solar plate to monitor it, we must specify the regex as:

<span class="temperature">(\d+\.*\,*\d*).*</span>

So, the system will retrieve the number "54" from the example, or any number with decimal positions.


Then, the complete directive will appear as:

getValue;Solar plate temperature;generic_data;<span class="temperature">(\d+\.*\,*\d*).*</span>

2.3 Desktop UX (PDR)

2.3.1 PDR deployment (Windows only)

The PDR system only works on Windows systems, and once run by the Pandora FMS agent it must run in process mode, as running in service mode will not work. Furthermore, it will not be possible to block the desktop session, so we recommend its use in virtual machines.

Prerequisites:

  • Windows OS in desktop mode, auto startup, and auto login.

To configure, execute:

control userpasswords2

Uncheck "Users must enter a user name and password to use this computer":

Ux5.JPG

To set up the environment create the following directories:

C:\PDR
C:\probes

Decompress the "PDR.zip" file from: https://pandorafms.com/library/pdr-cmd-for-ux-monitoring/ into C:\PDR.

To start the recorder, execute:

C:\PDR\pdr

2.3.2 Record a PDR session

Requirements:

  • Automation system (decompress pdr.zip in a directory of your choice) installed.
  • Start recording system (pdr.cmd).
Ux17.JPG

After loading, we will enter the recording environment:

Ux18.JPG

Select the actions and the zones where they are to be carried out. Here are some of the most common ones:

General actions:

Ux19.JPG

Flow control actions:

Ux20.JPG

More information about this scripting language: http://sikulix-2014.readthedocs.io/en/latest/index.html

When the project is recorded a file will be created with the following elements:

  • File .py with the script automation code.
  • Web browser control images.

A simple example of an execution would be something like monitoring to check that the Windows calculator works correctly. Let's take a look at how to go about it, one slide at a time.

1. Choose the action "click" and the area where we want to apply the action; the screen will change to "area selection" mode:

Ux21.JPG
Ux22.JPG

2. Enter the type action and the text "calc", wait for the calculator area to appear and click. Afterwards, we will wait for the calculator window to appear to enter the following actions:

Ux23.JPG

3. Then we will click on the corresponding buttons, selecting the areas as in the previous steps.

4. Lastly, save and replay clicking on Run.

Important notes:

  • If we double-click on any of the images within the recording environment, we can adjust details of the control capture, such as selecting the exact point where to click.
  • It is highly recommended to enter wait actions between each click to make sure that the execution will not be stopped due to a delay in the operating system.
  • The recorder will look for the area that looks like the screenshot, so we need to be especially careful if there is something highlighted when the button is placed on top of it (for example, the calculator buttons change to orange when the cursor is over them).

The image files can be viewed in the folder where the recording is saved, as well as the python file.

Ux24.JPG

Nota: You can customize the capture from the PDR recording engine by using the following code:

## OPTIONAL ##capture file names
import shutil
file = "C:\probes\screenshot_prueba.sikuli.png"
focusWindow = App.focusedWindow()
regionImage = capture(focusWindow)
shutil.move(regionImage, file)


This will create an image of the active window in the destination indicated by file with which we can make the screenshot at the specific time we want. We can customize this screenshot using coordinates, so we will specify the following syntax:

capture(x, y, w, h)

Where:

X: X position of the rectangle we want to capture

Y: Y position of the rectangle we want to capture

W: Width of the capture

H: Heigth of the capture



2.3.2.1 Recording a transaction session with Pandora UX PDR

Recording a specific session is not necessary, just record the sessions you need in different scripts. Pandora UX will return the results and organize them as a complex transaction. In the next point we will see how to make the call of the execution so that it works correctly.

If we are going to make several recordings to create a transactional process we must be careful when recording that the elements we are looking for are present. The manual execution of the recordings that we want to use in a single transactional process is recommended, followed by a manual execution to ensure that the actions take place as expected.

In the next example you can see a recording process that will constitute the second phase of the process from the example above. Take the result returned by the calculator and paste it into a note. It should look like this:

Ux40.JPG

Here's another sequence that consists of saving the text file in a specific location, overwriting the previous. This gives great flexibility, affording the possibility of parallel file monitoring, reflecting information from a variety of checks against heavyweight desktop applications. This third sequence consists of the following steps:

Ux43.JPG

2.3.3 Executing PDR sessions

2.3.3.1 Standard execution

To launch prerecorded PDR sessions, indicate work mode as file route pdr.cmd deployed in the corresponding point, the argument of said file being “-r”, the file that contains the session guidelines (-script), the directory where the screenshots will be saved (-folder) finishing ‘\’, optional. You can also user the parameter -retries to reexecute the script after a failure execution.

In the following execution the screenshot can be customized to capture only the active window:

pandora_ux_x64 -exe C:\PDR\pdr -args -r -script C:\pandora_ux\calculadora.sikuli -folder C:\pandora_ux\ -ss_config active -retries 3

The following modules will be returned:

  • UX_Time_project_name.
  • UX_Status_project_name.
  • UX_Control_Snapshot_project_name (only on first execute).

If any phase returns an error, the following module will be created:

  • UX_Snapshot_project_name.

An image from the active window will be shown (with -ss_config active) displaying the error when it occurred.

Correct output execution example:

<module>
   <name><![CDATA[UX_Status_calculadora.sikuli]]></name>
   <type>generic_proc</type>
   <![CDATA[1]]>
   <description><![CDATA[C:\pandora_ux\calculadora.sikuli execution completed Control snapshot rebuild ]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
</module>
<module>
   <name><![CDATA[UX_Time_calculadora.sikuli]]></name>
   <type>generic_data</type>
   <![CDATA[20.204]]>
   <description><![CDATA[C:\pandora_ux\calculadora.sikuli execution completed Control snapshot rebuilt ]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_calculadora.sikuli</module_parent>
</module>
<module>
   <name><![CDATA[UX_Control_Snapshot_calculadora.sikuli]]></name>
   <type>async_string</type>
   <![CDATA[data:image/png;base64, IBCAIAAAAOCnfhAAAAAXNSR…/4x79e/7757f8H2C00s1C73yMAAAAASUVORK5CYII=]]>
   <description><![CDATA[Control image rebuilt]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_calculadora.sikuli</module_parent>
</module>

Incorrect iutput execution example:

<module>
   <name><![CDATA[UX_Status_std.html]]></name>
   <type>generic_proc</type>
   <![CDATA[0]]>
   <description><![CDATA[Failed to execute verifyText]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
</module>
<module>
   <name><![CDATA[UX_Time_std.html]]></name>
   <type>generic_data</type>
   <![CDATA[15.463]]>
   <description><![CDATA[Failed to execute verifyText]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_std.html</module_parent>
</module>
<module>
   <name><![CDATA[UX_Snapshot_std.html]]></name>
   <type>async_string</type>
   <![CDATA[data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAA…JRU5ErkJggg==]]>
   <description><![CDATA[Image (last error)]]></description>
   <tags>UX</tags>
   <module_group>UX</module_group>
   <module_parent>UX_Status_std.html</module_parent>
</module>

If everything is A-OK add the execution line for the Pandora FMS agent installed on the Windows device as a plugin. The line looks like this (one line):

module_plugin C:\Users\artica\Documents\Producto\UX-Trans\ux\pandora_ux_x64.exe -exe C:\PDR\pdr.cmd -args -r -script C:\PDR\calc.sikuli -folder C:\PDR\ -ss_config active -checkpoint -post "taskkill /F /IM calc.exe"

As can be seen, the execution has more options. The parameter -checkpoint displays a screenshot of the final result, even if there are no errors to display, -post will execute actions once the session replay has finished, in which case close the calculation process initiated by the recording to avoid a system collapse due to an excess of open windows, or any similar problem.

Now that the agent has the execution line module_plugin ready launch it in process mode, executing the following from the admin command line:

"C:\Program Files\pandora_agent\PandoraAgent.exe" --process

Obviously, the complete route to the executable PandoraAgent.exe should correspond to your installation.

When launched you can see how actions are executed automatically according to the recording. From now on, the agent process must not be closed or a session initiated on the machine from a remote desktop or the executions may be interrupted. The machine should remain untouched, this being the reason for which virtual machines are recommended.

Imagen nueva pdr.png

Info.png

If you have other automatic monitoring tests, you can include them in Pandora FMS with the following execution.

 


pandora_ux.64 -exe <exe of the aumotatic monitoring system> -args <system parameters> -script <file PAHT of the test>

2.3.3.2 Transaction-based execution

Once there are various processes recorded, and verified for correct functioning, with the PDR, execute the following:

C:\Users\artica\Documents\Producto\UX-Trans\ux\pandora_ux_x64.exe -exe C:\PDR\pdr.cmd -args -r -t calculadora_trans -script C:\PDR\calc.sikuli,C:\PDR\savecalc.sikuli,C:\PDR\savefile.sikuli -folder C:\PDR\ -ss_config active

As can be seen, you only need to indicate the new script's route in the -script parameter, separated by a comma from the previous script, and employ the -t parameter with the name of the transaction which covers all the different phases. If the process contains more phases simply apply the same logic, e.g.:

pandora_ux_x64.exe -exe C:\PDR\pdr.cmd -args -r -t proceso_transaccional -script C:\PDR\script1,C:\PDR\script2,C:\PDR\script3,C:\PDR\script4 -folder C:\PDR\ -ss_config active

Add the following line to the agent configuration file:

module_plugin C:\Users\artica\Documents\Producto\UX-Trans\ux\pandora_ux_x64.exe -exe C:\PDR\pdr.cmd -args -r -t calculadora_trans -script C:\PDR\calc.sikuli,C:\PDR\savecalc.sikuli,C:\PDR\savefile.sikuli -folder C:\PDR\ -ss_config active -checkpoint -post "taskkill /F /IM calc.exe"

Thanks to the -checkpoint parameter screenshots of the final result from each phase can be viewed on the Pandora console.

The following modules will be returned, per phase:

  • UX_Time_project_name.phase_order.
  • UX_Status_project_name.phase_order.

If any of the phases returns an error, the following module will be created:

  • UX_Snapshot_project_name.phase_order.

In case of error it will also display a web image with the moment the error was produced.

It also returns global summary modules with the following IDs:

  • UX_Global_Time_project_name.
  • UX_Global_Status_project_name.
  • UX_Global_Snapshot_project_name.


When the agent is running in process mode and the modules are recovered by Pandora FMS they can be seen on the console. Once again, via hierarchy mode view, the relation among the modules, and the boundaries between phases, can be clearly seen:

Ux41.JPG

Details of the phases and time charts can be viewed in the transaction display:

Ux42.JPG

Error screenshots are only shown when the UX client (PWR) and the PWR server are both running on the same machine. Otherwise, the PWR server image delivery directory should be accessible for the client in order to view the image on Pandora.

Using these same modules you can create alerts and view the history in order to generate graphs and display the evolution over time of your systems.

It is recommendable to create a programmed task that boots the Pandora agent in process mode when the system boots up, to avoid interrupting executions, even when the system has been force-booted, and to ensure the agent executes even though the machine is on automatic login mode, without password protection.

Ux26.JPG
Ux27.JPG
Ux28.JPG
Ux29.JPG
Ux30.JPG
Ux31.JPG
Ux32.JPG

This guarantees that on the Windows instance in question the Pandora FMS agent will always be running in process mode even when the machine restarts, being always able to send the data harvested by the PDR probe.

Systems with various desktops can present problems, in which case it's recommendable to always use the configuration previously described, on any machine in desktop mode and autostart and a single desktop.

3 WEB User experience transactional monitoring (WUX)

3.1 Introduction

Pandora WUX is a Pandora FMS internal component, which allows users to automate web browser interaction.

In order to work, WUXServer must be connected to a Pandora Web Robot Daemon (PWRD) server.

PWRD is a service that executes step by step pre-recorded actions by the user in a web portal.

To achieve this, PWRD uses Selenium web automation technology.

It is therefore imperative that the user shows Pandora WUXServer the actions that must be performed to navigate through the web portal.

With this data, Pandora WUXServer will generate a report with the results of the performances, time spent, and screenshots with the possible errors found.

Pandora WUXServer is able to split the browsing sessions into phases, which simplifies the view and helps us to debug possible bottlenecks.



Wux simple.png


3.2 Configuration

In order to enable WUXServer on our Pandora FMS server,we will need to configure the Pandora Web Robot service (PWRD)

3.2.1 Pandora Web Robot Daemon (PWRD)

Pandora Web Robot Daemon is a service, which provides tools to automate web browser sessions, using a file containing a list of necessary actions in order to navigate the user's web portal.

It comes integrated with the Pandora server and we can find it in the folder /usr/share/pandora_server/util/pwrd when installing the Enterprise server (Linux) or in the module library (Windows)

It contains:

  • Firefox 43 Binary for Windows.
  • Pre-built profile to perform web browser sessions.
  • Server for session automation (Selenium server).


3.2.2 Installation on Linux

The Selenium component of the PWRD daemon requires java in order to work, we will need to install it:

 yum install java 

In order for PWRD to be able to launch Firefox on our Linux server, it will be necessary to install xorg-x11-server-Xvfb, gtk2 and gtk3 to be able to virtualize graphical environments.

yum install xorg-x11-server-Xvfb gtk2 gtk3

If its not available in your repos, you can find the RPMs files in following links: ftp://rpmfind.net/linux/centos/6.6/os/x86_64/Packages/xorg-x11-server-Xvfb-1.15.0-22.el6.centos.x86_64.rpm

ftp://rpmfind.net/linux/centos/7.4.1708/os/x86_64/Packages/gtk2-2.24.31-1.el7.x86_64.rpm

To do the RPMs installation manually:

yum install xorg-x11-server-Xvfb-1.15.0-22.el6.centos.x86_64.rpm
yum install gtk2-2.24.31-1.el7.x86_64.rpm

Then, proceed to install PWRD daemon, which can be found in the folder /usr/share/pandora_server/util/pwrd/ when installing the Enterprise server.

sudo /bin/bash /usr/share/pandora_server/util/pwrd/install_pwrd.sh --install [user] [directory]


In version 730 and later, the possibility of performing the customized installation with a user and a directory different from the default installation has been introduced.

Once installed, we must start it.

PWRD provides several working modes:

  • Standalone: This is the standard mode; it will launch a single PWRD instance, each will be assigned to a Pandora FMS server.
  • HUB: Concentrator mode. In this mode, PWRD service will not evaluate browser sessions directly; instead, "nodes" will have to be registered to execute tasks. This is the cluster mode from the PWRD service. Each HUB will be assigned to a Pandora FMS server.


3.2.2.1 PWRD in "Standalone" mode

PWRD in standalone mode, will start the daemon and will prepare it to execute the actions indicated by the user through WUXServer.


Wux pwrd standalone.png


# Start 
/etc/init.d/pwrd start
#Get status
/etc/init.d/pwrd status
# Stop it
/etc/init.d/pwrd stop


3.2.2.2 PWRD in "HUB" mode

Concentrator mode or HUB will start the daemon as a load balancer. In this working mode, the system will balance the load between all nodes registered in the system, assigning the execution of the navigation sessions, depending on their workload.

You can always check the status of the HUB by accessing the HUB administration console:

http://hub:4444/grid/console


Wux pwrd hub.png


# Start
/etc/init.d/pwrd start-hub
# Get status
/etc/init.d/pwrd status-hub
# Stop it
/etc/init.d/pwrd stop-hub


3.2.2.2.1 Add PWRD nodes to our HUB

To add a new PWRD node, you will need:

  • A hub (PWRD in HUB mode)
  • PWRD files, on the same or different machine
  • TCP/4444 connectivity from the computer that hosts the node to the computer that hosts the HUB.

In this work mode, the service will process all queued requests from the hub (HUB), returning the results of the executions to it. It will be the HUB who will speak exclusively with the WUXServer, being transparent to the one who is executing the actions of the user.


 # Start and register in the HUB, replace "hub" with the server IP PWRD HUB
 /etc/init.d/pwrd start-node http://hub:4444/grid/register
Check status
/etc/init.d/pwrd status-node
Stop it
/etc/init.d/pwrd stop-node

3.2.2.3 Pandora Server Configuration

Regardless of the chosen mode, once started, we can start assigning executions of our browsing sessions, adding the configuration parameters of WUXServer to the configuration file of our Pandora server.

Assuming that we have deployed our PWRD server on the same server that runs our Pandora FMS server, we would add the following configuration:

# add to /etc/pandora/pandora_server.conf
wuxserver 1
wux_host 127.0.0.1
wux_port 4444

3.2.3 Installation on Windows

Steps to prepare a working environment for Pandora WUX on Windows.

Nota: We strongly recommend Linux to host the PWRD service.

You can get the files needed for your deployment of the module library in https://library.pandorafms.com/index.php?sec=Library&sec2=repository&lng=en&action=view_PUI&id_PUI=816

3.2.3.1 PWRD deployment on Windows systems

Prerequisites:


We will create the following file distribution:

Using a profile for the checks is not essential, however it is recommended especially when using proxy or when using autocomplete passwords. To create a profile of Firefox follow the steps in the following screenshots:


Ux1.JPG


Ux2.JPG


Choose the destination directory:


Ux3.JPG


We must start Firefox with the new profile to customize options such as proxy, shown popups, etc:


Ux4.JPG


Then we install the service by executing the BAT file service_installer.bat provided and we will start the service:

net start PWRSRV

From this moment you have the Selenium server running on your machine. However, we can run it manually (by previously stopping the service) if we want to carry out debugging tasks, using the following command:

java -jar selenium-server-standalone-2.53.1.jar -firefoxProfileTemplate C:\selenium\firefox_profile -port 4444 -v

3.3 To record a PWR session

Before monitoring a user experience we must record the actions we want the system to perform, in order to browse through our web portal

To do this we have available the "PWR recorder" utility in the module library.

Run the downloaded file to start the recording environment. Once the environment is started, you can access the recorder by clicking on the following button:


Iniciar grabador.png



You should see a window like the one shown in the image:



Ux10.JPG



From this moment we will be able to browse the website that we want to monitor and the different actions of each step that we advance will appear in the recorder.

To stop recording we will use the following button, located at the top right of the recorder:



Ux11.JPG



After completing the actions, we can perform checks on the page, for example verify the existence of a certain text to ensure that the page loaded is correct. To do this we will right click on a section of text in the browser window while we continue to record, and select the option verifyText:


Ux13.JPG


A new step will appear in the recorder indicating the said text checking action:


Ux14.JPG



We can reproduce the entire sequence using the button Play Entire test suite and check that successfully completes:


Ux15.JPG



After verifying the validity of the sequence of navigation, save it (File -> Save Test Case).This will be the file that we will load when creating a new web analysis module in our Pandora console.


3.3.1 Record a transactional session for Pandora WUX

Pandora WUX allows you to split browsing monitoring of a website into multiple modules, representing each of the steps performed.

To insert a new control point and generate the phase modules (up until that point) right-click on the point where you want to identify the start phase.


Pwrd recorder transaction.PNG


We will write the next text as a comment:

phase_start:name_of_phase

The Phase will encompass the time and result of all commands that exist up until the following comment:

phase_end:name_of_phase

All commands that are run between a phase_start y phase_end label will be covered in this phase.


Pwrd recorder transaction2.PNG


3.4 Creating a Web Analytics module in Pandora FMS Console

Once we have our browsing session recorded, it's time to deploy it as a module in Pandora FMS console.

To do so , we will access to any agent associated with a server with WUX feature enabled and we will create a module "Web analysis":

Wux module.PNG



After selecting "create" we must fill in all fields showed in the form:

Wux new module form2.PNG
  • Module Name: Module WUX (Father), all sub-modules with monitoring results will be dependent on this new module.
  • Target Web site: For monitoring performance statistics of website.
  • Run performance test: Indicates that we want to monitor not only the browsing experience, but also the performance statistics of access to the target web site.
  • Historical data: Saving or not saving the historical data modules that monitor that user experience.
  • textarea section: in this text box we must copy (or upload) content files of the browsing session we have previously recorded.


3.4.1 Custom Macros

The operation of custom macros involves the substitution of certain text strings that are present in our file of browsing session with certain customizable values.

In the case of web analytics modules, this feature has been improved, allowing dynamic macros which allow these values to be translated into dates and times variables.

Why this feature then?

Let’s suppose that we need to monitor, through a browsing session, the proper functioning of a web application that focus on booking meeting rooms.

If we establish a date and time fixed, the system would probably cancel the reservation because at some point it might stay in the past , for example could be trying to book a room for last week.

We may also find a maximum of time to carry out that reservation, and that the system requires us to book the room within a certain period of time, for example, not exceeding the current month.

To avoid editing the macro every few days, and worrying about the configuration section, you can use the dynamic macros, telling the system to always book the room for the next day when doing the test.


To use this feature, the values must have a specific format, making possible the following substitutions:

@DATE_FORMAT (current date / time with user-defined format)
@DATE_FORMAT_nh (hours)
@DATE_FORMAT_nm (minutes)
@DATE_FORMAT_nd (days)
@DATE_FORMAT_ns (seconds)
@DATE_FORMAT_nM (month)
@DATE_FORMAT_nY (years)


"n" may be an unsigned number (positive) or negative.

And FORMAT follows the standard strftime de perl


Examples:

@DATE_%Y-%m-%d %H:%M:%S
@DATE_%H:%M:%S_300s
@DATE_%H:%M:%S_-1h

3.5 Uploading certificates to Firefox profile

It is possible that some of the configured checks will run against web pages that use self-signed certificates or signed by a CA not included in the Firefox default accepted CA, so in such cases it will be necessary to load the certificate in question into the Firefox profile being used.

To do this, the easiest way is to launch the browser in a graphical environment, access the URL and add the SSL certificate. Next we will explain how to do it both in Windows and Linux:

3.5.1 With PWRD deployed on Windows systems

In this case, since we will have a graphical environment, we only have to start the Firefox browser with the profile we are using for checks:

Profile 01.png
Profile 02.png

Once started, access the URL with the certificate you want to upload and add it as an exception for the browser:

Profile 03.png
Profile 04.png

Another possibility, if you want to accept any SSL certificate, would be to access the Firefox options, the "Privacy & Security" tab, and uncheck the field "Query OCSP responder servers to confirm the current validity of certificates":

Profile 05.png
Profile 06.png

3.5.2 With PWRD deployed on Linux systems

In general, when a Linux server is installed, it does not include a graphical desktop environment, so in order to follow the same steps as in the previous case we have to make a redirection of the graphical X to a computer where we do have a graphical desktop environment. This is done differently if we redirect the X to a graphical desktop in Linux or Windows.

3.5.2.1 Redirecting X11 to a Linux desktop

Doing this is very simple, since it will be enough to establish an SSH connection to the PWRD server with the "-X" parameter:

ssh -X [email protected]_ip_address

Having done this, if we launch the Firefox browser, we can see how it shows on our desktop:

firefox -p

In a default installation we will only see the default profile, so it would be advisable to do as in the Windows installation and create a new profile to use:

Profile 07.png
Profile 08.png
Profile 09.png

If you save the profile in a path other than /opt/firefox_profile, you will need to edit the PWRD start script /etc/init.d/pwrd to indicate the path to the new profile:

PWROPTS=" -firefoxProfileTemplate /path/to/profile/folder"

With the profile created, you can start the browser:

Profile 10.png

Once started, access the URL with the certificate you want to upload and add it as an exception for the browser:

Profile 11.png
Profile 12.png

Another possibility, if you want to accept any SSL certificate, would be to access the Firefox options, in the "Advanced > Certificates" tab, and deselect the field "Query the OCSP respondent servers to confirm the current validity of the certificates":

Profile 13.png
Profile 14.png

3.5.2.2 Redirecting X11 to a desktop in Windows

In the case of Windows, first we will have to install an X server to be able to do the redirection, as for example Xming. Once installed and started, we will be able to do the X redirection.

Using the SSH Putty client, we will have to go to the section "Connection > SSH > X11" before making the connection, and make sure that we mark the option "Enable X11 forwarding" as well as fill the field "X display location" as "localhost:0":

Profile 15.png

Then we go back to the "Session" section and establish the connection:

Profile 16.png

Once the connection is established, if we launch the Firefox browser on the PWRD server, we will be able to visualize it on the Windows desktop.

firefox -p

In a default installation we will only see the default profile, so it would be advisable to do as in the Windows installation and create a new profile to use:

Profile 17.png
Profile 18.png
Profile 19.png

If you save the profile in a path other than /opt/firefox_profile, you will need to edit the PWRD startup script /etc/init.d/pwrd to indicate the path to the new profile:

PWROPTS=" -firefoxProfileTemplate /path/to/profile/folder"

With the profile created, you can start the browser:

Profile 20.png

Once started, access the URL with the certificate you want to upload and add it as an exception for the browser:

Profile 21.png
Profile 22.png

Another possibility, if you want to accept any SSL certificate, would be to access the Firefox options, in the "Advanced > Certificates" tab, and deselect the field "Query the OCSP respondent servers to confirm the current validity of the certificates":

Profile 23.png
Profile 24.png

3.6 Data viewing

The information generated by WUX is displayed in modules as follows. Enabling view hierarchically in the list of modules, will help us to display the information much more clearly:


WUX modules.PNG



Within this hierarchy we will find the following modules:

  • module_Global_Status: Indicates the global state of complete browsing.
  • module_Global_Time: Indicates the global time lapsed of complete browsing.
  • module_Global_Screenshot: contains an image with the result of browsing error, it will be generated only in case of error
  • module_Phase X: Name of phase_Status: Indicates the status of the browsing in Phase X
  • module_Phase X: Name of phase_Time: Indicates time lapsed in Phase X

Example: Error capture view.


Wux screenshot.png


Template warning.png

Attention: If you upgraded from versions prior to Pandora FMS 7.0NG 712 you must make a small change.

 


In order to store the screenshots generated by WUXServer, you must execute the following operations on your database schema:

alter table tagente_estado modify column datos mediumtext;
alter table tagente_datos_string modify column datos mediumtext;

If you don't execute this actions, your screenshots may be incomplete or cropped

Accessing the WUX section Agent , we can view additional details of the transaction and its phases:

Wux nueva imagen.png


Website statistics are summarized in the following concepts:

  • Stats_TT: Total time to obtain the website.
  • Stats_TDNS: Total time to resolve the IP address of the target.
  • Stats_TTCP: Time spent on connecting via TCP..
  • Stats_TSSL: Time spent on establishing SSL communication.
  • Stats_TST : Time lapsed until data transfer started.
  • Stats_TTC : Time spent transferring data, It will gather all the transfer time of the resources.
  • Stats_TTR : Time used to transfer the content of the page.
  • Stats_TTR_main: Time lapsed to transfer the HTML code.
  • Stats_TTR_image: Time used to transfer resources of type of image (png|jpg|jpeg|bmp|tiff|gif|webp|svg).
  • Stats_TTR_css: Time spent on transferring style sheets.
  • Stats_TTR_js: Time spent on transferring the JavaScript files.

3.7 Assign alerts to a web analysis module

Alerts associated with Web analysis modules follow the same dynamics of operating as the entire system of alerts in Pandora FMS.

For compatibility, it is recommended to assign templates of alerts on self-generated sub-elements by the analytics web module, such as:

  • The state of the global browisng
  • Alerts on thresholds times
  • Emails warning template set as "always" for screenshots modules of results


3.8 New widgets for Dashboards

To be able to display user browsing information, two new widget types have been added:

Widget for status of browsing phases

Widget wux fases.PNG



Widget for website statistics

Widget wux stats.PNG