Pandora:Documentation en:Netflow

From Pandora FMS Wiki

Jump to: navigation, search

Go back to Pandora FMS documentation index

NetFlow

Introduction

The Pandora FMS versions 5 and above are designed to monitor the IP traffic by using the NetFlow protocol. This protocol allows to you review the traffic's most useful patterns and general data.

'NetFlow' is a network protocol, developed by Cisco Systems to collect IP traffic information. It has become an industrial standard for network traffic monitoring and is currently supported by several platforms besides Cisco's IOS and NXOS like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.



Netflow architecture.png



NetFlow

NetFlow-capable devices (NetFlow probes) are generating NetFlow records, which consist of small chunks of information which are sent to a central device or NetFlow Server (or NetFlow collector), which stores and processes that information.

Data is transmitted using the NetFlow protocol via UDP or SCTP protocols. A NetFlow record is a small packet which only contains statistical information about a connection, not the whole raw data or the payload.

There are several NetFlow implementations that may differ from the original specification and include additional information, but most of them provide at least the following:

  • The source's IP address.
  • The target's IP address.
  • The source's UDP or TCP port.
  • The target's UDP or TCP port.
  • The IP protocol.
  • An interface (SNMP ifIndex)
  • The type of service.

With time, some manufacturers have designed similar protocols with different names but for the same purpose:

  • 'Jflow' or 'cflowd' from Juniper Networks
  • 'NetStream' from 3Com/H3C/HP
  • 'NetStream' from Huawei
  • 'Cflowd' from Alcatel Lucent
  • 'Rflow' from Ericsson
  • 'AppFlow'

Pandora FMS also supports sFlow( Industry standard for packet export ), which allows to Pandora FMS to analyse sniffered packets at Layer 2 of the OSI model. Moreover, because sFlow is an standard, many vendors use it on their devices.

The NetFlow Collector

A NetFlow collector is a device (a PC or a Server), placed in a network to gather all the NetFlow information which is sent by routers and switches.

A NetFlow Server is required to receive and store that information. Pandora FMS uses 'nfcapd' for this purpose, and it's required to be installed before Pandora FMS is able to process any NetFlow-related data. Pandora FMS starts and stops this server automatically in the moment the need arises.

The NetFlow Probe

Probes are usually NetFlow-capable routers, configured to send NetFlow data to its collector - in our case, a Pandora FMS server with 'nfcapd' running.



NewNetFlowApproach.png



Installation and Requirements

Pandora FMS uses an open-source tool called 'nfcapd' to process all NetFlow traffic. This daemon is automatically started by the Pandora FMS Server. This system stores the data in binary files at a specific location. You're required to install 'nfcapd' on your system before working with NetFlow. 'nfcapd' listens on port 9995 UDP by default. Please keep in mind to open port 9995 UDP in case you have firewalls in place.

Installation of 'nfcapd'

You're required to install 'nfcapd' manually, because Pandora FMS is not going to install it by default. For more information on how to install it, please visit the
Official NFCAPD Project Page.

Pandora FMS uses the directory '/var/spool/pandora/data_in/netflow' by default to store all NetFlow data. The 'nfcapd' daemon is going to point to this directory when it's getting started by the Pandora FMS Server. Do not change it unless you know exactly what you're doing.

Pandora FMS requires the nfdump version 1.6.8p1 in order to process any NetFlow data properly.

In order to test your 'nfcapd' installation manually, please execute the command below.

nfcapd -l /var/spool/pandora/data_in/netflow -D

Please keep in mind that the Pandora FMS Console (and more specifically the web server which hosts it) requires access to the directory of '/var/spool/pandora/data_in/netflow' in order to read any NetFlow-related data files.

The NetFlow Probe Installation

If a NetFlow capable router is not available, but you use a Linux server to route your traffic, you may install a NetFlow software probe which sends all NetFlow-related information to its server.

In Linux there is a program called 'fprobe' which obtains the traffic and sends it to a NetFlow Server. By this program you're able to generate NetFlow traffic which goes through its interfaces, e.g.:

/usr/sbin/fprobe -ieth0 -fip 192.168.70.185:9995

Once the traffic has been generated, you're able to review the traffic's statistics by entering the following command:

nfdump -R /home/netflow_data/

The above mentioned command displays information similar to the one shown below.


Aggregated flows 1286
Top 10 flows ordered by packets:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:41:35.697   901.035 TCP     192.168.60.181:50935 ->     192.168.50.2:22        2105   167388     4
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:15.015     1.389 TCP      192.168.50.15:40044 ->     157.88.36.34:80         363    22496     1
2011-12-22 20:46:02.791    76.616 TCP     192.168.60.181:40500 ->    192.168.50.15:80         303    24309     1
2011-12-22 20:48:14.689     1.843 TCP      192.168.50.15:60101 ->   91.121.124.139:80         255    13083     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1  

Top 10 flows ordered by bytes:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1
2011-12-22 20:48:15.313     1.603 TCP       89.102.0.150:80    ->    192.168.50.15:52019      212   313432     1
2011-12-22 20:48:14.996     1.433 TCP     212.219.56.138:80    ->    192.168.50.15:36940      191   281104     1
2011-12-22 20:51:12.325    46.928 TCP      192.168.50.15:80    ->   192.168.60.181:40512      201   245118     1
2011-12-22 20:52:05.935    34.781 TCP      192.168.50.15:80    ->   192.168.60.181:40524      167   211608     1
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4 

Summary: total flows: 1458, total bytes: 5.9 M, total packets: 15421, avg bps: 49574, avg pps: 15, avg bpp: 399
Time window: 2011-12-22 20:40:46 - 2011-12-22 20:57:21
Total flows processed: 1458, Records skipped: 0, Bytes read: 75864
Sys: 0.006s flows/second: 208345.2   Wall: 0.006s flows/second: 221177.2  


If your system works properly, the following chapter is intended to configure Pandora FMS in order to use this particular configuration appropriately.

Working with NetFlow under Pandora FMS

Pandora FMS doesn't store NetFlow data in its database. The information is processed on demand in order to render reports.

Pandora FMS works with NetFlow data by using filters, which are sets of rules that match certain traffic patterns. A rule can be as simple as 'all the traffic from the 192.168.70.0/24 subnet' or a complex 'pcap' filter expression.

Once the filters are created, we're required to define reports that determine how the information matched by those filters is going to be displayed (e.g. charts and tables) and the time frame. The NetFlow reports can be accessed on demand like any other Pandora FMS reports.

There is also a live NetFlow Viewer to analyze the traffic, modify and create rules on the spot. It can be very useful to investigate problems or temporarily display a chart that we don't intend to save for a later usage.

Configuration

First of all, you're required to authorize NetFlow in order to become accessible from the 'Operation' and 'Administration' menus.



Netflow manager0.png



You can find the NetFlow option in the 'Configuration' chapter of the 'Administration' menu in which we specify the path in which the files of the Netflow traffic are captured, e.g. '/tmp/netflow'. It's also very important to determine whether the path to the 'nfcapd' daemon is appropriately specified or not.



Netflow manager.png



The configurable fields pertaining to this particular feature are the following:


Data Storage Path:
The directory in which the NetFlow data files are stored. IMPORTANT: The disk's access speed on which the NetFlow data is stored is usually the limiting performance factor.

Daemon Interval:
The time interval in seconds for the data rotation. The recommended value is '3600'. A bigger interval means potentially bigger files, which means less I/O overhead, but it also renders accessing the data for a specific time interval slower.

Daemon Binary Path:
The path to the 'nfcapd' binary.

Nfdump Binary Path:
The path to the 'nfdump' binary.

Nfexpire Binary Path:
The path to the 'nfexpire' binary. This program was designed to delete old NetFlow data.

Maximum Chart Resolution:
The maximum number of points which a NetFlow Area Chart is going to display. The higher the resolution the lower the performance. Values between '50' and '100' are recommended here.

Disable Live View Custom Filters:
If enabled, only Netflow filters previously created by an administrator can be used in the Netflow live view.

Netflow max. Lifetime:
The NetFlow data which are older than the specified number of days are going to be deleted.

Once the NetFlow configuration is enabled, the Pandora FMS Server is required to be restarted in order to be able to start the 'nfcapd' server. This server must be properly installed and accessible from the system path. Please check the server logs if you're unsure on that. This server is not going to appear in the Pandora FMS server view mode, because it isn't considered a Pandora FMS Server.

Filters

You may access the creation and edition of filters by clicking on 'Administration' and 'NetFlow Filters'. This section contains a list of already created filters which can be of course altered or deleted.

The configurable NetFlow filters pertaining to this particular feature are the following:

  • Name: It's recommended for the filter's name to be as descriptive and clear as necessary.
  • Group: A user is only able to create a filter or edit the group's filters it has access to.
  • Filter: There are two types of filters: Basic and advanced. Advanced filters allow the usage of advanced expressions in the same format as 'nfdump'. Basic filters can filter traffic by source and destination IP and source or destination port. Lists of comma-separated IPs or ports are also accepted here.
  • Aggregate by: All traffic data can be grouped by one of the following fields:

IP Origin: It displays the traffic of different origin for each IP.
IP Destination: It displays the traffic of different destinations for each IP.
Origin Port: It displays the traffic for each port of different origins.
Destiny Port: It displays the traffic for different destinations for each port.
Protocol: It displays the traffic for each protocol.
Any: The total data is going to be displayed by this one.

Output Format: The data is going to be displayed in the selected unit:

Kilobytes.
Kilobytes per second.
Megabytes.
Megabytes per second.

Basic web traffic filter example:



Netflow filter normal.png



Advanced intranet traffic filter example:



Netflow filter advanced.png



Here are other examples of advanced filters:

  • Capture traffic to or from 192.168.0.1:
host 192.168.0.1
  • Capture traffic to 192.168.0.1:
dst host 192.168.0.1
  • Capture traffic from 192.168.0.0/24:
src net 192.168.0.0/24
  • Capture HTTP and HTTPS traffic:
(port 80) or (port 443)
  • Capture all traffic except DNS:
port not 53
  • Capture SSH traffic to 192.168.0.1:
(port 22) and (dst host 192.168.0.1)

Reports

Netflow reports are integrated with Pandora FMS reports (see Reports for more information).

To create a report item, choose one of the available netflow report items.

Netflow report item types.png



And configure it. The following options are available:



Netflow report item configuration.png



  • Type: Item types will be explained below.
  • Filter: Netflow filter to use.
  • Description: Item description.
  • Period: Length of the interval of data to display.
  • Resolution: Data will be retrieved in blocks of size equal to the resolution. If Period / Resolution is bigger than the configure maximum chart resolution the resolution will be dynamically readjusted. For example, for a period of 1 day and a resolution of 1 hour 24 points will be drawn in the chart.
  • Max. values: Maximum number of elements for aggregates. For example, if a chart of HTTP traffic is drawn aggregated by source IP address and Max. values is set to 5, only 5 IP addresses will be shown.

There are five types of netflow report items:

  • Area chart: An area chart, either aggregated or unaggregated.



Netflow chart area aggregated.png



  • Pie chart: An aggregated pie chart.



Netflow chart pie.png



  • Data table: A text representation of the area chart.



Netflow table data.png



  • Statistics table: A text representation of the pie chart.



Netflow table statistics.png



  • Summary table: Traffic summary for the given period.



Netflow summary.png



Netflow live view

Filters can be visualized live from "Operation / Netflow Live View". This tool allows you to preview changes made to a filter and save it when the desired result is achieved. It is also possible to load and modify already existing filters.

See Reports and Filters to learn how to configure live view options.



Netflow live view.png



To modify an existing filter load if from the Load filter selector, make the desired changes and click on Update current filter.



Netflow update filter.png



To create a new filter, configure it, click on Save as new filter, enter a name and optionally select a group and click on Save as new filter again.



Netflow save filter.png



Go back to Pandora FMS documentation index