ntophttps://www.ntop.org
High Performance Network Monitoring Solutions based on Open Source and Commodity Hardware.Wed, 13 Dec 2017 15:57:43 +0000en-UShourly1https://wordpress.org/?v=4.9.126105464PF_RING and Network Namespaceshttps://www.ntop.org/guides/pf_ring-and-network-namespaces/
Wed, 13 Dec 2017 15:57:43 +0000https://www.ntop.org/?p=5975Continue reading →]]>Last week we made a couple of presentations at LinuxLab 2017 where we spoke about Containers, focusing on Network Namespaces support in PF_RING, and User and IoT-oriented Network Traffic Monitoring on Embedded Devices.

With the advent of Containers, processes isolation has become extremely easy and effective, to the point that ordinary Virtual Machines have been reconsidered. Many ntop users today are running traffic monitoring applications in Docker, thus it’s important to understand how Containers work and how to make the best use of them. Network isolation is provided by Network Namespaces, a native feature of the Linux kernel, that virtualize the network stack. With this talk we have seen what exactly happens under the hood of Network Namespaces, focusing on raw packet capture, and we learnt that even when we are not running containers in Linux, we are running in a Namespace.

Those who have not been able to attend this event can find our presentation slides below.

]]>5975Announcing ntopng 3.2 – The First Move Towards Active Network Monitoringhttps://www.ntop.org/announce/announcing-ntopng-3-2-the-first-move-towards-active-network-monitoring/
Mon, 04 Dec 2017 08:57:50 +0000https://www.ntop.org/?p=5945Continue reading →]]>Today we are glad to announce the new 3.2 stable release of ntopng. Among the most important new features available in this release, there is without any doubt an advanced network devices discovery functionality. Historically, ntopng has always been a fully passive monitoring tool. This release aims at complementing the information gathered from a purely passive packet capture with precious extra bits of data obtained by actively searching for devices. Network devices discovery glues together multiple techniques and heuristics, including ARP pinging, SNMP querying, SSDP discovery and MDNS names resolution. By opportunely combining the pieces of information obtained by actively probing network devices, ntopng is not only to discover them, but also to understand the services they provide as well as the operating system they run.

This is what the outcome of a active network device discovery looks like in ntopng.

For a detailed explanation of all the techniques and heuristics implemented, we refer the interested reader to this article.

ntopng release 3.2 is also incredibly more efficient when it comes to handling big traffic volumes. Indeed, I/O operations have been reduced to a great extent, thus alleviating the pressure on disks and, at the same time, making the software running faster. In addition, all the periodic activities that crunch hosts and interfaces traffic into time series data are now run by a thread pool that orchestrates their execution in parallel to fully leverage any modern multi-core system. The benefits of reduced I/O and parallel periodic activities execution together make ntopng sensibly more responsive also when browsing the web user interface.

Added support for banned sites detection with informative splash screen

Implement per-host/mac/pool flow drop count

nDPI traffic categories and RRDs

Implements MySQL database interoperability between ntopng and nProbe

Improvements

Flows sent by nProbe over ZMQ

Batched, compressed ZMQ flow format to optimize data exchange

Use of post-nat src/dst addresses and ports

Handles multiple balanced ZMQ endpoints

Periodic tasks performed by a thread-pool to optimize cores utilization

Hosts and devices are walked in batches to greatly reduce Lua VM memory

Full systemd support for Debian, Ubuntu, Centos, and Raspbian

Extended sFlow support to include sample packet drops and counter stats in interface views

Stacked applications and categories charts for ASes, Networks, etc

Security Fixes

More restrictive permissions for created files and directories

Fix of a possible dissectHTTP reads beyond end of payload

]]>5945nProbe 8.2 stable is out – A Wink At Next-Gen ASA Firewallshttps://www.ntop.org/nprobe/nprobe-8-2-stable-is-out-a-wink-at-next-gen-asa-firewalls/
Mon, 04 Dec 2017 08:55:44 +0000https://www.ntop.org/?p=5958Continue reading →]]>We are pleased to announce that the new 8.2 release of nProbe is out. This release features full Cisco ASA NetFlow support. ASA are industry’s first threat-focused next-generation firewalls that export a rich set of information through NetFlow. Being able to collect ASA data using nProbe will give you an advantage over collectors that only interpret standard NetFlow. Collected data can also be sent to ntopng over ZMQ to actually create a very effective solution for the monitoring and visualization of firewall-generated data.

ZMQ-based data export has been greatly improved in this release, too. ZMQ, a high-performance asynchronous messaging library, has always been used to send collected and monitored data from nProbe to ntopng in a JSON-encoded format. Nonetheless, some peculiarities of the JSON-encoded format were preventing ultra-high throughputs from being reached when exchanging data over ZMQ. This release heavily uses batching and compression to remove any possible bottleneck occurring in ZMQ communications.

Added support for flowDurationMillis Fixed bug for properly handling flowStart/flowEndMillis

]]>5958Announcing nDPI 2.2https://www.ntop.org/announce/announcing-ndpi-2-2/
Mon, 04 Dec 2017 07:50:50 +0000https://www.ntop.org/?p=5964Continue reading →]]>Today we are glad to release nDPI stable version 2.2. This minor release present several fixes and adds support for a handful of new protocols. It also features custom application categories to allow users to create personalized mappings between protocols and categories.

New Supported Protocols and Services

Improvements

Windows 10 detection from UA and indentation

Determine STUN flows that turn into RTP

Fixes for iQIYI and 1kxun

Android fingerprint

Added DHCP class identifier support

]]>5964Implementing PF_RING-based Hardware Flow Offload in Suricatahttps://www.ntop.org/ntop/implementing-pf_ring-based-hardware-flow-offload-in-suricata/
Thu, 16 Nov 2017 18:08:30 +0000https://www.ntop.org/?p=5936Continue reading →]]>Last month we have integrated hardware flow offload in PF_RING 7.0. This week Alfredo has presented at Suricon 2017 the integration of hardware flow offload with Suricata and demonstrated that with this technology you can significantly reduce packet drops and CPU load. Below you can see how NetFlow traffic analysis and Suricata can both benefit from this work.

]]>5936Using nDPI to Turn Wireshark Into a Traffic Monitoring Toolshttps://www.ntop.org/ndpi/wireshark_ndpi/
Tue, 14 Nov 2017 21:11:06 +0000https://www.ntop.org/?p=5930Continue reading →]]>Last week at Sharkfest EU we have shown how you can use nDPI and Lua scripting to turn Wireshark into a traffic monitoring tool.

We remind you that all the ntop contributions to Wireshark are open source and can be found on github.

We need your feedback and we could be glad if our community could give us guidance in the next steps. So please don’t be shy and send us a mail about 2018 plans. Thank you!

]]>5918Network Device Discovery. Part 1: Active Discoveryhttps://www.ntop.org/ntopng/network-device-discovery-part-1-active-discovery/
Fri, 27 Oct 2017 07:37:38 +0000https://www.ntop.org/?p=5899Continue reading →]]>Since its introduction in 1998, ntop(ng) has been a pure (well beside DNS address resolution if enabled) passive network monitoring tool. Recently we have complemented it with active device discovery in order to find out if there are silent devices in our network, and what services/OS our devices are featuring. In this article we will analyze how active discovery works, leaving to a future article the analysis of passive discovery.

Active discovery can be started on demand from the menu

or from the network preferences to enable periodic discovery

The result of network discovery is stored in redis and kept for future use. It allows to figure out the device type (is it a printer, a router or a tablet?) and their OS/features and advertised network services.

Above you can see a scan of a small home network. As you can see ntopng has recognised the router, and for my iMac detected the network services as well the exact computer model and type.

The first thing ntopng does is a “ARP ping” to all local subnet hosts to identify the active assets. ntopng reads the IP subnet/mask from the network internet interface (if the interface has no subnet, for instance when you have a span port, active discovery won’t work) and starts pinging all devices. We do this via ARP as it’s a reliable method contrary to ICMP ping that might be disabled for some hosts. At the end of this process we have a list of active hosts that is compared against the list of devices that you see under Devices -> Layer 2. If there is a device that answers ARP ping that was not listed under L2 we mark it as ghost device (i.e. a device that is active on our local subnet but that is silent/ghost)

We do a SSDP discovery so that devices (not all devices will answer to SSDP) can tell us the services they advertise. This works by sending a message to a multicast address and waiting for responses. Note that we might receive answers from hosts that do NOT belong to our subnet as they have seen the SSDP request: this is a good way to figure out it our local LAN is serving multiple networks or if somebody has created a private network (without permission?). So if you do that, make sure your devices are silent (note that ARP messages are sent in broadcast so even without SSDP you will be discovered by the network administrators), otherwise your secret network overlay will be detected. Also note that via SSDP is possible to learn the device capabilities as well the icon that is eventually stored inside the network device and that is shown by ntopng next to the IP address.

In the meantime while SSDP responses are being received, for all active hosts discovered via ARP, ntopng sends a SNMP request to learn more about the device. As the standard “public” community is used, we might discover only a part of the devices. SNMP helps detecting features of devices such as printers or access points/routers.

As small networks do not usually have a DNS, we use MDNS to resolve names (see for instance fritx.box) for local hosts when DNS is not available. Via MDNS it is also possible to know (in particular for Apple devices/phones) the advertised services and the device model/type. Recent OSX versions are much more verbose than old releases.

Done all this we merge the information so far collected and produce discovery information as you can see on the above screen. The discovery information is kept on redis so it survives across ntopng restarts.

In summary

If you have “hidden” devices on your network using a different subnet other than the one that you are supposed to use, make sure that devices are really hidden. But in all cases ARP will be broadcasted, so you will be caught soon or later.

If you have installed in your network a NAT router that hides your private devices, ntopng will figure this out. For instance looking at User-Agent in HTTP headers that report OS and not just browser. Beware of this or always use SSL that is unlikely as your OS is sending many spontaneous requests so you will be caught too.

SSDP/MDNS are very verbose protocols and advertise more information that you might think. Consider this when planning network security.

Finally we need your help! In our networks there are not so many different device types. Please send us reports form discovery in your network so we can improve the discovery process, or if you can send us a pull request with enhancements. Thank you!

]]>5899Introducing PF_RING 7.0 with Hardware Flow Offloadhttps://www.ntop.org/pf_ring/introducing-pf_ring-7-0-with-hardware-flow-offload/
Fri, 20 Oct 2017 19:54:37 +0000https://www.ntop.org/?p=5881Continue reading →]]>This is to announce a new PF_RING major release 7.0.
In addition to many improvements to the capture modules, drivers upgrades, containers isolation,
the main change of this release is the ability to offload flow processing to the network card (when supported by the underlying hw).

Flow offload is a great feature for cutting the CPU load when using applications doing intensive flow processing, as it’s possible to let the network card handle activities like flow classification (update flow statistics) and shunting (discard or bypass flows according to the application verdict). This saves CPU for further processing (e.g. DPI), or for running multiple applications on the same box (Netflow probe and traffic recording, or IDS). Enabling flow offload it is possible to receive from the capture stream both raw packets (with metadata including the flow ID) and flow records (in the form of periodic flow stats updates), and it is possible to shunt a specific flow providing the flow ID.

Flow offload is currently supported by 10/40G Accolade Technology adapters of the ANIC-Ku Series (tested on ANIC-20/40Ku, ANIC-80Ku), however PF_RING provides a generic API that is hardware agnostic, as always.

Soon we will post news on how to accelerate applications by leveraging on flow offload. This not only reduces the CPU load but it opens up to many new opportunities as combining on the same box flow-based analysis and packet-to-disk. For those who will attend Suricon 2017, you can hear how Suricata benefits from this new technology to move this IDS to 40/100 Gbit.

This article explains how to install and configure the ntopng datasource plugin, and how to build a dashboard for the visualization of ntopng-generated metrics.

A video tutorial is available as well:

Introduction

Grafana is an open platform for analytics and visualization. An extremely-well engineered architecture makes it completely agnostic to the storage where data resides. This means that you can build beautiful dashboards by simultaneously pulling points from data sources such as ntopng, MySQL and Influxdb, just to name a few. Grafana interacts with tens of different data sources by means of datasource plugins. Those plugins provide a standardized way to deliver points to Grafana. ntopng implements one of those datasource plugins, to expose metrics of monitored interfaces and hosts, including throughput (bps and pps) and Layer-7 application protocols e.g., (Facebook, Youtube, etc).

Exposed Metrics

ntopng exposes metrics for monitored interfaces as well as for monitored hosts. Each metric is identifiable with a unique, self-explanatory string. In general, interface metrics are prefixed with the string interface_ while host metrics are prefixed with the string host_. Similarly, a suffix indicates the measurement unit. Specifically, _bps and _pps are used for bit and packet rates (i.e., the number of bits and packets per second), whereas _total_bytes and _total_packets are used for the total number of bytes and packets over time, respectively.

After restarting Grafana, you can connect to its web User Interface (UI) and visit the Plugins page. ntopng will be listed under the datasources tab.

Configuring the ntopng Datasource

A new datasource with type ntopng will be available once the ntopng datasource plugin is installed. Multiple ntopng datasources can be created to connect to several running ntopng instances. The list of configured datasources is available at the Grafana ‘Data Sources’ page. The following image shows two ntopng datasource configured with the aim of connecting to two different ntopng instances running on separate machines.

Adding a new ntopng datasource is a breeze. Just hit the ‘+ Add datasource’ button inside the Grafana ‘Data Sources’ page. This will open an ‘Edit Data Source’ page that can be used to specify ntopng connection parameters.

To configure the ntopng datasource select ntopng as the datasource Type and give it a mnemonic Name that will help you identifying the datasource connection. The Url in the HTTP settings must point to a running ntopng instance, to the endpoint /lua/modules/grafana. For example, to connect to an ntopng running on host devel on port 3001, you have to use url http://devel:3001/lua/modules/grafana.

The Access method must be set to direct. Tick Basic Auth if your ntopng instance has authentication enabled and specify a username-password pair in fields User and Password. The pair must identify an ntopng user. Leave the Basic Auth checkbock unticked if ntopng has no authentication (--disable-login).

Finally, hit the button Save and Test to verify the datasource is working properly. A green message Success: Data source is working will appear to confirm the datasource is properly set up.

The following screenshot highlights the connection to an ntopng instance running on host devel on port 3001.

Building a Dashboard

Once the datasource is properly set up, you can visualize ntopng timeseries in any of your Grafana dashboards. Dashboards are flexible ensembles of panels. Each panel is meant to visualize a single timeseries. Panels are added in any dashboard by clicking on the ‘Add Row’ button that will allow you to choose among the available panel types.

Currently, ntopng provides timeseries that can be used effectively to build ‘Chart’ and ‘Singlestat’ panels.

Adding an Interface Speed Panel

To add an interface speed panel, select ‘Graph’ in the available panel types. A graph panel with random data will be automatically added to the dashboard. Click on the ‘Panel Title’ and select ‘Edit’. A configuration page as the following will appear:

There is a ‘Test data: random walk’ timeseries with random data by default. Drop it by clicking on the bin. To add ntopng metrics select one of the ntopng datasources configured from the ‘Panel Data Source’ dropdown. In the following image, an ntopng datasource named lab-monitor is selected:

Once the datasource is selected, you can click the ‘Add query’ button and start type a metric name. Autocompletion will automatically show all the available metrics matching the typed text. In the image above, interface eno1 bps is picked among all timeseries available. As soon as the metric is chosen, a chart will be populated. However, as shown below, the chart is sill pretty basic and some extra work is needed to configure the axis unit of measure as well as the title.

To change the chart title select tab ‘General’ and input the title:

More important, to set the unit of measure of the y-axis select tab ‘Axes’ and pick ‘bits/sec‘ from the ‘Unit’ dropdown.

The final result is shown in the picture below

Adding an Interface Layer-7 Application Protocols Panel

To add an interface application protocols panel the above instructions apply. Just make sure select the interface metric ending in _allprotocols_bps. In addition, as this metric carry more than one timeseries (one per application protocol), it is recommended to stack them by ticking the ‘Stack’ checkbox under the ‘Display’ tab.

The final result will appear similar to the following image

Adding the Interface Average Speed Panel

Using a ‘Singlestat’ panel it is possible to crunch a metric using an aggregation function. To visualize the average speed, you can add a ‘Singlestat’ panel, select the interface traffic timeseries, and configure avg as ‘Stat’ in the ‘Options’ tab, as well as bits/sec in the ‘Unit’.

A Full ntopng Grafana Dashboard

By putting together all the panels introduced above, you can build a complete dashboard as the one shown here

Remember that you can combine panels created with ntopng with panes created from other datasources (e.g., MySQL or InfluxDb). There is no limit on how you can combine panels to create dashboards!

Conclusion

ntopng features an handy datasource plugin that exposes monitored metrics to Grafana. Visualizing ntopng metrics in Grafana will allow you to show ntopng data inside the beautiful Grafana UI, and will give you enough flexibility to mix and match ntopng data with other data sources.