Open Source Network Virtualization

Since v1.5 the MidoNet Java Agent exposes many counters and metrics via JMX. Initially, these were related to the Agent’s workload and performance. However, since v1.7 MidoNet has support for overlay and underlay traffic meters. There four kinds of meters:

The first three don’t need to be configured, they are provided by default. MN Agent tracks them by default on all corresponding objects. The user defined meters need to be configured in rules (we’ll leave that for a future post).

Each Agent exposes its locally tracked meters via JMX. Remember that an Agent’s local meters only count packets/bytes that ingress locally, so usually it’s necessary to sum across all Agents in a deployment to get a meaningful count. That’s the case for Tenant Routers, for example.

mm-meter is a tool installed by the midolman package that allows you to query a single Agent’s meters. The following is example output for its list command. Notice that this Agent only tracks meters for two ports, they are two VM-facing ports on the same overlay bridge in my test deployment. This agent only sees packets transmitted from c040e61d and received at 64594d49 because the former is bound at a remote Agent and the latter is bound locally. Also, notice that the tunnel source and destination IP addresses are integers. This should probably be fixed in MidoNet itself, but for this post I fixed it in the MN Agent scraper.

You can query the packet and byte counters for a single meter. The following is example output for the get command. The last two argument specify 4 updates at 60 second intervals. You can see that the first stats show the total counters, afterwards only deltas are printed.

Now launch the Prometheus container: docker run -p 9090:9090 prom/prometheus

The Prometheus container is configured to monitor itself. Verify that Prometheus is running correctly by pointing your browser to http://<your-host-ip>:9090/. Scroll down to the Targets section, you should see one healthy target: http://localhost:9090/metrics.

Prometheus pulls metrics over HTTP. Since MN Agents only serve metrics via JMX, we’re going to need the JMX to Prometheus bridge. However, I’ve had to hack it a bit so that:

it can read the MN Agent’s MeteringMXBean. This bean has 2 methods: String[] listMeters() and FlowStats getMeter(String name). In contrast, Prometheus’ JMX Exporter expects one bean per metric.

Running the JMX Scraper

Download the tar-file of the modified JMX Exporter here. It contains the following:

JMX Exporter jar

MidoNet jars containing the MeteringMXBean and FlowStats classes

Shell script for running the JMX Exporter (run_mn_scraper.sh)

JMX Exporter configuration file (mn_scraper_config.json) with MN Agent IP addresses, and rules matching only MidoNet’s device, port and tunnel metrics. Later we can add rules to expose the MN Agent’s other metrics.

Prometheus configuration file that defines a job for pulling metrics from the JMX Exporter.

Prometheus rules file with rule definitions to pre-aggregate device and port meters across all MN Agents.

tar -zxvf prom_mn_jmx_scraper.tar.gz

cd prom_mn_jmx_scraper

edit mn_scraper_config.json to provide the comma-separated list of IP addresses of all the MN Agents you want to scrape. For example: “hostPort”: “119.15.127.100:7200,119.15.127.101:7200”

Launch run_mn_scraper.sh

The scraper runs an httpserver on port 7201. Edit the script if you want to change the port.

The JMX Exporter (when run as an http server as we’re doing) scrapes its target each time its URL is queried. Verify the Exporter is working correctly by pointing your browser to http://<your-host-ip>:7201/metrics. You should at least see the following at the end of the response:

Now verify that your MN Agents are being scraped. The response should include lines like the following, where “mn_agent” shows the IP of your Agent. You should be able to find all the IPs you listed in mn_scraper_config.json:

Updating Prometheus

Now we’re ready to reconfigure Prometheus to pull metrics from our JMX Exporter and to run the aggregation rules specified in our file.

If necessary, change the target address specified in prometheus.conf. Since Prometheus is running in a container, while the JMX Exporter runs on the host, Prometheus must send queries to the host’s address on the docker0 bridge (the container’s “NetworkMode” = “bridge”). On my host, ifconfig shows that docker0 has address 172.17.42.1, and that’s why the file prometheus.conf uses that address.

Now you can go to the “Graph” section of Prometheus’ dashboard http://119.15.121.38:9090/graph and you can graph the pre-aggregated metrics created by the rules. For example the following identifies the metric containing the bytes arriving in a given minute at the device with UUID 7303fdb8…