Implementing Dstat with Performance Co-Pilot

Dstat is a beloved tool by many, and a staple when diagnosing system performance issues. However, the original dstat is no longer actively developed. This poses an immediate problem for distributions like Fedora moving to a Python 3 stack, as it lacks a Python 3 implementation (both the tool itself, and its many plugins). It is also problematic in that the plugin system was relatively simplistic and in need of a significant redesign and rewrite to add new desired features.

Performance Co-Pilot to the rescue

Performance Co-Pilot (PCP) is a lightweight performance analysis toolkit. Used to easily collect metrics, analyze (both) live and historical performance data. It's also designed to be easily extensible, offering APIs and libraries to extract and make use of performance metrics from your own application.

We've taken those features, combined them with the original dstat UI code, and produced a new dstat. From an architectural perspective, PCP separates the collection of metrics from their analysis/display through the performance metrics api (PMAPI). This allows developers to write metric collections/sampling functionality once (efficiently), and reuse that implementation across different requests for reporting/visualizing the metric by abstracting it through calls with the PMAPI instead of the metric sources directly.

Adapting this to dstat

Similar to dstat, PCP has its own implementation of vmstat, iostat, and mpstat. Each of which utilize PCP's extensible collection API's to extract the metrics in a manner similar to their original tools.

We've done this again with dstat, and with our Python libraries, our libraries are compatible with Python 2 and Python 3 versions.

As with the original dstat, you can specify alternate/in-depth metric categories to display via command line:

Plugins

As we continued to develop this new dstat and strove towards feature parity, one important aspect of the dstat UI we wanted to ensure we explicitly included was plugins. However, because we're already able to abstract the metric collection details through the PMAPI, this was a natural place to evolve how we describe plugins within the dstat UI and combined it with a longstanding TODO of creating a configuration file functionality.

Now, all PCP dstat metrics displayed are from 'plugins' defined by config files. This brings an advantage of making it even easier to create plugins for PCP's dstat. For example, compare the nfs3 plugin implementations:

Allow to write buffered to disk

PCP has had its own archive format for the past 20 years, which are written by the 'pmlogger' program (usually run as a service). When a PCP client tool uses the PMAPI (as this dstat implementation does), they also gain access to metrics stored in archives without having to implement the functionality themselves. This means even the brand-new dstat implementation can display previously gathered values as if they were live, using PCP standard flags to specify the timeframe of metrics to display.

Look into interfacing with apps

Another todo we looked after is adding interfaces for apps/protocols like Amavis, Apache httpd, Bind, CIFS, dhcpd, dnsmasq, gfs, Samba, Squid and others.

While not a 1:1 match with the original dstat, we have coverage of more than 70 different applications/domains of metrics. A full list is on GitHub in the PCP repository. Several of interest for dstat users include PMDAs for Apache, Bind, CIFS, GFS2, Gluster, Libvirt, MySQL, NGINX, Postfix, Prometheus, Redis, XFS, and others.

Another TODO we address is "design proper object model and namespace for _all_ possible stats."

This is done in PCP with the Performance Metrics Name Space (PMNS). Each metric is given a name, which is used to reference it in any given analysis tool, as well as in historical archives.

Finally, there’s the "create client/server monitoring tool" TODO. In the context of PCP's dstat, one of the main benefits of having the Performance Metrics Collector Daemon (PMCD) enabled on the target host is using dstat remotely. This distributed nature was built into PCP from the ground up, and using the PMAPI allows the new dstat to access the functionality remotely.

We think that dstat is a fantastic tool, and that's why we want to see it continue being used, even in the Python 3 world, and want to help it continue to evolve and expand its available features. PCP is available in most modern Linux distros. For further documentation and information, see the Performance Co-Pilot homepage. Want to contribute? Feel free to open a pull request on GitHub, or send your configurations to pcp@groups.io.