Custom Reporting with Isilon OneFS API Calls

Have you ever wondered where the metrics you see in InsightIQ come from? InsightIQ uses OneFS API calls to gather information, and you can use the same API calls for custom reporting and scripting. Whether you’re interested in performance metrics relating to cluster capacity, CPU utilization, network latency & throughput, or disk activities, you have access to all of that information.

I spent a good deal of time already on how to make this work and investigating options that are available to make the gathered data be presentable in some useful manner. This is really just the beginning, I’m hoping to take some more time later to work on additional custom script examples that gather specific info and have useful output options. For now, this should get anyone started who’s interested in trying this out. This post also includes a list of available API calls you can make. I cover these basic steps in this post to get you started:

How to authenticate to the Isilon Cluster using cookies.

How to make the API call to the Isilon to generate the JSON output.

How to install the jq utility to parse JSON output files.

Some examples of using the jq utility to parse the JSON output.

Authentication

First I’ll go over how to authenticate to the Isilon cluster using cookies. You’ll have to create a credentials file first. Name the file auth.json and enter the following info into it:

Note that I am using root for this example, but it would certainly be possible to create a separate account on the Isilon to use for this purpose. Just give the account the Platform API and Statistics roles.

Once the file is created, you can make a session call to get a cookie:

Now that we have data in json format, we need to be able to parse it and change it into a more readable format. I’m looking to convert it to csv. There are many different scripts, tools, and languages available for that purpose online. I personally looked for a method that can be used in a simple bash script and jq is a good solution for that. I use Cygwin on a windows box for my scripts, but you can download any version you like for your flavor of OS. You can download the JQ parser here: https://github.com/stedolan/jq/releases.

Instructions for the installation of jq for Cygwin:

Download the latest source tarball for jq from https://stedolan.github.io/jq/download/

Open Cygwin to create the folder you’d like to extract it in

Copy the ‘jq-1.5.tar.gz’ file into your folder to make it available within Cygwin

From a Cygwin command shell, enter the following to uncompress the tarball file : ‘tar -xvzf jq-1.5.tar.gz’

Change folder location to the uncompressed folder e.g. ‘cd /jq-1.5’

Next enter ‘./configure’ and wait for the command to complete (about 5 minutes)

Then enter the commands ‘make’, followed by ‘make install’

You’re done.

Once jq is installed, we can play around with using it to make our json output more readable. One simple way to make it into a comma separated output, is with this command:

Using the –compact-output switch isn’t the only way to manipulate the data, and probably not the best way. I haven’t had much time to work with the @csv option in JQ, but it looks very promising. for this. Below are a few notes on using it, I will include more samples in an edit to this post or a new post in the future that relate this more directly to using it with the Isilon-generated output. I prefer to use csv files for report output due to the ease of working with them and manipulating them with scripts.

Order is significant for csv, but not for JSON fields. Specify the mapping from JSON named fields to csv positional fields by constructing an array of those fields, using [.date,.count,.title]:

If the input is not an array of objects but just a sequence of objects then we can omit the .[] – but then we can’t get the header at the top. It’s best to convert it to an array using the –slurp/-s option (or put [] around it if it’s generated within jq).

More to come on formatting JSON for Isilon in the future…

Isilon API calls

All of these specific API calls were pulled from the EMC community forum, I didn’t compose this list myself. It’s a list of the calls that InsightIQ makes to the OneFS API. They can be queried in exactly the same way that I demonstrated in the examples earlier in this post.

Please note the following about the API calls regarding time ranges:

Every call to the “/platform/1/statistics/current” APIs do not contain query parameters for &begin and &end time range.

Every call to the “/platform/1/statistics/history” APIs always contain query parameters for &begin and &end POSIX time range.