Using the InsightIQ iiq_data_export Utility

InsightIQ includes a very useful data export tool: iiq_data_export. It can be used with any version of OneFS beginning with 7.x. While the tool is compatible with older versions of the operating system, if you’re running OneFS v8.0 or higher it offers a much needed performance improvement. The improvements allow this to be a much more functional tool that can be used daily, and for quick reports it’s much faster than relying on the web interface.

Applications of this tool could include daily reports for application teams to monitor their data consumption, charge-back reporting processes, or administrative trending reports. The output is in csv format, so there are plenty of options for data manipulation and reporting in your favorite spreadsheet application.

The utility is a command line tool, so you will need to log in to the CLI with an ssh session to the Linux InsightIQ server. I generally use putty for that purpose. The utility works with either root or non-root users, so you won’t need elevated privileges – I log in with the standard administrator user account. The utility can be used to export both performance stats and file system analytics [fsa] data, but I’ll review some uses of iiq_data_export for file system analytics first, more specifically the directory data-module export option.

The default command line option for file system analytics include list, describe, and export:

Note that how deep you can go down the /ifs subdirectory tree depends on the FSA configuration in InsightIQ. By default InsightIQ will configure the “directory filter maximum depth” option to 5, allowing directory information as low as
/ifs/dir1/dir2/dir3/dir4/dir5. If you need to dive deeper the FSA config will need to be updated. To do so, go to the Configuration Page, FSA Configuration, then the “Directory Filter path_squash) maximum depth setting. Note that the larger the maximum depth the more storage space an individual FSA result will use.

First, let’s create a text file with a list of the subdirectories under /ifs that we want to report on. I’ll create a file named “directories.txt” in the /home/administrator folder on the InsightIQ server. You can use vi to create and save the file.

[administrator@corporate_iq1 ~]$ vi directories.txt[add the following in the vi editor...]
NFS_exports/warehouse/warehouse_dec2017/dir_t01
NFS_exports/warehouse/warehouse_dec2017/dir_cat
NFS_exports/warehouse/warehouse_dec2017/dir_set

I’ll then use vi again to create the script itself. You will need to substitute the cluster name and the job ID to match your environment.

As I mentioned at the beginning of this post, this command can also provide performance related information. Below are the default command line options.

usage: iiq_data_export perf list [-h] [--breakouts] [--clusters] [--data-modules]
Options:
-h, --help Show this help message and exit.
Mutually Exclusive Options:
--breakouts Displays the names of all breakouts that InsightIQ supports for
performance data modules. Each data module supports a subset of
breakouts.
--clusters Displays the names of all clusters that InsightIQ is monitoring.
--data-modules Displays the names of all available performance data modules.
iiq_data_export perf list: error: One of the mutually exclusive arguments are
required.

As an example, if I want to review the CPU utilization for the cluster, I’d type in the command below. It will show all of the CPU performance information for the specified cluster name. Once I’ve had more time to dive in to the performance reporting aspect of InsightIQ I’ll revisit and add to this post.