Thursday, June 9, 2016

I've been getting lots of questions about how to develop/debug scripts that go into the ExecuteScript processor in NiFi. One way to do this is to add a unit test to the nifi-scripting-processors submodule, and set the Script File property to your test script. However for this you basically need the full NiFi source.

To make things easier, I basically took a pared-down copy of the ExecuteScript processor (and its helper classes), added nifi-mock as a dependency, and slapped a command-line interface on it. This way with a single JAR you can run your script inside a dummy flow containing ExecuteScript.

The result is version 1.1.1 of the NiFi Script Tester utility, on GitHub and Bintray.

Basically it runs your script as a little unit test, and you can pipe stdin to become a flowfile, or point it at a directory and it will send every file as a flowfile, stuff like that. If your script doesn't need a flowfile, you can run it without specifying an input dir or piping in stdin, it will run once even without input. The usage is as follows:

Usage: java -jar nifi-script-tester-<version>-all.jar [options] <script file>
Where options may include:
-success Output information about flow files that were transferred to the success relationship. Defaults to true
-failure Output information about flow files that were transferred to the failure relationship. Defaults to false
-no-success Do not output information about flow files that were transferred to the success relationship. Defaults to false
-content Output flow file contents. Defaults to false
-attrs Output flow file attributes. Defaults to false
-all-rels Output information about flow files that were transferred to any relationship. Defaults to false
-all Output content, attributes, etc. about flow files that were transferred to any relationship. Defaults to false
-input=<directory> Send each file in the specified directory as a flow file to the script
-modules=<paths> Comma-separated list of paths (files or directories) containing script modules/JARs

As a basic example, let's say I am in the nifi-script-tester Git repo, and I've built using "gradle shadowJar" so my JAR is in build/libs. I can run one of the basic unit tests like so:

This shows minimal output, just the logging from the framework and the default summary statistic of "Flow Files transferred to success". Instead let's try a more comprehensive example, where I pipe in a JSON file, run my Groovy script from the JSON-to-JSON conversion post, and display all the attributes and contents and statistics from the run:

There are options to suppress or select various things such as relationships, attributes, flowfile contents, etc. As a final example let's look at the Hazelcast example from my previous post, to see how to add paths (files and directories) to the script tester, in the same way you'd set the Module Path property in the ExecuteScript processor:

This script would not work without the Hazelcast JARs, so this shows how the "-modules" option is used to add them to the classpath for testing.

The nifi-script-tester only supports Javascript and Groovy at the moment; including Jython (for example) would increase the JAR's size by 500% :/ Right now its only 8.6 MB, so a little big but not too bad.