Creating Models

One of the enhancements to cybersecurity most frequently requested is the ability to
augment the threat intelligence and enrichment processes with insights derived from machine
learning and statistical models. While valuable, this model management infrastructure has
significant challenges.

Applying the model management infrastructure might be both computationally and resource
intensive and could require load balancing and multiple versions of models.

Models require frequent training or updating to react to growing threats and new
patterns that emerge.

Models should be language and environment agnostic as much as possible. So, models
should include small-data and big-data libraries and languages.

To support these requirements, Hortonworks Cybersecurity Platform (HCP) powered by Metron
provides the following components:

A YARN application that listens for model deployment requests and upon execution,
registers their endpoints in ZooKeeper.

A command line deployment client that localizes the model payload onto HDFS and submits
a model request.

A Java client that interacts with ZooKeeper and receives updates about model state
changes (for example, new deployments and removals).

A series of Stellar functions for interacting with models deployed by the Model as a
Service infrastructure.

Install the YARN ApplicationThe YARN application listens for model deployment requests. Models are exposed as REST microservices that expose your model application as an endpoint. The YARN application takes the submitted request that specifies the model payload that includes a shell script and other model collateral which will start the microservice. Upon execution of the shell script that starts the model, the YARN application registers the endpoints in ZooKeeper.

Deploying ModelsAfter creating a model, you need to deploy the model onto HDFS and submit a request for one or more instances of the model.