Runtime parameters are parameters that you define in a pipeline and then call from within that same pipeline. When
you start the pipeline, you specify the parameter values to use. Use runtime parameters to specify values for pipeline
properties when you start the pipeline.

Runtime properties are properties that you define in a file local to the Data Collector and call from within a pipeline. With runtime properties, you can define different sets of values for different Data Collectors.

Similar to runtime properties, runtime resources are values that you define in a file local to the Data Collector and call from within a pipeline. But with runtime resources, you can restrict the permissions for the files
to secure sensitive information. Use runtime resources to load sensitive information from files at runtime.

Runtime Values

Runtime values are values that you define outside of the pipeline and use for stage
and pipeline properties. You can change the values for each pipeline run without having to
edit the pipeline.

You can use runtime values for any pipeline property that allows the use of the
expression language. You can, for example, use runtime values to represent batch sizes,
timeouts, directories, and URI. You cannot use runtime values to represent fields.

You can use the following methods of passing runtime values to pipelines:

Runtime parameters

Use runtime parameters when you want to specify the values for pipeline
properties when you start the pipeline.

You define runtime parameters when you configure the pipeline, and then you call
the parameters from within that pipeline. When you start the pipeline, you
specify the parameter values to use.

Runtime parameters are defined for a single pipeline - only that pipeline can
call them.

Use runtime properties when you want to define values for multiple pipeline
properties in a file.

You define runtime properties in a file local to the Data Collector, and then you call the properties from within a pipeline. At runtime, Data Collector loads the property values from the file. A runtime properties file can
contain multiple properties.

Runtime properties are defined for the entire Data Collector - any pipeline can call them.

Runtime resources

Use runtime resources when you want to secure sensitive information in files
with restricted permissions.

You define runtime resources in a file local to the Data Collector, and then you call the resources from within a pipeline. You can restrict the
permissions for the resource files to secure sensitive information. A resource
file must contain one piece of information.

Runtime resources are defined for the entire Data Collector - any pipeline can call them.

Note: When configuring runtime values for pipelines run by StreamSets Control Hub,
all Data Collector that run the pipeline must have the values locally defined in the expected
locations.

Using Runtime Parameters

Runtime parameters are parameters that you define in a pipeline and then call from
within that same pipeline. When you start the pipeline, you specify the parameter values to
use. Use runtime parameters to specify values for pipeline properties when you start the
pipeline.

Use runtime parameters to
define values for stage and pipeline properties. For example, you can define an error
directory parameter that points to different directories on a test machine and a
production machine. Or you can define a connection parameter that points to different
database connections for an origin in the test and production environments.

When you define a runtime parameter, you enter the default value to use. When you start
the pipeline, you can specify another value to override the default. When the pipeline
runs, the value replaces the name of the runtime parameter.

Note: If you shut down and then restart Data Collector
without stopping the pipeline, the pipeline continues running with the last set of
parameter values.

To implement runtime parameters, perform the following steps:

Define runtime parameters.

Use an expression in the pipeline to call a runtime parameter.

Start the pipeline with parameter values.

When you monitor a pipeline started with parameters, you can view the
parameter values that the pipeline is currently using.

For example, define a parameter named JDBCConnectionString with the default
value of jdbc:mysql://localhost:3306/sample.

Step 2. Call the Runtime Parameters

Use an expression in the pipeline to call a runtime parameter.

You can use runtime parameters to represent any stage or
pipeline property that allows the use of the StreamSets
expression language, including properties that display as text boxes, checkboxes, or
drop-down menus. You can also call a runtime parameter in the code developed for a
scripting processor.

Calling from Text Boxes

To call a runtime parameter in a stage or pipeline property that displays as a text box,
use the following syntax:

${<parameter name>}

For example, to use the JDBCConnectionString runtime parameter for the JDBC Multitable
Consumer origin, enter the following syntax for the JDBC Connection String
property:

${JDBCConnectionString}

You can call a runtime parameter from within an expression language function by simply
entering the parameter name. For example, the following expression returns the value of
the JDBCConnectionString runtime
parameter:

${record:value(JDBCConnectionString)}

You can use a runtime parameter to represent a part of a property. For example, you could
use a RootDir runtime parameter and append the rest of the directory in the property as
follows:

${RootDir}/logfiles

Calling from Checkboxes and Drop-Down Menus

To call a runtime parameter in a stage or pipeline property that displays as a
checkbox or drop-down menu, you first must convert the property to a text box.

Click the Use Parameter icon () next to the checkbox or drop-down menu to convert
the property to a text box, and then call the parameter using the required syntax.

For example, the following image shows that the Delivery Guarantee property that displays
as a drop-down menu has been converted to a text box so that a parameter can be called
from the property:

The parameter must evaluate to a valid option for the property type:

Checkboxes

Parameters called from properties that display as checkboxes must evaluate
to true or false.

Drop-down menus

Parameters called from properties that display as drop-down menus must
evaluate to a valid key value. Each option in the menu has an associated key
value.

For example, to use a parameter for the Delivery Guarantee property that
displays as a drop-down menu, the parameter must evaluate to one of the
valid key values, AT_LEAST_ONCE or
AT_MOST_ONCE, and not to one of the menu options,
At Least Once or At Most
Once.

To view a valid key value, select the desired option from the
menu, then click the Use Parameter icon. The text box
displays the key value for the selected option. For example, after selecting
At Least Once for the Delivery
Guarantee property and then clicking the Use
Parameter icon, the text box displays
AT_LEAST_ONCE as the key value for the
At Least Once option:

Calling from Scripting Processors

You can call a runtime parameter in the code developed for a scripting processor.

The method you use to call the runtime parameter depends on the following scripting
processor types:

JavaScript Evaluator or Jython Evaluator processor

Use the following syntax in any of the processor scripts:
${<parameter name>}. For example, the following line of
JavaScript code assigns the value of the NewFieldValue parameter to a map
field:

records[i].value.V= ${NewFieldValue}

Groovy Evaluator processor

Use the sdcFunctions.pipelineParameters() method in any of the
processor scripts to return a map of all runtime parameters defined for the
pipeline. For example, the following line of Groovy code assigns the value of
the CompanyParam parameter to the Company Name
field:

Step 3. Start the Pipeline with Parameters

When you start the pipeline, specify
the values to use for the parameters.

Note: If you want to use the default parameter values, you
can simply click the Start icon to start the
pipeline.

From the pipeline canvas, click the More icon, and then
click Start with Parameters.

If Start with Parameters is not enabled, the pipeline
is not valid.

The Start with Parameters dialog box
lists all parameters defined for the pipeline and their default
values.

Override any default values with the values you want to use for this pipeline
run.

Click Start.

Viewing Runtime Parameters

When you monitor a pipeline started with parameters, you can view the parameter
values that the pipeline is currently using.

Before you can view
the current runtime parameter values, verify that the monitoring settings include
the Runtime Parameters chart.

In the Summary tab of the Monitor panel, click the
More icon, and then click
Settings.

In the Monitoring Settings dialog box, click in the
Charts field and then select Runtime
Parameters.

Click Save.

When you monitor a pipeline that you started with parameter values, the
Runtime Parameters chart displays as follows:

The Runtime Parameters chart does not display when
you start a pipeline without parameters.

Using Runtime Properties

Runtime properties are properties that you define in a file local to the Data Collector and call from
within a pipeline. With runtime properties, you can define different sets of values for
different Data Collectors.

Use runtime
properties to define values for stage and pipeline properties. For example, you can define an
error directory runtime property that points to different directories on a test machine and a
production machine. Similarly, you might create test and production runtime properties for the
origin and destination stages.

When defining a runtime property, you can use a static value or an environment variable.

When calling the runtime property, you can use it as part of a larger property definition.
For example, you can set a runtime property to the HOME environment variable, which would
differ on different machines, and then use the runtime property as a base directory for a
longer directory.

To implement runtime properties, perform the following steps:

Define runtime properties.

Use an expression in the pipeline to call a runtime property.

Step 1. Define Runtime Properties

You can define
runtime properties in the Data Collector
configuration file, sdc.properties, or in a separate runtime properties file.

If you define the properties in a separate runtime properties file, use the required
procedure for your installation type.

Data Collector
configuration file

Use the following steps to define runtime properties in the Data Collector
configuration file:

In the $SDC_CONF/sdc.properties file, configure the
runtime.conf.location property as
follows:

runtime.conf.location=embedded

To define the runtime properties in the $SDC_CONF/sdc.properties
file, use either of the following formats:

To use a value for a runtime property, use the following format:

runtime.conf_<property name>=<value>

For example,
the following runtime property defines a Hadoop FS directory template:

runtime.conf_HDFSDirTemplate=/HDFS/DirectoryTemplate

To use an environment variable for a runtime property, use the following
format:

runtime.conf_<property name>=${env("<environment_variable>")}

For
example, the following runtime property defines a base directory, setting it
to the HOME environment
variable:

runtime.conf_BaseDir=${env("HOME")}

Restart Data Collector
to enable the changes.

Separate runtime properties file for RPM and tarball

Use the following steps to define runtime properties in a separate runtime properties
file for an RPM or tarball installation:

Create a text file and save it in a directory relative to the
$SDC_CONF directory.

To define runtime properties in a separate text file, use either of the following
formats:

To use a value for a runtime property, use the following format:

<property name>=<value>

For example,
the following runtime property defines a Hadoop FS directory
template:

HDFSDirTemplate=/HDFS/DirectoryTemplate

To use an environment variable for a runtime property, use the following
format:

<property name>=${env("<environment_variable>")}

For
example, the following runtime property defines a base
directory, setting it to the HOME environment
variable:

BaseDir=${env("HOME")}

In the Data Collector
configuration file, $SDC_CONF/sdc.properties, configure the
runtime.conf.location property to point to the relative location of the
separate runtime properties file.

For example, the following separate runtime
properties file is located in a runtime directory that is
relative to the $SDC_CONF
directory:

runtime.conf.location=runtime/test-runtime.properties

Restart Data Collector
to enable the changes.

Separate runtime properties file for Cloudera Manager

Use the following steps to define runtime properties in a separate runtime properties
file for a Cloudera Manager installation:

Create a text file and define the runtime properties in the text file using either
of the following formats:

To use a value for a runtime property, use the following format:

<property name>=<value>

For example,
the following runtime property defines a Hadoop FS directory
template:

HDFSDirTemplate=/HDFS/DirectoryTemplate

To use an environment variable for a runtime property, use the following
format:

<property name>=${env("<environment_variable>")}

For
example, the following runtime property defines a base
directory, setting it to the HOME environment
variable:

BaseDir=${env("HOME")}

Save the text file in the same directory on every node that runs Data Collector.

In Cloudera Manager, select the StreamSets service and
click Configuration.

On the Configuration page, in the Data Collector
Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field,
add the following line to define the runtime properties file
directory:

In the Data Collector Advanced Configuration Snippet (Safety Valve) for
sdc.properties field, configure the
runtime.conf.location property to point to the separate
runtime properties file by adding the following
line:

Step 2. Call the Runtime Property

Use the runtime:conf function to call a runtime property. You can use
runtime properties to represent any stage or pipeline property that allows the use of the
expression language.

To call
a runtime property, use the following syntax:

${runtime:conf(<property name>)}

Note: If you defined the runtime properties in the Data Collector
configuration file, enter just <property name> and not runtime.conf_<property name>.

For example, to use the HDFSDirTemplate runtime property for the Hadoop FS destination, enter
the following syntax for the Directory Template
property:

${runtime:conf('HDFSDirTemplate')}

You can use a runtime property to represent a part of a property. For example, you could use a
RootDir runtime property and append the rest of the directory in the property as
follows:

${runtime:conf('RootDir')}/logfiles

Using Runtime Resources

Similar to runtime properties, runtime resources are values that you define in a file
local to the Data Collector and
call from within a pipeline. But with runtime resources, you can restrict the permissions
for the files to secure sensitive information. Use runtime resources to load sensitive
information from files at runtime.

Use runtime resources to
define sensitive values for stage and pipeline properties. You can, for example, use
runtime resources to represent user names and passwords, or OAuth authentication
information.

Step 1. Define Runtime Resources

For each resource, create a text file and save it in the $SDC_RESOURCES
directory.

A file must contain one piece of information to be used when the resource
is called.

Optionally, restrict the permissions for the file.

Generally, anyone can read a file.
To restrict permissions, configure the file so only the owner has read or write
permissions for the file - in octals, that's 600 or 400. And the owner must be the
system user that runs the Data Collector.

When you use the resource in the pipeline, you specify whether the file is
restricted.

Step 2. Call the Runtime Resource

Use the runtime:loadResource or
runtime:loadResourceRaw function to call a runtime resource. You can
use runtime resources to represent sensitive information in any stage or pipeline property
that allows the use of the expression language.

Note: In most cases, you'll use the runtime:loadResource function which
trims any leading or trailing whitespace characters from the file. However, if
needed, you can also use the runtime:loadResourceRaw function which
includes any leading or trailing whitespace characters in the file.

To
call a runtime resource, use the following syntax:

${runtime:loadResource(<file name>, <restricted: true | false>)}

For example, the following expression returns the contents of the JDBCpassword.txt file,
trimming any leading or trailing whitespace characters. The file contains a password and
is restricted so only the owner can read the
file: