In this example we are going to see how to execute batch pipelines. In this pipeline, we will read Employee information by joining multiple tables from a JDBC datasource and with salary above 5K. We will apply two different business logic on active and inactive employees. Data of the active employees will be enriched with “country name” based on their “location id” by looking up from an existing database and finally pushed into a datastore. All the inactive employees are stored separately in a different datastore.

A sample pipeline of Spark batch is available on Git for your reference. Import the pipeline by following below steps:

Persist: It stores data in memory. Since we are forking two parallel flows, they require persisted data.

Filter: Responsible for filtering the incoming data based on the conditions defined. We are using two filter in the pipeline. The filter will emit records which has a 'false' value in 'is_active' field and the one beneath it, emits records with 'true' value in the field 'is_active'.

Enricher: It enriches the incoming data by using an inbuilt function- lookupRDBMS (String query). It populates country_name field of the message based on the location_id of the incoming data. Below mentioned is the code snippet used:

$lookupRDBMS("select country_name as country from countries c, locations l where l.country_id = c.country_id and l.location_id="+@{CloudTrialMessage.location_id})

• JDBC Emitter: This is the last component of pipeline, which is responsible for pushing data into any RDBMS table mentioned in the component configurations. All the mapped message fields are inserted into the columns of the defined table.

Table schema:

Defined few tables which are available for reading and writing of data as:

In this case we have dumped our data into an RDBMS datastore. In order to view it, click the “View Data” link from the left hand side panel.

Select JDBC as the component.

You can view your data after specifying the following field values:

• Connection: Name of the JDBC connection from where data needs to be fetched.• Table: Name of source table where data resides.• Fetch Count: Number of records to be fetched. (Defaults to 10.)• Inspect: Please follow this URL for more details.