Automated Data Pipeline

Automated Data Pipeline is about removing the laborious manual processes involved in any stage of data processing. Data integration is not only about connecting data sources with data targets; it’s also about automating the process – triggering based on events, scheduling, monitoring, trouble-shooting and running workflows that handle errors, cleanup, file handling, etc.

Now, we’ll deploy our project into a production environment and schedule its execution, moving it from CloverETL Designer to CloverETL Server. This is the beginning stage of automating our data pipeline.

Finally, we’ll fully automate the execution of the data pipeline in CloverETL Server. We will watch for the arrival of new input files, process them using our graph, back up the input files and log the entire process.