Data Pipeline use cases

There are plenty of use cases when AWS Data Pipelines could save a fortune and speed up business decisions. Firstly, it is serverless that means you do not have anything on your server if you do not want to. Everything starts entirely on AWS side and you pay only per execution.

Advantages of Data Pipeline

Analyse daily users’ behaviour through extracting data from logs

Analyse transactions for payment system

Analyse stock exchange reports. And many more

So, Data Pipeline allows you to spin up entire infrastructure needed for Hadoop cluster. Run all logic you desire to process your data with and shut down.

Main steps of implementing Data Pipeline

Bootstrapping cluster. At this point settings which are organizing how many core and slave instances are gonna be launched. What kind of memory settings for instances and how much memory JVMs will use while running Hadoop tasks.

Keep in mind that each of these listed steps could be joined with another SNS service. This is very powerful because it could lead to joining your app with data pipeline progress and results. So every step whether it succeeded or failed will generate an event which will send HTTP request to your app server informing about that step status. The app in order could take according action like informing users that results are ready or start another Data Pipeline based on results of previous. Important to remember that data pipeline could perform once or on some scheduled period. This means your Death Star cluster starting every midnight does important work and shuts itself down.

Steps could be various and with any complexity you wish. In simple words, every step is a Hadoop Java application. This also gives an advantage. Each step could be run separately in testing environment making sure it’s properly tested and ready to perform on live data.

Put pipeline definition. Here where you’re putting all your details and configure what should actually happen. Important thing here is when you assign your pipeline to SNS to track what happens, your app has to be able to confirm SNS subscription.