Posts

How many ways you know to send data to AWS Kinesis Firehose?

Amazon Kinesis Firehose is a fully managed service that loads streaming data reliably to Amazon Redshift and other AWS services.

How to send data to Kinesis Firehose

You can send data to Firehose delivery stream directly or through other collectionsystems. Let’s take a look at a few examples.

Amazon Kinesis Agent

You can write to Amazon Kinesis Firehose using Amazon Kinesis Agent. Amazon Kinesis Agent is a stand-alone Java software application that offers a way to collect and send data to Firehose. The agent continuously monitors a set of files and sends new data to your Firehose delivery stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers.

Use case

You could analyze a live stream of Apache access log data and aggregate the total request for each HTTP response type every minute.

Firehose API

Writing to a Firehose Delivery Stream using the AWS SDK. You can use the Firehose API to send data to a Firehose delivery stream using the AWS SDK for Java, .NET, Node.js, Python, or Ruby.

Use case

You could move from your internal queue to a Kinesis Firehose service.

Fluentd plugin

Use case

You could use the Fluentd’s Amazon Kinesis Output plugin to aggregate semi-structured logs in real-time.

AWS Service Proxy

Using AWS API Gateway via Service proxy. It’s possible to create an API Gateway as a proxy to an AWS service, such as Amazon SNS, Amazon S3, Amazon Kinesis.

Use case

One use case where it’s valuable to have immediate insights is analyzing clickstream data. In the world of digital advertising, an impression is when an ad is displayed in a browser and a clickthrough represents a user clicking on that ad. All client web requests can be handled by Amazon API Gateway. This handles the web request traffic to get data into your analytics pipeline by being a service proxy to an Amazon Kinesis.