AWS Kinesis

AWS Kinesis is a cloud based web service that enables managing streaming data such as stock quotes, gaming data, social network (facebook) data, geospacial data such as Lyft/Uber/Swiggy or data collected by iOT devices

Kinesis Streams

Use Amazon Kinesis Data Streams to collect and process large streams of data records in real time.

You’ll create data-processing applications, known as Amazon Kinesis Data Streams applications. A typical Amazon Kinesis Data Streams application reads data from a Kinesis data stream as data records. These applications can use the Kinesis Client Library, and they can run on Amazon EC2 instances.

The processed records can be sent to dashboards, used to generate alerts, dynamically change pricing and advertising strategies, or send data to a variety of other AWS services.

Producers send data to Kinesis Streams

Data is stored from default 24 hours to max 7 days

Stream data is stored in shards (capacity of stream = sum of the capacities of all the shards inside the stream)

These streams are consumed by consumers (EC2)

Kinesis Firehose

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk.

You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified.

You can also configure Kinesis Data Firehose to transform your data before delivering it.

No need to worry about shards/streams

record is the data of interest that your data producer sends to a Kinesis data delivery stream. A record can be as large as 1,000 KB.

Data is not stored. Its immediately sent to the consumer

Kinesis Data Analytics

you can process and analyze streaming data using standard SQL. The service enables you to quickly author and run powerful SQL code against streaming sources to perform time series analytics, feed real-time dashboards, and create real-time metrics.

To get started with Kinesis Data Analytics, you create a Kinesis data analytics application that continuously reads and processes streaming data.

The service supports ingesting data from Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose streaming sources. Then, you author your SQL code using the interactive editor and test it with live streaming data. You can also configure destinations where you want Kinesis Data Analytics to send the results.