Using Kinesis in streaming solutions

Keyboard Shortcuts

Lynn Langit will help you bring all the Amazon Web Services together by putting streaming services into context in a real-world solutions scenario. In this video you will learn how to consume data out of streams by exploring AmazonLabs on Github to master Kinesis, understand Firehose, use ETL partners, and apply Kinesis integration.

- [Voiceover] Okay, we just went through…these new services and…if you're like me you're probably wondering,…how do I fit all this stuff together…and what does this look like in a real world scenario.…So let me talk to you about a customer solution…that I worked with a team to build…and I think that'll help you to put this into context.…So, the idea was that was we had log files…being dumped into a bucket from other sources.…Again, this is a simplified architecture.…There were a number of buckets based on…geographic area, products, so on and so forth.…But just for the purposes of understanding…how to actually use Streams and Firehoses,…this is a simplification of the architecture we built.…

So you have data coming in in log files,…and the log files are of varying sizes,…they come in at different times.…So the first thing that we used…was a Lambda function,…which was basically a listener on the bucket.…So when files came in,…that function would listen…and then would act like a trigger…on the data into the bucket,…

Author

Released

7/18/2016

Learn what you need to know to implement cloud-based big data solutions using the right mix of Amazon Web Services—from big data and cloud architect and AWS community hero Lynn Langit.

Starting with top-level categories of storage, data, computer, and services, Lynn guides you through planning your ideal AWS architecture, providing service demos using the AWS Console, command-line interface, and other tools. Learn when to use which service for which business case, such as Docker or Lambda or DynamoDB or Aurora? She shows how to script creation of services such as S3 buckets and EC2 instances, create and populate a managed data warehouse, and develop a data processing pipeline that works for you. Chapter 6 covers the AWS Internet of Things (IoT) services.

These exercises can help you build proof-of-concepts, minimum viable products, and deployable solutions to scale and support big data initiatives at your company.