Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

published:29 Apr 2017

views:54508

www.datameer.com
It's clear how to represent a data file, but it's not necessarily clear how to represent a data stream. Jay Kreps, develoer of Kafka, diagrams how he solved this problem with Kafka.

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

published:04 Nov 2014

views:57242

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

published:07 Nov 2015

views:91202

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

published:11 Nov 2017

views:10168

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

published:15 Oct 2015

views:52668

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

published:26 Aug 2014

views:45834

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

History

Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. In November 2014, several engineers who built Kafka at LinkedIn created a new company named Confluent with a focus on Kafka.

Enterprises that use Kafka

The following is a list of notable enterprises that have used or are using Kafka:

Surname

A surname or family name is a name added to a given name. In many cases, a surname is a family name and many dictionaries define "surname" as a synonym of "family name". In the western hemisphere, it is commonly synonymous with last name because it is usually placed at the end of a person's given name.

The style of having both a family name (surname) and a given name (forename) is far from universal. In many countries, it is common for ordinary people to have only one name or mononym.

The concept of a "surname" is a relatively recent historical development, evolving from a medieval naming practice called a "byname". Based on an individual's occupation or area of residence, a byname would be used in situations where more than one person had the same name.

Apache people traditionally have lived in Eastern Arizona, Northern Mexico (Sonora and Chihuahua), New Mexico, West Texas, and Southern Colorado. Apacheria, their collective homelands, consisted of high mountains, sheltered and watered valleys, deep canyons, deserts, and the southern Great Plains. The Apache tribes fought the Spanish and Mexican peoples for centuries. The first Apache raids on Sonora appear to have taken place during the late 17th century. In 19th-century confrontations, the U.S. Army found the Apache to be fierce warriors and skillful strategists.

Apache groups are politically autonomous. The major groups speak several different languages and developed distinct and competitive cultures. The current division of Apache groups includes Western Apache, Chiricahua, Mescalero, Jicarilla, Lipan, and Plains Apache (also known as the Kiowa-Apache). Apache groups live in Oklahoma and Texas and on reservations in Arizona and New Mexico. Apache people have moved throughout the United States and elsewhere, including urban centers.

Raw data, i.e. unprocessed data, is a collection of numbers, characters; data processing commonly occurs by stages, and the "processed data" from one stage may be considered the "raw data" of the next. Field data is raw data that is collected in an uncontrolled in situ environment. Experimental data is data that is generated within the context of a scientific investigation by observation and recording.

The Latin word "data" is the plural of "datum", and still may be used as a plural noun in this sense. Nowadays, though, "data" is most commonly used in the singular, as a mass noun (like "information", "sand" or "rain").

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

10:52

How Does Apache Kafka Work? [Diagram]

How Does Apache Kafka Work? [Diagram]

How Does Apache Kafka Work? [Diagram]

www.datameer.com
It's clear how to represent a data file, but it's not necessarily clear how to represent a data stream. Jay Kreps, develoer of Kafka, diagrams how he solved this problem with Kafka.

Introduction to Apache Kafka by Joe Stein

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

39:01

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

48:20

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

11:48

Understanding Kafka with Legos

Understanding Kafka with Legos

Understanding Kafka with Legos

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

3:55

Introduction to Apache Kafka

Introduction to Apache Kafka

Introduction to Apache Kafka

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

1:30:40

Developing Real-Time Data Pipelines with Apache Kafka

Developing Real-Time Data Pipelines with Apache Kafka

Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

What is Apache Kafka?

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitioning LinkedIn to a fully stream-based architecture, he is now helping a number of Silicon Valley tech companies do the same thing with his new company, Confluent.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

50:55

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

10:12

1. Intro to Streams | Apache Kafka® Streams API

1. Intro to Streams | Apache Kafka® Streams API

1. Intro to Streams | Apache Kafka® Streams API

Write an app: http://kafka.apache.org/documentation/streams | The StreamsAPI of Apache Kafka is the easiest way to write mission-critical real-time applications and microservices with all the benefits of Kafka's server-side cluster technology. It allows you to build standard Java or Scala applications that are elastic, highly scalable, and fault-tolerant, and don’t require a separate processing cluster technology. Applications can be deployed on containers, VMs, or bare-metal hardware, to the cloud or on-premises.
MORE ON STREAMS
http://kafka.apache.org/documentation/streams
https://docs.confluent.io/current/streams/introduction.html
https://github.com/confluentinc/kafka-streams-examples
CONNECT
Subscribe: http://youtube.com/c/confluent?sub_confirmation=1
Site: http://confluent.io
GitHub: https://github.com/confluentinc
Facebook: https://facebook.com/confluentinc
Twitter: https://twitter.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit http://confluent.io

Apache Kafka is the most popular open source project for building real-time data pipelines and streaming applications. In this video, you will learn the key concepts of Kafka including the setup, configuration, and ingesting streaming data.
For learning more, you can follow our step-by-step tutorial below:
How To Install Apache Kafka on Ubuntu 14.04 - https://do.co/2sswlCy
To learn more about DigitalOcean: https://www.digitalocean.com/
Follow us on Twitter: https://twitter.com/digitalocean
Like us on Facebook: https://www.facebook.com/DigitalOcean
Follow us on Instagram: https://www.instagram.com/thedigitalocean
We're hiring: http://grnh.se/aicoph1

This Apache Kafka tutorial will help you master the basics of Apache Kafka including concepts of KafkaCluster, Kafka Data Model, Kafka Topic, Kafka Architecture and Use Case of Kakfa at LinkedIn. Apache Kafka is an open-source stream processing platform developed by the ApacheSoftwareFoundation written in Scala and Java.
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.
Subscribe to Simplilearn channel for more Big Data and Hadoop Tutorials - https://www.youtube.com/user/Simplilearn?sub_confirmation=1
Check our Big Data TrainingVideoPlaylist: https://www.youtube.com/playlist?list=PLEiEAq2VkUUJqp1k-g5W1mo37urJQOdCZ
Big Data and Analytics Articles - https://www.simplilearn.com/resources/big-data-and-analytics?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
To gain in-depth knowledge of Big Data and Hadoop, check our Big Data Hadoop and SparkDeveloperCertification Training Course: https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
#bigdata #bigdatatutorialforbeginners #bigdataanalytics #bigdatahadooptutorialforbeginners #bigdatacertification #HadoopTutorial
- - - - - - - - -
About Simplilearn's Big Data and Hadoop Certification Training Course:
The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab.
Mastering real-time data processing using Spark: You will learn to do functional programming in Spark, implement Spark applications, understand parallel processing in Spark, and use Spark RDD optimization techniques. You will also learn the various interactive algorithm in Spark and use Spark SQL for creating, transforming, and querying data form.
As a part of the course, you will be required to execute real-life industry-based projects using CloudLab. The projects included are in the domains of Banking, Telecommunication, Social media, Insurance, and E-commerce. This Big Data course also prepares you for the Cloudera CCA175 certification.
- - - - - - - -
What are the course objectives of this Big Data and Hadoop Certification Training Course?
This course will enable you to:
1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, AvroSchema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
- - - - - - - - - - -
Who should take up this Big Data and Hadoop Certification Training Course?
Big Data career opportunities are on the rise, and Hadoop is quickly becoming a must-know technology for the following professionals:
1. Software Developers and Architects
2. Analytics Professionals
3. SeniorIT professionals
4. Testing and Mainframe professionals
5. Data Management Professionals
6. Business Intelligence Professionals
7. Project Managers
8. Aspiring Data Scientists
- - - - - - - -
For more updates on courses and tips follow us on:
- Facebook : https://www.facebook.com/Simplilearn
- Twitter: https://twitter.com/simplilearn
- LinkedIn: https://www.linkedin.com/company/simplilearn
- Website: https://www.simplilearn.com
Get the android app: http://bit.ly/1WlVo4u
Get the iOS app: http://apple.co/1HIO5J0

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

published: 29 Apr 2017

How Does Apache Kafka Work? [Diagram]

www.datameer.com
It's clear how to represent a data file, but it's not necessarily clear how to represent a data stream. Jay Kreps, develoer of Kafka, diagrams how he solved this problem with Kafka.

Introduction to Apache Kafka by Joe Stein

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

published: 07 Nov 2015

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

published: 23 Feb 2017

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier ...

published: 11 Nov 2017

Understanding Kafka with Legos

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

published: 15 Oct 2015

Introduction to Apache Kafka

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

published: 26 Aug 2014

Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to a...

What is Apache Kafka?

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitioning LinkedIn to a fully stream-based architecture, he is now helping a number of Silicon Valley tech companies do the same thing with his new company, Confluent.

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluen...

published: 24 Oct 2017

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can...

published: 12 Mar 2018

1. Intro to Streams | Apache Kafka® Streams API

Write an app: http://kafka.apache.org/documentation/streams | The StreamsAPI of Apache Kafka is the easiest way to write mission-critical real-time applications and microservices with all the benefits of Kafka's server-side cluster technology. It allows you to build standard Java or Scala applications that are elastic, highly scalable, and fault-tolerant, and don’t require a separate processing cluster technology. Applications can be deployed on containers, VMs, or bare-metal hardware, to the cloud or on-premises.
MORE ON STREAMS
http://kafka.apache.org/documentation/streams
https://docs.confluent.io/current/streams/introduction.html
https://github.com/confluentinc/kafka-streams-examples
CONNECT
Subscribe: http://youtube.com/c/confluent?sub_confirmation=1
Site: http://confluent.io
GitH...

Apache Kafka is the most popular open source project for building real-time data pipelines and streaming applications. In this video, you will learn the key concepts of Kafka including the setup, configuration, and ingesting streaming data.
For learning more, you can follow our step-by-step tutorial below:
How To Install Apache Kafka on Ubuntu 14.04 - https://do.co/2sswlCy
To learn more about DigitalOcean: https://www.digitalocean.com/
Follow us on Twitter: https://twitter.com/digitalocean
Like us on Facebook: https://www.facebook.com/DigitalOcean
Follow us on Instagram: https://www.instagram.com/thedigitalocean
We're hiring: http://grnh.se/aicoph1

This Apache Kafka tutorial will help you master the basics of Apache Kafka including concepts of KafkaCluster, Kafka Data Model, Kafka Topic, Kafka Architecture and Use Case of Kakfa at LinkedIn. Apache Kafka is an open-source stream processing platform developed by the ApacheSoftwareFoundation written in Scala and Java.
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.
Subscribe to Simplilear...

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven archite...

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

Introduction to Apache Kafka by Joe Stein

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this ligh...

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementatio...

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk make...

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

Understanding Kafka with Legos

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into parti...

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

Introduction to Apache Kafka

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apac...

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

What is Apache Kafka?

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitionin...

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitioning LinkedIn to a fully stream-based architecture, he is now helping a number of Silicon Valley tech companies do the same thing with his new company, Confluent.

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitioning LinkedIn to a fully stream-based architecture, he is now helping a number of Silicon Valley tech companies do the same thing with his new company, Confluent.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It...

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices a...

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

1. Intro to Streams | Apache Kafka® Streams API

Write an app: http://kafka.apache.org/documentation/streams | The StreamsAPI of Apache Kafka is the easiest way to write mission-critical real-time application...

Write an app: http://kafka.apache.org/documentation/streams | The StreamsAPI of Apache Kafka is the easiest way to write mission-critical real-time applications and microservices with all the benefits of Kafka's server-side cluster technology. It allows you to build standard Java or Scala applications that are elastic, highly scalable, and fault-tolerant, and don’t require a separate processing cluster technology. Applications can be deployed on containers, VMs, or bare-metal hardware, to the cloud or on-premises.
MORE ON STREAMS
http://kafka.apache.org/documentation/streams
https://docs.confluent.io/current/streams/introduction.html
https://github.com/confluentinc/kafka-streams-examples
CONNECT
Subscribe: http://youtube.com/c/confluent?sub_confirmation=1
Site: http://confluent.io
GitHub: https://github.com/confluentinc
Facebook: https://facebook.com/confluentinc
Twitter: https://twitter.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit http://confluent.io

Write an app: http://kafka.apache.org/documentation/streams | The StreamsAPI of Apache Kafka is the easiest way to write mission-critical real-time applications and microservices with all the benefits of Kafka's server-side cluster technology. It allows you to build standard Java or Scala applications that are elastic, highly scalable, and fault-tolerant, and don’t require a separate processing cluster technology. Applications can be deployed on containers, VMs, or bare-metal hardware, to the cloud or on-premises.
MORE ON STREAMS
http://kafka.apache.org/documentation/streams
https://docs.confluent.io/current/streams/introduction.html
https://github.com/confluentinc/kafka-streams-examples
CONNECT
Subscribe: http://youtube.com/c/confluent?sub_confirmation=1
Site: http://confluent.io
GitHub: https://github.com/confluentinc
Facebook: https://facebook.com/confluentinc
Twitter: https://twitter.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit http://confluent.io

Apache Kafka is the most popular open source project for building real-time data pipelines and streaming applications. In this video, you will learn the key con...

Apache Kafka is the most popular open source project for building real-time data pipelines and streaming applications. In this video, you will learn the key concepts of Kafka including the setup, configuration, and ingesting streaming data.
For learning more, you can follow our step-by-step tutorial below:
How To Install Apache Kafka on Ubuntu 14.04 - https://do.co/2sswlCy
To learn more about DigitalOcean: https://www.digitalocean.com/
Follow us on Twitter: https://twitter.com/digitalocean
Like us on Facebook: https://www.facebook.com/DigitalOcean
Follow us on Instagram: https://www.instagram.com/thedigitalocean
We're hiring: http://grnh.se/aicoph1

Apache Kafka is the most popular open source project for building real-time data pipelines and streaming applications. In this video, you will learn the key concepts of Kafka including the setup, configuration, and ingesting streaming data.
For learning more, you can follow our step-by-step tutorial below:
How To Install Apache Kafka on Ubuntu 14.04 - https://do.co/2sswlCy
To learn more about DigitalOcean: https://www.digitalocean.com/
Follow us on Twitter: https://twitter.com/digitalocean
Like us on Facebook: https://www.facebook.com/DigitalOcean
Follow us on Instagram: https://www.instagram.com/thedigitalocean
We're hiring: http://grnh.se/aicoph1

This Apache Kafka tutorial will help you master the basics of Apache Kafka including concepts of KafkaCluster, Kafka Data Model, Kafka Topic, Kafka Architecture and Use Case of Kakfa at LinkedIn. Apache Kafka is an open-source stream processing platform developed by the ApacheSoftwareFoundation written in Scala and Java.
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.
Subscribe to Simplilearn channel for more Big Data and Hadoop Tutorials - https://www.youtube.com/user/Simplilearn?sub_confirmation=1
Check our Big Data TrainingVideoPlaylist: https://www.youtube.com/playlist?list=PLEiEAq2VkUUJqp1k-g5W1mo37urJQOdCZ
Big Data and Analytics Articles - https://www.simplilearn.com/resources/big-data-and-analytics?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
To gain in-depth knowledge of Big Data and Hadoop, check our Big Data Hadoop and SparkDeveloperCertification Training Course: https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
#bigdata #bigdatatutorialforbeginners #bigdataanalytics #bigdatahadooptutorialforbeginners #bigdatacertification #HadoopTutorial
- - - - - - - - -
About Simplilearn's Big Data and Hadoop Certification Training Course:
The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab.
Mastering real-time data processing using Spark: You will learn to do functional programming in Spark, implement Spark applications, understand parallel processing in Spark, and use Spark RDD optimization techniques. You will also learn the various interactive algorithm in Spark and use Spark SQL for creating, transforming, and querying data form.
As a part of the course, you will be required to execute real-life industry-based projects using CloudLab. The projects included are in the domains of Banking, Telecommunication, Social media, Insurance, and E-commerce. This Big Data course also prepares you for the Cloudera CCA175 certification.
- - - - - - - -
What are the course objectives of this Big Data and Hadoop Certification Training Course?
This course will enable you to:
1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, AvroSchema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
- - - - - - - - - - -
Who should take up this Big Data and Hadoop Certification Training Course?
Big Data career opportunities are on the rise, and Hadoop is quickly becoming a must-know technology for the following professionals:
1. Software Developers and Architects
2. Analytics Professionals
3. SeniorIT professionals
4. Testing and Mainframe professionals
5. Data Management Professionals
6. Business Intelligence Professionals
7. Project Managers
8. Aspiring Data Scientists
- - - - - - - -
For more updates on courses and tips follow us on:
- Facebook : https://www.facebook.com/Simplilearn
- Twitter: https://twitter.com/simplilearn
- LinkedIn: https://www.linkedin.com/company/simplilearn
- Website: https://www.simplilearn.com
Get the android app: http://bit.ly/1WlVo4u
Get the iOS app: http://apple.co/1HIO5J0

This Apache Kafka tutorial will help you master the basics of Apache Kafka including concepts of KafkaCluster, Kafka Data Model, Kafka Topic, Kafka Architecture and Use Case of Kakfa at LinkedIn. Apache Kafka is an open-source stream processing platform developed by the ApacheSoftwareFoundation written in Scala and Java.
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a "massively scalable pub/sub message queue architected as a distributed transaction log," making it highly valuable for enterprise infrastructures to process streaming data.
Subscribe to Simplilearn channel for more Big Data and Hadoop Tutorials - https://www.youtube.com/user/Simplilearn?sub_confirmation=1
Check our Big Data TrainingVideoPlaylist: https://www.youtube.com/playlist?list=PLEiEAq2VkUUJqp1k-g5W1mo37urJQOdCZ
Big Data and Analytics Articles - https://www.simplilearn.com/resources/big-data-and-analytics?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
To gain in-depth knowledge of Big Data and Hadoop, check our Big Data Hadoop and SparkDeveloperCertification Training Course: https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training?utm_campaign=Bigdata-Kafka-U4y2R3v9tlY&utm_medium=Tutorials&utm_source=youtube
#bigdata #bigdatatutorialforbeginners #bigdataanalytics #bigdatahadooptutorialforbeginners #bigdatacertification #HadoopTutorial
- - - - - - - - -
About Simplilearn's Big Data and Hadoop Certification Training Course:
The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab.
Mastering real-time data processing using Spark: You will learn to do functional programming in Spark, implement Spark applications, understand parallel processing in Spark, and use Spark RDD optimization techniques. You will also learn the various interactive algorithm in Spark and use Spark SQL for creating, transforming, and querying data form.
As a part of the course, you will be required to execute real-life industry-based projects using CloudLab. The projects included are in the domains of Banking, Telecommunication, Social media, Insurance, and E-commerce. This Big Data course also prepares you for the Cloudera CCA175 certification.
- - - - - - - -
What are the course objectives of this Big Data and Hadoop Certification Training Course?
This course will enable you to:
1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, AvroSchema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
- - - - - - - - - - -
Who should take up this Big Data and Hadoop Certification Training Course?
Big Data career opportunities are on the rise, and Hadoop is quickly becoming a must-know technology for the following professionals:
1. Software Developers and Architects
2. Analytics Professionals
3. SeniorIT professionals
4. Testing and Mainframe professionals
5. Data Management Professionals
6. Business Intelligence Professionals
7. Project Managers
8. Aspiring Data Scientists
- - - - - - - -
For more updates on courses and tips follow us on:
- Facebook : https://www.facebook.com/Simplilearn
- Twitter: https://twitter.com/simplilearn
- LinkedIn: https://www.linkedin.com/company/simplilearn
- Website: https://www.simplilearn.com
Get the android app: http://bit.ly/1WlVo4u
Get the iOS app: http://apple.co/1HIO5J0

Ben Stopford's talk "BuildingEventDriven Services with Apache Kafka and KafkaStreams" from the 2018 MicroCPH Conference.
Abstract:
Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

published: 22 May 2018

Interactive real time dashboards on data streams using Kafka, Druid, and Superset

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer.
Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases.
This talk presents an open source real time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of ...

with Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and KafkaStreams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used...

Apache Kafka, KSQL, and Streaming Analytics

What are you doing to meet the increasing demand for real-time analytics? DeveloperChampion Bjoern Rost suggests you'd do well to take advantage of Apache Kafka and KSQL in this interview recorded at OracleCode 2018 in Chicago.
https://developer.oracle.com/
https://cloud.oracle.com/en_US/tryit

Introducing AMQ Streams—data streaming with Apache Kafka

Apache Kafka has emerged as a leading platform for building real-time data pipelines. It provides support for high-throughput/low-latency messaging, as well as sophisticated development options that cover all the stages of a distributed data-streaming pipeline—from ingestion to processing.
In this session, we'll introduce and demonstrate AMQStreams, an enterprise-grade distribution of Apache Kafka. We'll showcase how AMQ Streams can be deployed on OpenShift to simplify the task of configuring, deploying, and managing Kafka clusters and Kafka-based applications at scale.
With AMQ Streams now joining Broker and Interconnect, AMQ now offers a rich suite of messaging technologies covering every type of connectivity challenge, for example, IoT data ingestion, enterprise application integrati...

Ben Stopford's talk "BuildingEventDriven Services with Apache Kafka and KafkaStreams" from the 2018 MicroCPH Conference.
Abstract:
Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

Ben Stopford's talk "BuildingEventDriven Services with Apache Kafka and KafkaStreams" from the 2018 MicroCPH Conference.
Abstract:
Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

Interactive real time dashboards on data streams using Kafka, Druid, and Superset

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness....

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer.
Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases.
This talk presents an open source real time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of Kafka with Druid, which enables immediate exploration and provides low-latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of complete stack, discuss its key features, and discuss performance characteristics from real-world use cases.
Speaker: Nishant Bangarwa, Software Engineer, Hortonworks

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer.
Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases.
This talk presents an open source real time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of Kafka with Druid, which enables immediate exploration and provides low-latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of complete stack, discuss its key features, and discuss performance characteristics from real-world use cases.
Speaker: Nishant Bangarwa, Software Engineer, Hortonworks

** KafkaOnlineTraining : https://www.edureka.co/apache-kafka *
In this Kafka SparkStreaming video, we are demonstrating how Spark streaming works with Kafka. In this video, we have discussed Apache Kafka & Apache Spark briefly. Finally, we have explained the integration of Kafka & Spark Streaming. Topics covered in this Kafka Spark Tutorial video are:
1. What is Kafka?
2. Kafka Components
3. Kafka architecture
4. What is Spark?
5. Spark Component
6. Kafka Spark Integration
7. Kafka Spark Streaming Project
As mentioned in the video, you can go through these Kafka & Spark videos:
Kafka Tutorial: https://www.youtube.com/watch?v=hyJZP-rgooc
Spark Tutorial: https://www.youtube.com/watch?v=9mELEARcxJo
Spark Streaming: https://www.youtube.com/watch?v=uD_q4Rm4i2Q
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Kafka playlist here: https://goo.gl/jEZfLj
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led Online Course, 40 hours of assignment and 30 hours of project work
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. At the end of the training you will have to undergo a 2-hour LIVE Practical Exam based on which we will provide you a Grade and a VerifiableCertificate!
- - - - - - - - - - - - - -
About the Course
Edureka’s Apache Kafka certification training is designed to help you become a Kafka developer. During this course, our expert Kafka instructors will help you:
1. Learn Kafka and its components
2. Set up an end to end Kafka cluster along with Hadoop and YARN cluster
3. Integrate Kafka with real time streaming systems like Spark & Storm
4. Describe the basic and advanced features involved in designing and developing a high throughput messaging system
5. Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
6. Get insights of Kafka Producer & Consumer APIs
7. Understand Kafka Stream APIs
8. Work on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and MessagingSystems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Kafka?
Kafka is used heavily in the Big Data space as a reliable way to ingest and move large amounts of data very quickly.
​LinkedIn, Yahoo, Twitter, Netflix, Uber, Goldman Sachs,PayPal, Airbnb​ ​​& other fortune 500 companies use Kafka.
The average salary of a Software Engineer with Apache Kafka skill is $87,500 per year. (Payscale.com salary data).
- - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

** KafkaOnlineTraining : https://www.edureka.co/apache-kafka *
In this Kafka SparkStreaming video, we are demonstrating how Spark streaming works with Kafka. In this video, we have discussed Apache Kafka & Apache Spark briefly. Finally, we have explained the integration of Kafka & Spark Streaming. Topics covered in this Kafka Spark Tutorial video are:
1. What is Kafka?
2. Kafka Components
3. Kafka architecture
4. What is Spark?
5. Spark Component
6. Kafka Spark Integration
7. Kafka Spark Streaming Project
As mentioned in the video, you can go through these Kafka & Spark videos:
Kafka Tutorial: https://www.youtube.com/watch?v=hyJZP-rgooc
Spark Tutorial: https://www.youtube.com/watch?v=9mELEARcxJo
Spark Streaming: https://www.youtube.com/watch?v=uD_q4Rm4i2Q
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Kafka playlist here: https://goo.gl/jEZfLj
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led Online Course, 40 hours of assignment and 30 hours of project work
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. At the end of the training you will have to undergo a 2-hour LIVE Practical Exam based on which we will provide you a Grade and a VerifiableCertificate!
- - - - - - - - - - - - - -
About the Course
Edureka’s Apache Kafka certification training is designed to help you become a Kafka developer. During this course, our expert Kafka instructors will help you:
1. Learn Kafka and its components
2. Set up an end to end Kafka cluster along with Hadoop and YARN cluster
3. Integrate Kafka with real time streaming systems like Spark & Storm
4. Describe the basic and advanced features involved in designing and developing a high throughput messaging system
5. Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
6. Get insights of Kafka Producer & Consumer APIs
7. Understand Kafka Stream APIs
8. Work on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and MessagingSystems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Kafka?
Kafka is used heavily in the Big Data space as a reliable way to ingest and move large amounts of data very quickly.
​LinkedIn, Yahoo, Twitter, Netflix, Uber, Goldman Sachs,PayPal, Airbnb​ ​​& other fortune 500 companies use Kafka.
The average salary of a Software Engineer with Apache Kafka skill is $87,500 per year. (Payscale.com salary data).
- - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

with Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reactio...

with Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and KafkaStreams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.

with Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and KafkaStreams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.

Apache Kafka, KSQL, and Streaming Analytics

What are you doing to meet the increasing demand for real-time analytics? DeveloperChampion Bjoern Rost suggests you'd do well to take advantage of Apache Kaf...

What are you doing to meet the increasing demand for real-time analytics? DeveloperChampion Bjoern Rost suggests you'd do well to take advantage of Apache Kafka and KSQL in this interview recorded at OracleCode 2018 in Chicago.
https://developer.oracle.com/
https://cloud.oracle.com/en_US/tryit

What are you doing to meet the increasing demand for real-time analytics? DeveloperChampion Bjoern Rost suggests you'd do well to take advantage of Apache Kafka and KSQL in this interview recorded at OracleCode 2018 in Chicago.
https://developer.oracle.com/
https://cloud.oracle.com/en_US/tryit

Introducing AMQ Streams—data streaming with Apache Kafka

Apache Kafka has emerged as a leading platform for building real-time data pipelines. It provides support for high-throughput/low-latency messaging, as well as ...

Apache Kafka has emerged as a leading platform for building real-time data pipelines. It provides support for high-throughput/low-latency messaging, as well as sophisticated development options that cover all the stages of a distributed data-streaming pipeline—from ingestion to processing.
In this session, we'll introduce and demonstrate AMQStreams, an enterprise-grade distribution of Apache Kafka. We'll showcase how AMQ Streams can be deployed on OpenShift to simplify the task of configuring, deploying, and managing Kafka clusters and Kafka-based applications at scale.
With AMQ Streams now joining Broker and Interconnect, AMQ now offers a rich suite of messaging technologies covering every type of connectivity challenge, for example, IoT data ingestion, enterprise application integration, secure cross-data center connectivity, and data streaming. In this session, we'll also compare the different AMQ messaging engines and highlight their respective sweet-spots.
Learn more: https://agenda.summit.redhat.com

Apache Kafka has emerged as a leading platform for building real-time data pipelines. It provides support for high-throughput/low-latency messaging, as well as sophisticated development options that cover all the stages of a distributed data-streaming pipeline—from ingestion to processing.
In this session, we'll introduce and demonstrate AMQStreams, an enterprise-grade distribution of Apache Kafka. We'll showcase how AMQ Streams can be deployed on OpenShift to simplify the task of configuring, deploying, and managing Kafka clusters and Kafka-based applications at scale.
With AMQ Streams now joining Broker and Interconnect, AMQ now offers a rich suite of messaging technologies covering every type of connectivity challenge, for example, IoT data ingestion, enterprise application integration, secure cross-data center connectivity, and data streaming. In this session, we'll also compare the different AMQ messaging engines and highlight their respective sweet-spots.
Learn more: https://agenda.summit.redhat.com

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

published: 07 Nov 2015

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

published: 23 Feb 2017

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier ...

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can...

published: 12 Mar 2018

Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to a...

We all know Microservices come in all shapes and sizes. This include performing a range of tasks from handling user interaction, all the way to do the heavy lifting for complex compute operations, exposing different APIs, such as HTTP based web services, RPC technologies such as Avro or using queuing technologies such as RabbitMQ. While Microservices are also scalable on their own merits, but what if the volume of data is just too big?
Apache Kafka is a highly scalable messaging system that was built to handle the internal messaging of one of the largest websites in the world, and support Big Data workloads in addition to its pub-sub capabilities.
In this talk we will see how Kafka and its surrounding eco-system can fuel Microservices that need to scale to handle large amounts of data.

published: 25 Sep 2017

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

This talk was recorded at ScalaDaysNew York, 2016. Follow along on Twitter @scaladays and on the website for more information http://scaladays.org/.
Abstract:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and r...

published: 17 Jun 2016

I can't believe it's not a queue: Using Kafka with Spring

Recorded at SpringOne Platform 2016.
Speaker: Joe Kutner, Confluent
Slides: http://www.slideshare.net/SpringCentral/i-cant-believe-its-not-a-queue-using-kafka-with-spring
Your existing message system is great, until it gets overloaded. Then what? That's when you should try Kafka.
Kafka is designed to be resilient. It takes the stress out of moving from a Spring monolith into a scalable system of microservices. Since you can capture every event that happens in your app, it's great for logging. You can even use Kafka's distributed, ordered log to simulate production load in your staging environment.
Come learn about Kafka, where it fits in your Spring app, and how to make it do things message queues simply can't.

published: 06 Feb 2017

Kafka as a Message Queue: can you do it, and should you do it? by Adam Warski

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and ...

published: 09 Nov 2017

Lessons learned form Kafka in production (Tim Berglund, Confluent)

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more...

published: 04 Sep 2017

Distributed Commit Logs with Apache Kafka by James Ward

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log providing a traditional publish / subscribe interface. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it.
James Ward is a PrincipalPlatformEvangelist at Salesforce.com.
[DME-3874]

"I ♥ Logs: Apache Kafka, Stream Processing, and Real-time Data" - Jay Kreps of LinkedIn
Colloquium on Computer SystemsSeminarSeries (EE380) presents the current research in design, implementation, analysis, and use of computer systems. Topics range from integrated circuits to operating systems and programming languages. It is free and open to the public, with new lectures each week.
Learn more: http://bit.ly/WinYX5

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluen...

Speaker: Sreekanth Ramakrishnan, DataEngineeringTeam @ Lazada
Apache Kafka, has become one of the center pieces of most of enterprise big data stacks. We will be talking about things which we have learnt while getting our Kafka cluster production ready. We will also be talking about what happens in a Kafka cluster when things go wrong, data loss scenarios, things to watch out when you deploy Apache Kafka.
EventPage: http://www.meetup.com/BigData-Hadoop-SG/events/230062826/
Produced by Engineers.SG
Help us caption & translate this video!
http://amara.org/v/IMB5/

published: 11 Apr 2016

Николай Рекубратский — KAFKA - масштабируемый брокер сообщений

Apache Kafka Tutorials For Beginners

Apache Kafka is an open-source stream processing platform written in Scala & Java programming language. Kafka is mainly used for handling real time data feed. The main benefit of using Kafka is it offers real time data, easy scalability and also easy to distribute. In this Kafka tutorial for beginners, you will learn introduction to Kafka and how to install Kafka.
Check out our brand new Kickstarter project! This time we are creating an immersive DevOps Engineer E-degree Program. Hurry, grab the Early Bird offers before they get over: https://kck.st/2HekFLk
Thank you for watching! We’d love to know your thoughts in the comments section below. Also, don’t forget to hit the ‘like’ button and ‘subscribe’ to ‘Eduonix Learning Solutions’ for regular updates. https://goo.gl/BCmVLG
Follow Ed...

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven archite...

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementatio...

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk make...

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It...

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices a...

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

We all know Microservices come in all shapes and sizes. This include performing a range of tasks from handling user interaction, all the way to do the heavy lif...

We all know Microservices come in all shapes and sizes. This include performing a range of tasks from handling user interaction, all the way to do the heavy lifting for complex compute operations, exposing different APIs, such as HTTP based web services, RPC technologies such as Avro or using queuing technologies such as RabbitMQ. While Microservices are also scalable on their own merits, but what if the volume of data is just too big?
Apache Kafka is a highly scalable messaging system that was built to handle the internal messaging of one of the largest websites in the world, and support Big Data workloads in addition to its pub-sub capabilities.
In this talk we will see how Kafka and its surrounding eco-system can fuel Microservices that need to scale to handle large amounts of data.

We all know Microservices come in all shapes and sizes. This include performing a range of tasks from handling user interaction, all the way to do the heavy lifting for complex compute operations, exposing different APIs, such as HTTP based web services, RPC technologies such as Avro or using queuing technologies such as RabbitMQ. While Microservices are also scalable on their own merits, but what if the volume of data is just too big?
Apache Kafka is a highly scalable messaging system that was built to handle the internal messaging of one of the largest websites in the world, and support Big Data workloads in addition to its pub-sub capabilities.
In this talk we will see how Kafka and its surrounding eco-system can fuel Microservices that need to scale to handle large amounts of data.

This talk was recorded at ScalaDaysNew York, 2016. Follow along on Twitter @scaladays and on the website for more information http://scaladays.org/.
Abstract:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and reliable messaging
- Evolve message payloads while maintaining backwards and forwards compatibility.

This talk was recorded at ScalaDaysNew York, 2016. Follow along on Twitter @scaladays and on the website for more information http://scaladays.org/.
Abstract:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and reliable messaging
- Evolve message payloads while maintaining backwards and forwards compatibility.

Recorded at SpringOne Platform 2016.
Speaker: Joe Kutner, Confluent
Slides: http://www.slideshare.net/SpringCentral/i-cant-believe-its-not-a-queue-using-kafka-with-spring
Your existing message system is great, until it gets overloaded. Then what? That's when you should try Kafka.
Kafka is designed to be resilient. It takes the stress out of moving from a Spring monolith into a scalable system of microservices. Since you can capture every event that happens in your app, it's great for logging. You can even use Kafka's distributed, ordered log to simulate production load in your staging environment.
Come learn about Kafka, where it fits in your Spring app, and how to make it do things message queues simply can't.

Recorded at SpringOne Platform 2016.
Speaker: Joe Kutner, Confluent
Slides: http://www.slideshare.net/SpringCentral/i-cant-believe-its-not-a-queue-using-kafka-with-spring
Your existing message system is great, until it gets overloaded. Then what? That's when you should try Kafka.
Kafka is designed to be resilient. It takes the stress out of moving from a Spring monolith into a scalable system of microservices. Since you can capture every event that happens in your app, it's great for logging. You can even use Kafka's distributed, ordered log to simulate production load in your staging environment.
Come learn about Kafka, where it fits in your Spring app, and how to make it do things message queues simply can't.

Kafka as a Message Queue: can you do it, and should you do it? by Adam Warski

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge proc...

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.
We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.
Kmq is open-source and available at https://github.com/softwaremill/kmq.
# AdamWarskiI am one of the co-founders of SoftwareMill, where I code mainly using Scala and other interesting technologies. I am involved in open-source projects, such as Macwire, Supler, ElasticMQ and others. I have been a speaker at major conferences, such as JavaOne, Devoxx and ScalaDays.
Apart from writing closed- and open-source software, in my free time I try to read the Internet on various (functional) programming-related subjects, any ideas or insights usually end up on my blog: http://www.warski.org/blog

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.
We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.
Kmq is open-source and available at https://github.com/softwaremill/kmq.
# AdamWarskiI am one of the co-founders of SoftwareMill, where I code mainly using Scala and other interesting technologies. I am involved in open-source projects, such as Macwire, Supler, ElasticMQ and others. I have been a speaker at major conferences, such as JavaOne, Devoxx and ScalaDays.
Apart from writing closed- and open-source software, in my free time I try to read the Internet on various (functional) programming-related subjects, any ideas or insights usually end up on my blog: http://www.warski.org/blog

Lessons learned form Kafka in production (Tim Berglund, Confluent)

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep ...

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.

Distributed Commit Logs with Apache Kafka by James Ward

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log providing a traditional publish / subscribe interface. Now open source t...

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log providing a traditional publish / subscribe interface. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it.
James Ward is a PrincipalPlatformEvangelist at Salesforce.com.
[DME-3874]

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log providing a traditional publish / subscribe interface. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it.
James Ward is a PrincipalPlatformEvangelist at Salesforce.com.
[DME-3874]

"I ♥ Logs: Apache Kafka, Stream Processing, and Real-time Data" - Jay Kreps of LinkedIn
Colloquium on Computer SystemsSeminarSeries (EE380) presents the current research in design, implementation, analysis, and use of computer systems. Topics range from integrated circuits to operating systems and programming languages. It is free and open to the public, with new lectures each week.
Learn more: http://bit.ly/WinYX5

"I ♥ Logs: Apache Kafka, Stream Processing, and Real-time Data" - Jay Kreps of LinkedIn
Colloquium on Computer SystemsSeminarSeries (EE380) presents the current research in design, implementation, analysis, and use of computer systems. Topics range from integrated circuits to operating systems and programming languages. It is free and open to the public, with new lectures each week.
Learn more: http://bit.ly/WinYX5

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Keynote: Apache Kafka and the Rise of the StreamingPlatform - Neha Narkhede, Co-Founder & CTO, Confluent
Streaming platforms are emerging as a new trend. However, what exactly is a streaming platform? With Apache Kafka at the core, it’s an entirely new perspective on managing the flow of data. Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration, a streaming platform is a new way to stream, store and process data across the business. In this keynote, Neha will share examples of Kafka in action and why Kafka is becoming a central nervous system that ties together the modern, digital business.
About Neha Narkhede
Neha Narkhede is co-founder and CTO at Confluent, the company behind the popular Apache Kafka streaming platform. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Speaker: Sreekanth Ramakrishnan, DataEngineeringTeam @ Lazada
Apache Kafka, has become one of the center pieces of most of enterprise big data stacks. We will be talking about things which we have learnt while getting our Kafka cluster production ready. We will also be talking about what happens in a Kafka cluster when things go wrong, data loss scenarios, things to watch out when you deploy Apache Kafka.
EventPage: http://www.meetup.com/BigData-Hadoop-SG/events/230062826/
Produced by Engineers.SG
Help us caption & translate this video!
http://amara.org/v/IMB5/

Speaker: Sreekanth Ramakrishnan, DataEngineeringTeam @ Lazada
Apache Kafka, has become one of the center pieces of most of enterprise big data stacks. We will be talking about things which we have learnt while getting our Kafka cluster production ready. We will also be talking about what happens in a Kafka cluster when things go wrong, data loss scenarios, things to watch out when you deploy Apache Kafka.
EventPage: http://www.meetup.com/BigData-Hadoop-SG/events/230062826/
Produced by Engineers.SG
Help us caption & translate this video!
http://amara.org/v/IMB5/

Apache Kafka is an open-source stream processing platform written in Scala & Java programming language. Kafka is mainly used for handling real time data feed. The main benefit of using Kafka is it offers real time data, easy scalability and also easy to distribute. In this Kafka tutorial for beginners, you will learn introduction to Kafka and how to install Kafka.
Check out our brand new Kickstarter project! This time we are creating an immersive DevOps Engineer E-degree Program. Hurry, grab the Early Bird offers before they get over: https://kck.st/2HekFLk
Thank you for watching! We’d love to know your thoughts in the comments section below. Also, don’t forget to hit the ‘like’ button and ‘subscribe’ to ‘Eduonix Learning Solutions’ for regular updates. https://goo.gl/BCmVLG
Follow Eduonix on other social networks:
■ Facebook: http://bit.ly/2nL2p59
■ Linkedin: http://bit.ly/2nKWhKa
■ Instagram: http://bit.ly/2nL8TRu | @eduonix
■ Twitter: http://bit.ly/2eKnxq8

Apache Kafka is an open-source stream processing platform written in Scala & Java programming language. Kafka is mainly used for handling real time data feed. The main benefit of using Kafka is it offers real time data, easy scalability and also easy to distribute. In this Kafka tutorial for beginners, you will learn introduction to Kafka and how to install Kafka.
Check out our brand new Kickstarter project! This time we are creating an immersive DevOps Engineer E-degree Program. Hurry, grab the Early Bird offers before they get over: https://kck.st/2HekFLk
Thank you for watching! We’d love to know your thoughts in the comments section below. Also, don’t forget to hit the ‘like’ button and ‘subscribe’ to ‘Eduonix Learning Solutions’ for regular updates. https://goo.gl/BCmVLG
Follow Eduonix on other social networks:
■ Facebook: http://bit.ly/2nL2p59
■ Linkedin: http://bit.ly/2nKWhKa
■ Instagram: http://bit.ly/2nL8TRu | @eduonix
■ Twitter: http://bit.ly/2eKnxq8

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

10:52

How Does Apache Kafka Work? [Diagram]

www.datameer.com
It's clear how to represent a data file, but it's not necessarily clear h...

Introduction to Apache Kafka by Joe Stein

See the full post here: http://www.hakkalabs.co/articles/apache-kafka
Apache Kafka is a commit log for your entire data center and infrastructure. In this lightning talk, Joe Stein, founder of Big DataOpen SourceSecurityLLC, gives a brief introduction to Kafka and talks about the producers, consumers, and client libraries it has to offer.
This talk was given at the Apache Kafka NYC meetup at Tapad.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

39:01

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to rea...

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

48:20

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

11:48

Understanding Kafka with Legos

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and ...

Understanding Kafka with Legos

I show how Apache Kafka works with Legos. I show how a publish/subscribe system works and how topics are used. Then, I show Kafka uses breaks up data into partitions and uses a commit log. Finally, I talk about how Kafka frees up space with log compactions and the methods for choosing which data is deleted.

3:55

Introduction to Apache Kafka

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation...

Introduction to Apache Kafka

This is a tutorial on Apache Kafka presented by QBurst.
For more details and installation steps, you can refer to our blog
http://blog.qburst.com/2014/06/apache-kafka
For any other queries you can contact us at marketing@qburst.com

Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

What is Apache Kafka?

www.datameer.com
Creator of Apache Kafka, Jay Kreps, shares how Kafka first came to fruition. After spending the last five years building Kafka and transitioning LinkedIn to a fully stream-based architecture, he is now helping a number of Silicon Valley tech companies do the same thing with his new company, Confluent.

10:41

What is Apache Kafka? | Tech Primers

This video covers what is apache kafka and its architecture
Twitter: https://twitter.com/...

History

Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. In November 2014, several engineers who built Kafka at LinkedIn created a new company named Confluent with a focus on Kafka.

Enterprises that use Kafka

The following is a list of notable enterprises that have used or are using Kafka:

- Disk space... - Bandwidth ... - Apache ... The server based on Apache is because it is an old acquaintance of mine and I use it on my local server for tests, I do not know the litestep and I do not know how much difference or effect it will affect me in my work, being working in apache locally and remotely with litestep ; for this reason I prefer to use Apache ... ....

The bats fell silent, however, as Eastwood’s Division I recruit AshleyHitchcock stifled the Apaches on the scoreboard and the unbeaten Eagles advanced to the regional finals with a 7-0 blanking of Fairview at Findlay High School... The opportunity was one of three on the day for Fairview with runners at second and third with one out in which the Apaches were unable to score....

Ben Stopford's talk "BuildingEventDriven Services with Apache Kafka and KafkaStreams" from the 2018 MicroCPH Conference.
Abstract:
Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

35:53

Interactive real time dashboards on data streams using Kafka, Druid, and Superset

When interacting with analytics dashboards, in order to achieve a smooth user experience, ...

Interactive real time dashboards on data streams using Kafka, Druid, and Superset

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer.
Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases.
This talk presents an open source real time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of Kafka with Druid, which enables immediate exploration and provides low-latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of complete stack, discuss its key features, and discuss performance characteristics from real-world use cases.
Speaker: Nishant Bangarwa, Software Engineer, Hortonworks

** KafkaOnlineTraining : https://www.edureka.co/apache-kafka *
In this Kafka SparkStreaming video, we are demonstrating how Spark streaming works with Kafka. In this video, we have discussed Apache Kafka & Apache Spark briefly. Finally, we have explained the integration of Kafka & Spark Streaming. Topics covered in this Kafka Spark Tutorial video are:
1. What is Kafka?
2. Kafka Components
3. Kafka architecture
4. What is Spark?
5. Spark Component
6. Kafka Spark Integration
7. Kafka Spark Streaming Project
As mentioned in the video, you can go through these Kafka & Spark videos:
Kafka Tutorial: https://www.youtube.com/watch?v=hyJZP-rgooc
Spark Tutorial: https://www.youtube.com/watch?v=9mELEARcxJo
Spark Streaming: https://www.youtube.com/watch?v=uD_q4Rm4i2Q
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Kafka playlist here: https://goo.gl/jEZfLj
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led Online Course, 40 hours of assignment and 30 hours of project work
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. At the end of the training you will have to undergo a 2-hour LIVE Practical Exam based on which we will provide you a Grade and a VerifiableCertificate!
- - - - - - - - - - - - - -
About the Course
Edureka’s Apache Kafka certification training is designed to help you become a Kafka developer. During this course, our expert Kafka instructors will help you:
1. Learn Kafka and its components
2. Set up an end to end Kafka cluster along with Hadoop and YARN cluster
3. Integrate Kafka with real time streaming systems like Spark & Storm
4. Describe the basic and advanced features involved in designing and developing a high throughput messaging system
5. Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
6. Get insights of Kafka Producer & Consumer APIs
7. Understand Kafka Stream APIs
8. Work on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and MessagingSystems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Kafka?
Kafka is used heavily in the Big Data space as a reliable way to ingest and move large amounts of data very quickly.
​LinkedIn, Yahoo, Twitter, Netflix, Uber, Goldman Sachs,PayPal, Airbnb​ ​​& other fortune 500 companies use Kafka.
The average salary of a Software Engineer with Apache Kafka skill is $87,500 per year. (Payscale.com salary data).
- - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

with Lucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and KafkaStreams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.

Introduction to Apache Kafka by James Ward

Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Kafka tutorial video gives an introduction to Kafka, Kafka architecture, Kafka cluster setup and hand's on session. This Kafka video is ideal for beginners.
To attend a live class, click here: http://goo.gl/BWpYsE
This video will help you learn:
• What is Apache Kafka ?
• Architecture of Kafka
• Multiple ways of setting Kafka cluster
• Comparing Kafka with other messaging systems
• Hands-On : Getting started with Kafka
The topics related to ‘Apache Kafka’ have been widely covered in our course.
For more information, please write back to us at sales@edureka.co
Call us at US: 1800 275 9730 (toll free) or India: +91-8880862004

39:01

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to rea...

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka

Neha Narkhede talks about the experience at LinkedIn moving from batch-oriented ETL to real-time streams using Apache Kafka and how the design and implementation of Kafka was driven by this goal of acting as a real-time platform for event data. She covers some of the challenges of scaling Kafka to hundreds of billions of events per day at Linkedin, supporting thousands of engineers, etc.
Download the slides & audio at InfoQ: http://bit.ly/2ldN6P0
This presentation was recorded at QCon San Francisco 2016. The next QCon is in London, March 5-7, 2018. Check out the tracks and speakers: http://bit.ly/2hxsoN1
For more awesome presentations on innovator and early adopter topics check out InfoQ’s selection of talks from conferences worldwide: http://bit.ly/2lRQCll

53:03

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

Event Driven Services come in many shapes and sizes from tiny functions that dip into an e...

Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

EventDriven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the KafkaStreamsAPI and it’s transactional guarantees.
# Ben Stopford
Ben is an engineer and architect working on the Apache Kafka CoreTeam at Confluent Inc (the commercial company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for a large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.

( Apache KafkaTraining: https://www.edureka.co/apache-kafka )
This Apache Kafka Tutorial video will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture..So, the topics which we will be discussing in this Apache Kafka Tutorial are:
1. Need of MessagingSystem
2. What isKafka?
3. Kafka Features
4. Kafka Components
5. Kafka architecture
6. Installing Kafka
7. Working with SingleNode Single BrokerCluster
Subscribe to our channel to get video updates. Hit the subscribe button above.
Check our complete Hadoop playlist here: https://goo.gl/hzUO0m
- - - - - - - - - - - - - -
How it Works?
1. This is a 5 WeekInstructor led OnlineCourse, with assignments and project work.
2. We have a 24x7 One-on-One LIVETechnical Support to help you with any problems you might face or any clarifications you may require during the course.
3. Edureka certifies you as an Apache Kafka expert based on the project reviewed by our expert panel.
- - - - - - - - - - - - - -
About the Course
Apache Kafka Certification Training is designed to provide you with the knowledge and skills to become a successful Kafka Big DataDeveloper. The training encompasses the fundamental concepts (such as Kafka Cluster and Kafka API) of Kafka and covers the advanced topics (such as Kafka Connect, Kafka streams, Kafka Integration with Hadoop, Storm and Spark) thereby enabling you to gain expertise in Apache Kafka.
After the completion of Real-Time Analytics with Apache Kafka course at Edureka, you should be able to:
Learn Kafka and its components
Set up an end to end Kafka cluster along with Hadoop and YARN cluster
Integrate Kafka with real time streaming systems like Spark & Storm
Describe the basic and advanced features involved in designing and developing a high throughput messaging system
Use Kafka to produce and consume messages from various sources including real time streaming sources like Twitter
Get insights of Kafka API
Get an insights of Kafka API
Understand Kafka StreamAPIsWork on a real-life project, ‘Implementing Twitter Streaming with Kafka, Flume, Hadoop & Storm
- - - - - - - - - - - - - -
Who should go for this course?
This course is designed for professionals who want to learn Kafka techniques and wish to apply it on Big Data. It is highly recommended for:
Developers, who want to gain acceleration in their career as a "Kafka Big Data Developer"
Testing Professionals, who are currently involved in Queuing and Messaging Systems
Big Data Architects, who like to include Kafka in their ecosystem
Project Managers, who are working on projects related to Messaging Systems
Admins, who want to gain acceleration in their careers as a "Apache Kafka Administrator
- - - - - - - - - - - - - -
Why Learn Apache Kafka?
Kafka training helps you gain expertise in Kafka Architecture, Installation, Configuration, PerformanceTuning, Kafka Client APIs like Producer, Consumer and Stream APIs, Kafka Administration, Kafka Connect API and Kafka Integration with Hadoop, Storm and Spark using Twitter Streaming use case.
- - - - - - - - - - - - - -
Please write back to us at sales@edureka.co or call us at +91 88808 62004 for more information.
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
CustomerReview:
Michael Harkins, System Architect, Hortonworks says: “The courses are top rate. The best part is live instruction, with playback. But my favourite feature is viewing a previous class. Also, they are always there to answer questions, and prompt when you open an issue if you are having any trouble. Added bonus ~ you get lifetime access to the course you took!!! ~ This is the killer education app...I've take two courses, and I'm taking two more.”

50:55

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 year...

Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional RESTAPIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
GuidoSchmutz works for the OraclePlatinumPartner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and ApplicationIntegration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and FastData solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for FusionMiddleware and SOA and a regular speaker at international conferences.

Developing Real-Time Data Pipelines with Apache Kafka

Speaker: Joe SteinBig DataTrack
Slides: http://www.slideshare.net/SpringCentral/developing-realtime-data-pipelines-with-apache-kafka-53201942
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.

We all know Microservices come in all shapes and sizes. This include performing a range of tasks from handling user interaction, all the way to do the heavy lifting for complex compute operations, exposing different APIs, such as HTTP based web services, RPC technologies such as Avro or using queuing technologies such as RabbitMQ. While Microservices are also scalable on their own merits, but what if the volume of data is just too big?
Apache Kafka is a highly scalable messaging system that was built to handle the internal messaging of one of the largest websites in the world, and support Big Data workloads in addition to its pub-sub capabilities.
In this talk we will see how Kafka and its surrounding eco-system can fuel Microservices that need to scale to handle large amounts of data.

48:20

Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner

This talk was recorded at ScalaDaysNew York, 2016. Follow along on Twitter @scaladays and on the website for more information http://scaladays.org/.
Abstract:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and reliable messaging
- Evolve message payloads while maintaining backwards and forwards compatibility.

I can't believe it's not a queue: Using Kafka with Spring

Recorded at SpringOne Platform 2016.
Speaker: Joe Kutner, Confluent
Slides: http://www.slideshare.net/SpringCentral/i-cant-believe-its-not-a-queue-using-kafka-with-spring
Your existing message system is great, until it gets overloaded. Then what? That's when you should try Kafka.
Kafka is designed to be resilient. It takes the stress out of moving from a Spring monolith into a scalable system of microservices. Since you can capture every event that happens in your app, it's great for logging. You can even use Kafka's distributed, ordered log to simulate production load in your staging environment.
Come learn about Kafka, where it fits in your Spring app, and how to make it do things message queues simply can't.

47:44

Kafka as a Message Queue: can you do it, and should you do it? by Adam Warski

Using Kafka's offset commit mechanism we can implement a message processing system with at...

Kafka as a Message Queue: can you do it, and should you do it? by Adam Warski

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.
We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.
Kmq is open-source and available at https://github.com/softwaremill/kmq.
# AdamWarskiI am one of the co-founders of SoftwareMill, where I code mainly using Scala and other interesting technologies. I am involved in open-source projects, such as Macwire, Supler, ElasticMQ and others. I have been a speaker at major conferences, such as JavaOne, Devoxx and ScalaDays.
Apart from writing closed- and open-source software, in my free time I try to read the Internet on various (functional) programming-related subjects, any ideas or insights usually end up on my blog: http://www.warski.org/blog

45:06

Lessons learned form Kafka in production (Tim Berglund, Confluent)

Many developers have already wrapped their minds around the basic architecture and APIs of...

Lessons learned form Kafka in production (Tim Berglund, Confluent)

Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.

54:51

Distributed Commit Logs with Apache Kafka by James Ward

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log pr...

Distributed Commit Logs with Apache Kafka by James Ward

Apache Kafka was created at LinkedIn as a resilient and scalable distributed commit log providing a traditional publish / subscribe interface. Now open source through Apache, Kafka is being used by numerous large enterprises for a variety of use cases. This session will introduce the basics of Kafka and walk through some code examples that will show how to begin using it.
James Ward is a PrincipalPlatformEvangelist at Salesforce.com.
[DME-3874]