SlideShare a Scribd company logo

Building Streaming Data Applications Using Apache Kafka

•
•
Slim Baltagi
Slim Baltagi

Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: 1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why? 2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka

Related slideshows

Kafka presentation
Kafka presentationKafka presentation
Kafka presentation
 
Consumer offset management in Kafka
Consumer offset management in KafkaConsumer offset management in Kafka
Consumer offset management in Kafka
 
Apache Kafka® Security Overview
Apache Kafka® Security OverviewApache Kafka® Security Overview
Apache Kafka® Security Overview
 
1 of 50
Download to read offline
Los Angeles, California August 5th 2017 Slim Baltagi Building Streaming Data Applications Using Apache Kafka
Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 2
Batch data Streaming data 3
Stream Processor Destination Systems Event Streams Collector Apps Sensors Devices Other Sources Sourcing & Integration Analytics & Processing Serving & Consuming 4 1. A typical Streaming Data Application Event Streams Broker Event Streams Processor Destination Systems Event Streams Collectors Apps Sensors Databases Other Source Systems A very simplified diagram!
Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 5
2. Apache Kafka as a platform for building and running streaming data applications ØApache Kafka is an open source streaming data platform (a new category of software!) • to import event streams from other source data systems into Kafka and export event streams from Kafka to destination data systems • to transport and store event streams • to process event streams live as they occur. 6
2.1 Kafka Core: Event Streams Transport and Storage 2.1.1 What is Kafka Core? 2.1.2 Before Kafka Core? 2.1.3 Why Kafka Core? 7
2.1.1 What is Kafka Core? Ø Kafka is a software written in Scala and Java and originally developed by Linkedin in 2010. Ø It was open sourced as an apache project in 2011 and became a Top Level Project in 2012. Ø After 7 years, it is graduating to version 1.0 in October 2017!! Ø Kafka Core is an enterprise messaging system to: • publish event streams • subscribe to event streams • store event streams Ø Kafka Core is the the ‘digital nervous system’ connecting all enterprise data and systems of many notable companies. Ø Diverse and rapidly growing user base across many industries and verticals. 8
2.1.2 Before Kafka Core? 9 Ø Before Kafka Core, Linkedin had to build many custom data pipelines, for streaming and queueing data, that use point to point communication and need to be constantly scaled individually. Total connections = N producers * M consumers Search Security Fraud Detection Application User Tracking Operational Logs Operational Metrics Hadoop Search Monitoring Data Warehouse Espresso Cassandra Oracle
2.1.2 Before Kafka Core? 10 Ø Traditional enterprise message systems such as RabbitMQ, Apache ActiveMQ, IBM WebSphere MQ, TIBCO EMS could not help because of these limitations: • They can’t accommodate the web-scale requirements of Linkedin • Producers and consumers are really coupled from a performance perspective because of the ‘slow consumer problem’. • Messages are sent into a central message spool and stored only until they are processed, acknowledged and then they are deleted. Ø Linkedin had to create a new tool as it could not leverage traditional enterprise message systems because of their limitations.
2.1.3 Why Kafka Core? Ø With Kafka Core, Linkedin built a central hub to host all of its event streams, a universal data pipeline and asynchronous services. Total connections = N producers + M consumers 11 Search Security Fraud Detection Application User Tracking Operational Logs Operational MetricsEspresso Cassandra Oracle Hadoop Log Search Monitoring Data Warehouse Kafka
2.1.3 Why Kafka Core? Ø Apache Kafka is modeled as an append only distributed log which is suitable to model event streams. ØApache Kafka comes with out-the-box features such as: • High throughput • Low latency • Distributed - Horizontal scaling • Support for multiple consumers • Configurable persistence • Automatic recovery from failure • Polyglot ready with its support for many languages • Security: support for encrypted data transfer 12
2.2 Kafka Connect: Event Import and Export 2.2.1 What is Kafka Connect? 2.2.2 Before Kafka Connect? 2.2.3 Why Kafka Connect? 13
2.2.1 What is Kafka Connect? Ø Kafka Connect is a framework, included in Apache Kafka since Kafka 0.9 release on November 24th 2015, to rapidly stream events: • from external data systems into Kafka • out of Kafka to external data systems. ØReady to use pre-built Kafka connectors ØREST service to define and manage Kafka connectors ØRuntime to run Kafka connectors in standalone or distributed mode ØJava API to build custom Kafka connectors 14
Building Streaming Data Applications Using Apache Kafka
2.2.2 Before Kafka Connect? Ø Before Kafka Connect, to import data from other systems to Kafka or to export data from Kafka to other systems, you have 4 options: Option 1: Build your own Do It Yourself (DIY) solution: custom code using the Kafka producer API or the Kafka consumer API. Option 2: Use one of the many existing tools such as Linkedin Camus/Gobblin for Kafka to HDFS export, Flume, Sqoop, Logstash, Apache Nifi, StreamSets, ETL tool such as Talend, Pentaho, … Option 3: Use stream processors to import data to Kafka or export it from Kafka! Example: Storm, Spark Streaming, Flink, Samza, … Option 4: Use Confluent REST Proxy API (open source project maintained by Confluent) to read and write data to Kafka Ø Each one of the 4 options above to import/export data to Kafka has its own advantages and disadvantages. 16
2.2.3 Why Kafka Connect? Ø Using the Kafka Connect framework to stream data in and out of Kafka has the following advantages: • alleviates the burden of writing custom code or learning and integrating with a new tool to stream data in and out of Kafka for each data system! • use pre-built Kafka connectors to a variety of data systems just by writing configuration files and submitting them to Connect with minimal or no code necessary • Out-of-the-box features such as auto recovery, auto failover, automated load balancing, dynamic scaling, exactly-once delivery guarantees, … • Out-of-the box integration with the Schema Registry to capture schema information from sources if it is present • enables to build custom Kafka connectors leveraging the Kafka Connect framework 17
2.3 Kafka Streams: Event processing 2.3.1 What is Kafka Streams? 2.3.2 Before Kafka Streams? 2.3.3 Why Kafka Streams? 18
2.3.1 What is Kafka Streams? Ø Kafka Streams is a lightweight open source Java library, included in Apache Kafka since 0.10 release in May 2016, for building stream processing applications on top of Apache Kafka. Ø Kafka Streams is specifically designed to consume from & produce data to Kafka topics. Ø A high-level and declarative API for common patterns like filter, map, aggregations, joins, stateful and stateless processing. Ø A low-level and imperative API for building topologies of processors, streams and tables. 19
2.3.2 Before Kafka Streams? ØBefore Kafka Streams, to process the data in Kafka you have 4 options: • Option 1: Dot It Yourself (DIY) – Write your own ‘stream processor’ using Kafka client libs, typically with a narrower focus. • Option 2: Use a library such as AkkaStreams- Kafka, also known as Reactive Kafka, RxJava, or Vert.x • Option 3: Use an existing open source stream processing framework such as Apache Storm, Spark Streaming, Apache Flink or Apache Samza for transforming and combining data streams which live in Kafka. • Option 4: Use an existing commercial tool for stream processing with adapter to Kafka such as IBM InfoSphere Streams, TIBCO StreamBase, … ØEach one of the 4 options above of processing data in Kafka has advantages and disadvantages. 20
2.3.3 Why Kafka Streams? Ø Processing data in Kafka with Kafka Streams has the following advantages: • No need to learn another framework or tool for stream processing as Kafka Streams is already a library included in Kafka • No need of external infrastructure beyond Kafka. Kafka is already your cluster! • Operational simplicity obtained by getting rid of an additional stream processing cluster. • Kafka Streams inherits operational characteristics ( low latency, elasticity, fault- tolerance, …) from Kafka. • Low barrier to entry: You can quickly write and run a small-scale proof-of-concept on a single machine 21
2.3.3 Why Kafka Streams? • As a normal library, Kafka Streams is easier to compose with other Java libraries and integrate with your existing applications and services • Kafka Streams runs in your application code and imposes no change in the Kafka cluster infrastructure, or within Kafka. • Kafka Streams comes with abstractions and features for easier and efficient processing of event streams: • KStream and KTable as the two basic abstractions and there is a duality between them: • KStream = immutable log • KTable = mutable materialized view • Interactive Queries: Local queryable state is a fundamental primitive in Kafka Streams 22
2.3.3 Why Kafka Streams? • Exactly-One semantics and local transactions: • Time as a critical aspect in stream processing and how it is modeled and integrated: Event time, Ingestion time, Processing time. • Windowing to control how to group records that have the same key for stateful operations such as aggregations or joins into so-called windows. 23
Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 24
3. Code and Demo of an end-to-end Streaming Data Application using Kafka 3.1 Scenario of this demo 3.2 Architecture of this demo 3.3 Setup of this demo 3.4 Results of this demo 3.5 Stopping the demo!
3.1. Scenario of this demo ØThis demo consists of: • reading live stream of data (tweets) from Twitter using Kafka Connect connector for Twitter • storing them in Kafka broker leveraging Kafka Core as publish-subscribe message system. • performing some basic stream processing on tweets in Avro format from a Kafka topic using Kafka Streams library to do the following: • Raw word count - every occurrence of individual words is counted and written to the topic wordcount (a predefined list of stopwords will be ignored) • 5-Minute word count - words are counted per 5 minute window and every word that has more than 3 occurrences is written to the topic wordcount5m • Buzzwords - a list of special interest words can be defined and those will be tracked in the topic buzzwords 26
3.1. Scenario of this demo ØThis demo is adapted from one that was given by Sönke Liebau on July 27th 2016 from OpenCore, Germany. See blog entry titled: ‘Processing Twitter Data with Kafka Streams” http://www.opencore.com/blog/2016/7/kafka-streams-demo/ and related code at GitHub https://github.com/opencore/kafkastreamsdemo ØWhat is specific to this demo : • Use of a Docker container instead of the confluent platform they are providing with their Virtual Machine defined in Vagrant. • Use of Kafka Connect UI from Landoop for easy and fast configuration of Twitter connector and also other Landoop’s Fast Data Web UIs. 27
3.2. Architecture of this demo 28
3.3. Setup of this demo Step 1: Setup your Kafka Development Environment Step 2: Get twitter credentials to connect to live data Step 3: Get twitter live data into Kafka broker Step 4: Write and test the application code in Java Step 5: Run the application 29
Step 1: Setup your Kafka Development Environment ØThe easiest way to get up and running quickly is to use a Docker container with all components needed. ØFirst, install Docker on your desktop or on the cloud https://www.docker.com/products/overview and start it 30 30
Step 1: Setup your Kafka Development Environment ØSecond, install Fast-data-dev, a Docker image for Kafka developers which is packaging: • Kafka broker • Zookeeper • Open source version of the Confluent Platform with its Schema registry, REST Proxy and bundled connectors • Certified DataMountaineer Connectors (ElasticSearch, Cassandra, Redis, ..) • Landoop's Fast Data Web UIs : schema-registry, kafka-topics, kafka-connect. • Please note that Fast Data Web UIs are licensed under BSL. You should contact Landoop if you plan to use them on production clusters with more than 4 nodes. by executing the command below, while Docker is running and you are connected to the internet: docker run --rm -it --net=host landoop/fast-data-dev • If you are on Mac OS X, you have to expose the ports instead: docker run --rm -it -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev • This will download the fast-data-dev Docker image from the Dock Hub. https://hub.docker.com/r/landoop/fast-data-dev/ • Future runs will use your local copy. • More details about Fast-data-dev docker image https://github.com/Landoop/fast-data-dev 31
Step 1: Setup your Kafka Development Environment ØPoints of interest: • the -p flag is used to publish a network port. Inside the container, ZooKeeper listens at 2181 and Kafka at 9092. If we don’t publish them with -p, they are not available outside the container, so we can’t really use them. • the –e flag sets up environment variables. • the last part specifies the image we want to run: landoop/fast-data-dev • Docker will realize it doesn’t have the landoop/fast-data- dev image locally, so it will first download it. ØThat's it. • Your Kafka Broker is at localhost:9092, • your Kafka REST Proxy at localhost:8082, • your Schema Registry at localhost:8081, • your Connect Distributed at localhost:8083, • your ZooKeeper at localhost:2181 32
Step 1: Setup your Kafka Development Environment ØAt http://localhost:3030, you will find Landoop's Web UIs for: • Kafka Topics • Schema Registry • as well as a integration test report for connectors & infrastructure using Coyote. https://github.com/Landoop/coyote ØIf you want to stop all services and remove everything, simply hit Control+C. 33
Step 1: Setup your kafka Development Environment ØExplore Integration test results at http://localhost:3030/coyote-tests/ 34
Step 2: Get twitter credentials to connect to live data ØNow that our single-node Kafka cluster is fully up and running, we can proceed to preparing the input data: • First you need to register an application with Twitter. • Second, once the application is created copy the Consumer key and Consumer Secret. • Third, generate the Access Token Access and Secret Token required to give your twitter account access to the new application ØFull instructions are here: https://apps.twitter.com/app/new 35
Step 3: Get twitter live data into Kafka broker ØFirst, create a new Kafka Connect for Twitter 36
Step 3: Get twitter live data into Kafka broker ØSecond, configure this Kafka Connect for Twitter to write to the topic twitter by entering your own track.terms and also the values of twitter.token, twitter.secret, twitter.comsumerkey and twitter.consumer.secret 37
Step 3: Get twitter live data into Kafka broker ØKafka Connect for Twitter is now configured to write data to the topic twitter. 38
Step 3: Get twitter live data into Kafka broker ØData is now being written to the topic twitter. 39
Step 4: Write and test the application code in Java Ø Instead of writing our own code for this demo, we will be leveraging an existing code from GitHub by Sonke Liebau: https://github.com/opencore/kafkastreamsdemo 40
Step 4: Write and test the application code in Java Ø git clone https://github.com/opencore/kafkastreamsdemo Ø Edit the buzzwords.txt file with your own works and probably one of the twitter terms that you are watching live: 41
Step 4: Write and test the application code in Java Ø Edit the pom.xml to reflect the Kafka version compatible with Confluent Data platform/Landoop. See https://github.com/Landoop/fast-data-dev/blob/master/README.md 42
Step 5: Run the application Ø The next step is to run the Kafka Streams application that processes twitter data. Ø First, install Maven http://maven.apache.org/install.html Ø Then, compile the code into a fat jar with Maven. $ mvn package 43
Step 5: Run the application ØTwo jar files will be created in the target folder: 1. KafkaStreamsDemo-1.0-SNAPSHOT.jar – Only your project classes 2. KafkaStreamsDemo-1.0-SNAPSHOT-jar-with-dependencies.jar – Project and dependency classes in a single jar. 44
Step 5: Run the application Ø Then java -cp target/KafkaStreamsDemo-1.0-SNAPSHOT- jar-with-dependencies.jar com.opencore.sapwebinarseries.KafkaStreamsDemo Ø TIP: During development: from your IDE, from CLI … Kafka Streams Application Reset Tool, available since Apache Kafka 0.10.0.1, is great for playing around. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Application+Reset+Tool 45
3.4. Results of this demo ØOnce the above is running, the following topics will be populated with data : • Raw word count - Every occurrence of individual words is counted and written to the topic wordcount (a predefined list of stopwords will be ignored) • 5-Minute word count - Words are counted per 5 minute window and every word that has more than three occurrences is written to the topic wordcount5m • Buzzwords - a list of special interest words can be defined and those will be tracked in the topic buzzwords - the list of these words can be defined in the file buzzwords.txt 46
3.4. Results of this demo ØAccessing the data generated by the code is as simple as starting a console consumer which is shipped with Kafka • You need first to enter the container to use any tool as you like: docker run --rm -it --net=host landoop/fast-data-dev bash • Use the following command to check the topics: • kafka-console-consumer --topic wordcount --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true • kafka-console-consumer --topic wordcount5m --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true • kafka-console-consumer --topic buzzwords --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true 47
3.4. Results of this demo 48
3.5. Stopping the demo! ØTo stop the Kafka Streams Demo application: • $ ps – A | grep java • $ kill -9 <PID> ØIf you want to stop all services in fast-data-dev Docker image and remove everything, simply hit Control+C. 49
Thank you! Let’s keep in touch! @SlimBaltagi https://www.linkedin.com/in/slimbaltagi sbaltagi@gmail.com 50

More Related Content

Building Streaming Data Applications Using Apache Kafka

  • 1. Los Angeles, California August 5th 2017 Slim Baltagi Building Streaming Data Applications Using Apache Kafka
  • 2. Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 2
  • 4. Stream Processor Destination Systems Event Streams Collector Apps Sensors Devices Other Sources Sourcing & Integration Analytics & Processing Serving & Consuming 4 1. A typical Streaming Data Application Event Streams Broker Event Streams Processor Destination Systems Event Streams Collectors Apps Sensors Databases Other Source Systems A very simplified diagram!
  • 5. Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 5
  • 6. 2. Apache Kafka as a platform for building and running streaming data applications ØApache Kafka is an open source streaming data platform (a new category of software!) • to import event streams from other source data systems into Kafka and export event streams from Kafka to destination data systems • to transport and store event streams • to process event streams live as they occur. 6
  • 7. 2.1 Kafka Core: Event Streams Transport and Storage 2.1.1 What is Kafka Core? 2.1.2 Before Kafka Core? 2.1.3 Why Kafka Core? 7
  • 8. 2.1.1 What is Kafka Core? Ø Kafka is a software written in Scala and Java and originally developed by Linkedin in 2010. Ø It was open sourced as an apache project in 2011 and became a Top Level Project in 2012. Ø After 7 years, it is graduating to version 1.0 in October 2017!! Ø Kafka Core is an enterprise messaging system to: • publish event streams • subscribe to event streams • store event streams Ø Kafka Core is the the ‘digital nervous system’ connecting all enterprise data and systems of many notable companies. Ø Diverse and rapidly growing user base across many industries and verticals. 8
  • 9. 2.1.2 Before Kafka Core? 9 Ø Before Kafka Core, Linkedin had to build many custom data pipelines, for streaming and queueing data, that use point to point communication and need to be constantly scaled individually. Total connections = N producers * M consumers Search Security Fraud Detection Application User Tracking Operational Logs Operational Metrics Hadoop Search Monitoring Data Warehouse Espresso Cassandra Oracle
  • 10. 2.1.2 Before Kafka Core? 10 Ø Traditional enterprise message systems such as RabbitMQ, Apache ActiveMQ, IBM WebSphere MQ, TIBCO EMS could not help because of these limitations: • They can’t accommodate the web-scale requirements of Linkedin • Producers and consumers are really coupled from a performance perspective because of the ‘slow consumer problem’. • Messages are sent into a central message spool and stored only until they are processed, acknowledged and then they are deleted. Ø Linkedin had to create a new tool as it could not leverage traditional enterprise message systems because of their limitations.
  • 11. 2.1.3 Why Kafka Core? Ø With Kafka Core, Linkedin built a central hub to host all of its event streams, a universal data pipeline and asynchronous services. Total connections = N producers + M consumers 11 Search Security Fraud Detection Application User Tracking Operational Logs Operational MetricsEspresso Cassandra Oracle Hadoop Log Search Monitoring Data Warehouse Kafka
  • 12. 2.1.3 Why Kafka Core? Ø Apache Kafka is modeled as an append only distributed log which is suitable to model event streams. ØApache Kafka comes with out-the-box features such as: • High throughput • Low latency • Distributed - Horizontal scaling • Support for multiple consumers • Configurable persistence • Automatic recovery from failure • Polyglot ready with its support for many languages • Security: support for encrypted data transfer 12
  • 13. 2.2 Kafka Connect: Event Import and Export 2.2.1 What is Kafka Connect? 2.2.2 Before Kafka Connect? 2.2.3 Why Kafka Connect? 13
  • 14. 2.2.1 What is Kafka Connect? Ø Kafka Connect is a framework, included in Apache Kafka since Kafka 0.9 release on November 24th 2015, to rapidly stream events: • from external data systems into Kafka • out of Kafka to external data systems. ØReady to use pre-built Kafka connectors ØREST service to define and manage Kafka connectors ØRuntime to run Kafka connectors in standalone or distributed mode ØJava API to build custom Kafka connectors 14
  • 16. 2.2.2 Before Kafka Connect? Ø Before Kafka Connect, to import data from other systems to Kafka or to export data from Kafka to other systems, you have 4 options: Option 1: Build your own Do It Yourself (DIY) solution: custom code using the Kafka producer API or the Kafka consumer API. Option 2: Use one of the many existing tools such as Linkedin Camus/Gobblin for Kafka to HDFS export, Flume, Sqoop, Logstash, Apache Nifi, StreamSets, ETL tool such as Talend, Pentaho, … Option 3: Use stream processors to import data to Kafka or export it from Kafka! Example: Storm, Spark Streaming, Flink, Samza, … Option 4: Use Confluent REST Proxy API (open source project maintained by Confluent) to read and write data to Kafka Ø Each one of the 4 options above to import/export data to Kafka has its own advantages and disadvantages. 16
  • 17. 2.2.3 Why Kafka Connect? Ø Using the Kafka Connect framework to stream data in and out of Kafka has the following advantages: • alleviates the burden of writing custom code or learning and integrating with a new tool to stream data in and out of Kafka for each data system! • use pre-built Kafka connectors to a variety of data systems just by writing configuration files and submitting them to Connect with minimal or no code necessary • Out-of-the-box features such as auto recovery, auto failover, automated load balancing, dynamic scaling, exactly-once delivery guarantees, … • Out-of-the box integration with the Schema Registry to capture schema information from sources if it is present • enables to build custom Kafka connectors leveraging the Kafka Connect framework 17
  • 18. 2.3 Kafka Streams: Event processing 2.3.1 What is Kafka Streams? 2.3.2 Before Kafka Streams? 2.3.3 Why Kafka Streams? 18
  • 19. 2.3.1 What is Kafka Streams? Ø Kafka Streams is a lightweight open source Java library, included in Apache Kafka since 0.10 release in May 2016, for building stream processing applications on top of Apache Kafka. Ø Kafka Streams is specifically designed to consume from & produce data to Kafka topics. Ø A high-level and declarative API for common patterns like filter, map, aggregations, joins, stateful and stateless processing. Ø A low-level and imperative API for building topologies of processors, streams and tables. 19
  • 20. 2.3.2 Before Kafka Streams? ØBefore Kafka Streams, to process the data in Kafka you have 4 options: • Option 1: Dot It Yourself (DIY) – Write your own ‘stream processor’ using Kafka client libs, typically with a narrower focus. • Option 2: Use a library such as AkkaStreams- Kafka, also known as Reactive Kafka, RxJava, or Vert.x • Option 3: Use an existing open source stream processing framework such as Apache Storm, Spark Streaming, Apache Flink or Apache Samza for transforming and combining data streams which live in Kafka. • Option 4: Use an existing commercial tool for stream processing with adapter to Kafka such as IBM InfoSphere Streams, TIBCO StreamBase, … ØEach one of the 4 options above of processing data in Kafka has advantages and disadvantages. 20
  • 21. 2.3.3 Why Kafka Streams? Ø Processing data in Kafka with Kafka Streams has the following advantages: • No need to learn another framework or tool for stream processing as Kafka Streams is already a library included in Kafka • No need of external infrastructure beyond Kafka. Kafka is already your cluster! • Operational simplicity obtained by getting rid of an additional stream processing cluster. • Kafka Streams inherits operational characteristics ( low latency, elasticity, fault- tolerance, …) from Kafka. • Low barrier to entry: You can quickly write and run a small-scale proof-of-concept on a single machine 21
  • 22. 2.3.3 Why Kafka Streams? • As a normal library, Kafka Streams is easier to compose with other Java libraries and integrate with your existing applications and services • Kafka Streams runs in your application code and imposes no change in the Kafka cluster infrastructure, or within Kafka. • Kafka Streams comes with abstractions and features for easier and efficient processing of event streams: • KStream and KTable as the two basic abstractions and there is a duality between them: • KStream = immutable log • KTable = mutable materialized view • Interactive Queries: Local queryable state is a fundamental primitive in Kafka Streams 22
  • 23. 2.3.3 Why Kafka Streams? • Exactly-One semantics and local transactions: • Time as a critical aspect in stream processing and how it is modeled and integrated: Event time, Ingestion time, Processing time. • Windowing to control how to group records that have the same key for stateful operations such as aggregations or joins into so-called windows. 23
  • 24. Agenda 1. A typical streaming data application 2. Apache Kafka as a platform for building and running streaming data applications 3. Code and demo of an end-to-end Kafka-driven streaming data application 24
  • 25. 3. Code and Demo of an end-to-end Streaming Data Application using Kafka 3.1 Scenario of this demo 3.2 Architecture of this demo 3.3 Setup of this demo 3.4 Results of this demo 3.5 Stopping the demo!
  • 26. 3.1. Scenario of this demo ØThis demo consists of: • reading live stream of data (tweets) from Twitter using Kafka Connect connector for Twitter • storing them in Kafka broker leveraging Kafka Core as publish-subscribe message system. • performing some basic stream processing on tweets in Avro format from a Kafka topic using Kafka Streams library to do the following: • Raw word count - every occurrence of individual words is counted and written to the topic wordcount (a predefined list of stopwords will be ignored) • 5-Minute word count - words are counted per 5 minute window and every word that has more than 3 occurrences is written to the topic wordcount5m • Buzzwords - a list of special interest words can be defined and those will be tracked in the topic buzzwords 26
  • 27. 3.1. Scenario of this demo ØThis demo is adapted from one that was given by Sönke Liebau on July 27th 2016 from OpenCore, Germany. See blog entry titled: ‘Processing Twitter Data with Kafka Streams” http://www.opencore.com/blog/2016/7/kafka-streams-demo/ and related code at GitHub https://github.com/opencore/kafkastreamsdemo ØWhat is specific to this demo : • Use of a Docker container instead of the confluent platform they are providing with their Virtual Machine defined in Vagrant. • Use of Kafka Connect UI from Landoop for easy and fast configuration of Twitter connector and also other Landoop’s Fast Data Web UIs. 27
  • 28. 3.2. Architecture of this demo 28
  • 29. 3.3. Setup of this demo Step 1: Setup your Kafka Development Environment Step 2: Get twitter credentials to connect to live data Step 3: Get twitter live data into Kafka broker Step 4: Write and test the application code in Java Step 5: Run the application 29
  • 30. Step 1: Setup your Kafka Development Environment ØThe easiest way to get up and running quickly is to use a Docker container with all components needed. ØFirst, install Docker on your desktop or on the cloud https://www.docker.com/products/overview and start it 30 30
  • 31. Step 1: Setup your Kafka Development Environment ØSecond, install Fast-data-dev, a Docker image for Kafka developers which is packaging: • Kafka broker • Zookeeper • Open source version of the Confluent Platform with its Schema registry, REST Proxy and bundled connectors • Certified DataMountaineer Connectors (ElasticSearch, Cassandra, Redis, ..) • Landoop's Fast Data Web UIs : schema-registry, kafka-topics, kafka-connect. • Please note that Fast Data Web UIs are licensed under BSL. You should contact Landoop if you plan to use them on production clusters with more than 4 nodes. by executing the command below, while Docker is running and you are connected to the internet: docker run --rm -it --net=host landoop/fast-data-dev • If you are on Mac OS X, you have to expose the ports instead: docker run --rm -it -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev • This will download the fast-data-dev Docker image from the Dock Hub. https://hub.docker.com/r/landoop/fast-data-dev/ • Future runs will use your local copy. • More details about Fast-data-dev docker image https://github.com/Landoop/fast-data-dev 31
  • 32. Step 1: Setup your Kafka Development Environment ØPoints of interest: • the -p flag is used to publish a network port. Inside the container, ZooKeeper listens at 2181 and Kafka at 9092. If we don’t publish them with -p, they are not available outside the container, so we can’t really use them. • the –e flag sets up environment variables. • the last part specifies the image we want to run: landoop/fast-data-dev • Docker will realize it doesn’t have the landoop/fast-data- dev image locally, so it will first download it. ØThat's it. • Your Kafka Broker is at localhost:9092, • your Kafka REST Proxy at localhost:8082, • your Schema Registry at localhost:8081, • your Connect Distributed at localhost:8083, • your ZooKeeper at localhost:2181 32
  • 33. Step 1: Setup your Kafka Development Environment ØAt http://localhost:3030, you will find Landoop's Web UIs for: • Kafka Topics • Schema Registry • as well as a integration test report for connectors & infrastructure using Coyote. https://github.com/Landoop/coyote ØIf you want to stop all services and remove everything, simply hit Control+C. 33
  • 34. Step 1: Setup your kafka Development Environment ØExplore Integration test results at http://localhost:3030/coyote-tests/ 34
  • 35. Step 2: Get twitter credentials to connect to live data ØNow that our single-node Kafka cluster is fully up and running, we can proceed to preparing the input data: • First you need to register an application with Twitter. • Second, once the application is created copy the Consumer key and Consumer Secret. • Third, generate the Access Token Access and Secret Token required to give your twitter account access to the new application ØFull instructions are here: https://apps.twitter.com/app/new 35
  • 36. Step 3: Get twitter live data into Kafka broker ØFirst, create a new Kafka Connect for Twitter 36
  • 37. Step 3: Get twitter live data into Kafka broker ØSecond, configure this Kafka Connect for Twitter to write to the topic twitter by entering your own track.terms and also the values of twitter.token, twitter.secret, twitter.comsumerkey and twitter.consumer.secret 37
  • 38. Step 3: Get twitter live data into Kafka broker ØKafka Connect for Twitter is now configured to write data to the topic twitter. 38
  • 39. Step 3: Get twitter live data into Kafka broker ØData is now being written to the topic twitter. 39
  • 40. Step 4: Write and test the application code in Java Ø Instead of writing our own code for this demo, we will be leveraging an existing code from GitHub by Sonke Liebau: https://github.com/opencore/kafkastreamsdemo 40
  • 41. Step 4: Write and test the application code in Java Ø git clone https://github.com/opencore/kafkastreamsdemo Ø Edit the buzzwords.txt file with your own works and probably one of the twitter terms that you are watching live: 41
  • 42. Step 4: Write and test the application code in Java Ø Edit the pom.xml to reflect the Kafka version compatible with Confluent Data platform/Landoop. See https://github.com/Landoop/fast-data-dev/blob/master/README.md 42
  • 43. Step 5: Run the application Ø The next step is to run the Kafka Streams application that processes twitter data. Ø First, install Maven http://maven.apache.org/install.html Ø Then, compile the code into a fat jar with Maven. $ mvn package 43
  • 44. Step 5: Run the application ØTwo jar files will be created in the target folder: 1. KafkaStreamsDemo-1.0-SNAPSHOT.jar – Only your project classes 2. KafkaStreamsDemo-1.0-SNAPSHOT-jar-with-dependencies.jar – Project and dependency classes in a single jar. 44
  • 45. Step 5: Run the application Ø Then java -cp target/KafkaStreamsDemo-1.0-SNAPSHOT- jar-with-dependencies.jar com.opencore.sapwebinarseries.KafkaStreamsDemo Ø TIP: During development: from your IDE, from CLI … Kafka Streams Application Reset Tool, available since Apache Kafka 0.10.0.1, is great for playing around. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Application+Reset+Tool 45
  • 46. 3.4. Results of this demo ØOnce the above is running, the following topics will be populated with data : • Raw word count - Every occurrence of individual words is counted and written to the topic wordcount (a predefined list of stopwords will be ignored) • 5-Minute word count - Words are counted per 5 minute window and every word that has more than three occurrences is written to the topic wordcount5m • Buzzwords - a list of special interest words can be defined and those will be tracked in the topic buzzwords - the list of these words can be defined in the file buzzwords.txt 46
  • 47. 3.4. Results of this demo ØAccessing the data generated by the code is as simple as starting a console consumer which is shipped with Kafka • You need first to enter the container to use any tool as you like: docker run --rm -it --net=host landoop/fast-data-dev bash • Use the following command to check the topics: • kafka-console-consumer --topic wordcount --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true • kafka-console-consumer --topic wordcount5m --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true • kafka-console-consumer --topic buzzwords --new- consumer --bootstrap-server 127.0.0.1:9092 --property print.key=true 47
  • 48. 3.4. Results of this demo 48
  • 49. 3.5. Stopping the demo! ØTo stop the Kafka Streams Demo application: • $ ps – A | grep java • $ kill -9 <PID> ØIf you want to stop all services in fast-data-dev Docker image and remove everything, simply hit Control+C. 49
  • 50. Thank you! Let’s keep in touch! @SlimBaltagi https://www.linkedin.com/in/slimbaltagi sbaltagi@gmail.com 50