SlideShare a Scribd company logo

What is Apache Kafka? Why is it so popular? Should I use it?

•
•
Guido Schmutz
Guido Schmutz

This document discusses Apache Kafka and provides an overview of its key properties and use cases. It notes that Kafka is a publish-subscribe messaging system that is horizontally scalable, highly available, durable, and schema-less. It can be used for streaming data integration and as a central data bus. The document outlines how Kafka fits into a complete streaming data architecture, including ingesting data from various sources, stream processing, batch integration, visualization, and connecting to data lakes, databases, and microservices. It positions Kafka as the central nervous system that streams can connect to and interact with.

Related slideshows

Building Event-Driven (Micro) Services with Apache Kafka
Building Event-Driven (Micro) Services with Apache KafkaBuilding Event-Driven (Micro) Services with Apache Kafka
Building Event-Driven (Micro) Services with Apache Kafka
 
Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?
 
Event Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data ArchitectureEvent Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data Architecture
 
1 of 25
Download to read offline
BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz What is Apache Kafka? Why is it so popular? Should I use it? Guido Schmutz Trivadis Speed Session 2019
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 174th edition
Event Hub Kafka Message Broker – Key properties • Publish / Subscribe Messaging – message can be consumed by 0 – n consumers • horizontally scalable – throughput increases with more nodes • highly available – no SPOF • durable – messages are not lost • Schema-less – Kafka broker has no knowledge on message content and format
Event Hub Vehicle Weather Streaming Data Sources
Event Hub Stream Data Integration Stream Data Integration Vehicle Weather Streaming Data Sources Streaming Data Ingestion
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Stream Analytics • Stream-to-Stream Joins • Stream-to-Table Joins • Time Windowed State Management • Event Pattern Detection • Machine Learning Model Execution (Inference)
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize Data Lake Ingestion • Machine Learning • Graph Algorithms • Natural Language Processing
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Batch Visualize Streaming Visualization
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Result Store Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Highly Decoupled Modern Apps
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Highly Decoupled Modern Apps
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Data Source talks to Kafka through MQTT
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Kafka becomes central nervous system for data
Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
• No SPoF, highly available • Consumer polls for new messages Apache Kafka • horizontally scalable, guaranteed order
Apache Kafka
Apache Kafka – Streaming Platform Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka MQTT Proxy REST Proxy
Meine Präsentationen @ DOAG 2019 Location Intelligence: Real-Time Geofencing mit Kafka Mittwoch, 20. Nov., 14:00 - 14:45 Visualisierung von Streaming-Daten Donnerstag, 21. Nov., 12:00 - 12:45 Bi-direktionale Integration (Oracle) RDBMS und Apache Kafka Donnerstag, 21. Nov., 14:00 - 14:45
What is Apache Kafka? Why is it so popular? Should I use it?

More Related Content

What is Apache Kafka? Why is it so popular? Should I use it?

  • 1. BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz What is Apache Kafka? Why is it so popular? Should I use it? Guido Schmutz Trivadis Speed Session 2019
  • 2. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 174th edition
  • 3. Event Hub Kafka Message Broker – Key properties • Publish / Subscribe Messaging – message can be consumed by 0 – n consumers • horizontally scalable – throughput increases with more nodes • highly available – no SPOF • durable – messages are not lost • Schema-less – Kafka broker has no knowledge on message content and format
  • 5. Event Hub Stream Data Integration Stream Data Integration Vehicle Weather Streaming Data Sources Streaming Data Ingestion
  • 6. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Stream Analytics • Stream-to-Stream Joins • Stream-to-Table Joins • Time Windowed State Management • Event Pattern Detection • Machine Learning Model Execution (Inference)
  • 7. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize Data Lake Ingestion • Machine Learning • Graph Algorithms • Natural Language Processing
  • 8. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
  • 9. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
  • 10. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 11. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 12. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 13. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 14. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Batch Visualize Streaming Visualization
  • 15. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Result Store Integration
  • 16. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Highly Decoupled Modern Apps
  • 17. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Highly Decoupled Modern Apps
  • 18. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Data Source talks to Kafka through MQTT
  • 19. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Kafka becomes central nervous system for data
  • 20. Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  • 21. • No SPoF, highly available • Consumer polls for new messages Apache Kafka • horizontally scalable, guaranteed order
  • 23. Apache Kafka – Streaming Platform Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka MQTT Proxy REST Proxy
  • 24. Meine Präsentationen @ DOAG 2019 Location Intelligence: Real-Time Geofencing mit Kafka Mittwoch, 20. Nov., 14:00 - 14:45 Visualisierung von Streaming-Daten Donnerstag, 21. Nov., 12:00 - 12:45 Bi-direktionale Integration (Oracle) RDBMS und Apache Kafka Donnerstag, 21. Nov., 14:00 - 14:45