SlideShare a Scribd company logo

Event Hub (i.e. Kafka) in Modern Data Architecture

•
•
Guido Schmutz
Guido Schmutz

Today's modern data architectures and the their implementations contain an Event Hub. What are the benefits of placing an Event Hub in a Modern Data (Analytics) Architecture? What exactly is an Event Hub and what capabilities should it provide? Why is Apache Kafka the most popular realization of an Event Hub? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Hub. Then the session will highlight the different architecture styles which can be supported using an Event Hub (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Hub the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.

1 of 38
Download to read offline
http://guidoschmutz@wordpress.com@gschmutz Event Hub (Kafka) in Modern Data Architecture Guido Schmutz
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 23 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 192nd edition
Event Hub (i.e. Kafka) in Modern Data Architecture
What exactly is an Event Hub?
Event Hub Event Hub – as a starting point
Event Hub Event Hub – an Infrastructure with these capabilities 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients
Kafka – the most popular Event Hub
Kafka – the most popular Event Hub Kafka Cluster Consumer 1 Consumer 2 Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1 10 12 3 5 6 7 14 8 9 11 12 Producer3Producer3 ConsumerConsumer 3 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
Event Hub Event Hub – capabilities supported by Kafka 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients Light Grey = limited support
• Cloud Services • Cloud Services with Kafka API • Kafka Cloud Services Event Hub - Kafka Alternatives? Cloud Services? • traditional Message Brokers (with a lot of limitations regarding Event Hub capabilities) • Apache Pulsar • Solace • Pravega (Dell Streaming Platform) • Oracle AQ (Kafka API coming) AQ
Event Hub - core building block of a Modern Data Architecture
Event Hub Event Hub – as a starting point Vehicle Environ mental Streaming Data Sources Ware house E-Comm erce
Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Data Integration for integrating various data sources E-Comm erce Stream Data Integration
Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Using Edge Computing and Stream Data Integration • MQTT as a gateway to Kafka E-Comm erce Stream Data Integration Streaming Data Sources
Stream Data Integration – Kafka Connect / StreamSets • declarative style, simple data flows • framework is part of Apache Kafka • Many connectors available • Single Message Transforms (SMT) • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing (micro- batching) • custom sources, sinks, processors
Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Analytics • Time Windowed State Management • Stream-to-Table Joins • Stream-to-Stream Joins • Event Pattern Detection • Machine Learning Model Execution (Inference) [1] E-Comm erce Stream Data Integration Gateway
Stream Analytics - Kafka Streams • Programmatic API, “just” a Java library • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once • Stream Processing with zero coding using SQL-like language • built on top of Kafka Streams • interactive (CLI) and headless (cmd file) trucking_ driver Kafka Broker Java Application Kafka Streams ksqlDB trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Analytics • Push results back to new topic so other interested parties can use it too! E-Comm erce Stream Data Integration Streaming Data Sources Gateway
Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Data Integration to callback to Data Source (to Actuator) E-Comm erce Stream Data Integration Streaming Data Sources Gateway
Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Streaming Visualization • ksqlDB pull queries or Kafka Streams Interactive Queries allow to query state of stream processor [2] E-Comm erce Stream Data Integration Streaming Data Sources Stream Data Integration Gateway
Event Hub Stream Analytics Legacy App Stream Data IntegrationCDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration • Stream-to-Table join E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
Kafka as an Event Hub 10 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Providing “Materialized Views” in RDBMS or NoSQL Datastores E-Comm erce Stream Data Integration Streaming Data Sources Gateway • Bootstrap ”Materialized View” from event history Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • Microservice participates as both a consumer and producer of events E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • 2nd microservice consumes events from 1st Bootstrap from event history [3] E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Bi-Directional Legacy Systems Integration [4]AQ E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Hybrid Cloud Scenario AQ E-Comm erce Stream Data Integration Streaming Data Sources Event Hub Mirroring Event Hub Gateway Legacy Data Sources
Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Batch Analytics Event Hub as “Virtualized” Data Lake for Batch Analytics E-Comm erce Stream Data Integration Streaming Data Sources 1st Micro service 2nd Micro service Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC AQ Gateway Legacy Data Sources
Kafka Storage Local Storage Tiered Storage (Confluent Enterprise) Broker 1 Broker 2 Broker 3 Broker 1 Broker 2 Broker 3 Object Storage hothot & cold cold 10 10 Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize “Materialized” Data Lake for Batch Analytics E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Serverless/Function as a Service (FaaS) E-Comm erce Stream Data Integration Streaming Data Sources Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information! E-Comm erce Stream Data Integration Streaming Data Sources Legacy Data Sources
Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information! E-Comm erce Stream Data Integration Streaming Data Sources Log as a first-class citizen! Turning the database Inside out! Legacy Data Sources
Summary
Ref Architecture Service Event Stream Bulk Data Flow Bulk Source Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Modern Data Platform Event Stream Event Stream
Reference 1. Stream Processing Concepts and Frameworks 2. Streaming Visualization 3. Building event-driven (Micro)Services with Apache Kafka 4. Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Event Hub (i.e. Kafka) in Modern Data Architecture

More Related Content

Event Hub (i.e. Kafka) in Modern Data Architecture

  • 1. http://guidoschmutz@wordpress.com@gschmutz Event Hub (Kafka) in Modern Data Architecture Guido Schmutz
  • 2. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 23 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 192nd edition
  • 4. What exactly is an Event Hub?
  • 5. Event Hub Event Hub – as a starting point
  • 6. Event Hub Event Hub – an Infrastructure with these capabilities 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients
  • 7. Kafka – the most popular Event Hub
  • 8. Kafka – the most popular Event Hub Kafka Cluster Consumer 1 Consumer 2 Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1 10 12 3 5 6 7 14 8 9 11 12 Producer3Producer3 ConsumerConsumer 3 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
  • 9. Event Hub Event Hub – capabilities supported by Kafka 1. topic semantics (publish/subscribe) – message can be consumed by 0 – n consumers 2. queue semantics – messages can be consumed by exactly one consumer 3. horizontally scalable – throughput increases with more resources 4. auto-scaling – up and down-scaling upon load 5. highly available – no single point of failure 6. Control/handle back-pressure 7. durable – messages may not be lost 8. schema-less – no knowledge on message content and format 9. Efficient support of Stream and Batch Consumers (offline and with large Backlog) 10. (Unlimited) Retention of messages (long term storage) 11. Guaranteed ordering of messages 12. Support re-consumption of events 13. Access control – control over who can produce and consume which events 14. interoperable – support for different clients Light Grey = limited support
  • 10. • Cloud Services • Cloud Services with Kafka API • Kafka Cloud Services Event Hub - Kafka Alternatives? Cloud Services? • traditional Message Brokers (with a lot of limitations regarding Event Hub capabilities) • Apache Pulsar • Solace • Pravega (Dell Streaming Platform) • Oracle AQ (Kafka API coming) AQ
  • 11. Event Hub - core building block of a Modern Data Architecture
  • 12. Event Hub Event Hub – as a starting point Vehicle Environ mental Streaming Data Sources Ware house E-Comm erce
  • 13. Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Data Integration for integrating various data sources E-Comm erce Stream Data Integration
  • 14. Event Hub Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Using Edge Computing and Stream Data Integration • MQTT as a gateway to Kafka E-Comm erce Stream Data Integration Streaming Data Sources
  • 15. Stream Data Integration – Kafka Connect / StreamSets • declarative style, simple data flows • framework is part of Apache Kafka • Many connectors available • Single Message Transforms (SMT) • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing (micro- batching) • custom sources, sinks, processors
  • 16. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Streaming Data Sources Ware house Using Stream Analytics • Time Windowed State Management • Stream-to-Table Joins • Stream-to-Stream Joins • Event Pattern Detection • Machine Learning Model Execution (Inference) [1] E-Comm erce Stream Data Integration Gateway
  • 17. Stream Analytics - Kafka Streams • Programmatic API, “just” a Java library • fault-tolerant local state • Fixed, Sliding and Session Windowing • Stream-Stream / Stream-Table Joins • At-least-once and exactly-once • Stream Processing with zero coding using SQL-like language • built on top of Kafka Streams • interactive (CLI) and headless (cmd file) trucking_ driver Kafka Broker Java Application Kafka Streams ksqlDB trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
  • 18. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Analytics • Push results back to new topic so other interested parties can use it too! E-Comm erce Stream Data Integration Streaming Data Sources Gateway
  • 19. Event Hub Stream Analytics Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Stream Data Integration to callback to Data Source (to Actuator) E-Comm erce Stream Data Integration Streaming Data Sources Gateway
  • 20. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Using Streaming Visualization • ksqlDB pull queries or Kafka Streams Interactive Queries allow to query state of stream processor [2] E-Comm erce Stream Data Integration Streaming Data Sources Stream Data Integration Gateway
  • 21. Event Hub Stream Analytics Legacy App Stream Data IntegrationCDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration • Stream-to-Table join E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
  • 22. Kafka as an Event Hub 10 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
  • 23. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house (Right-Time) Legacy Systems Integration E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
  • 24. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Providing “Materialized Views” in RDBMS or NoSQL Datastores E-Comm erce Stream Data Integration Streaming Data Sources Gateway • Bootstrap ”Materialized View” from event history Legacy Data Sources
  • 25. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • Microservice participates as both a consumer and producer of events E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
  • 26. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Modern Event-Driven Apps (aka. Microservices) • 2nd microservice consumes events from 1st Bootstrap from event history [3] E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
  • 27. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Bi-Directional Legacy Systems Integration [4]AQ E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources Legacy Data Sources
  • 28. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Hybrid Cloud Scenario AQ E-Comm erce Stream Data Integration Streaming Data Sources Event Hub Mirroring Event Hub Gateway Legacy Data Sources
  • 29. Event Hub Stream Analytics Streaming Visualize Stream Data Integration Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Batch Analytics Event Hub as “Virtualized” Data Lake for Batch Analytics E-Comm erce Stream Data Integration Streaming Data Sources 1st Micro service 2nd Micro service Stream Data Integration NoSQL RDBMS Micro-Batch Visualize Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC AQ Gateway Legacy Data Sources
  • 30. Kafka Storage Local Storage Tiered Storage (Confluent Enterprise) Broker 1 Broker 2 Broker 3 Broker 1 Broker 2 Broker 3 Object Storage hothot & cold cold 10 10 Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based 1. topic semantics 2. queue semantics 3. horizontally scalable 4. auto-scaling 5. highly available 6. back-pressure 7. durable 8. schema-less/opaque 9. Stream and Batch Consumers 10. (Unlimited) Retention 11. Guaranteed ordering 12. re-consumption of events 13. Access Control 14. Interoperable
  • 31. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize “Materialized” Data Lake for Batch Analytics E-Comm erce Stream Data Integration Streaming Data Sources Gateway Legacy Data Sources
  • 32. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Serverless/Function as a Service (FaaS) E-Comm erce Stream Data Integration Streaming Data Sources Legacy Data Sources
  • 33. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration 1st Micro service 2nd Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information! E-Comm erce Stream Data Integration Streaming Data Sources Legacy Data Sources
  • 34. Event Hub Stream Analytics Legacy App Machine IIoT Stream Data IntegrationCDC Stream Data Integration CDC Streaming Visualize Stream Data Integration Micro service Micro service Serverless FaaS Stream Data Integration Stream Data Integration Vehicle Environ mental Ware house Gateway Batch Data Integration Stream Data Integration NoSQL RDBMS Data Lake / DWH Batch Visualize Batch Analytics Micro-Batch Visualize Event Hub becomes the central nervous system for your information! E-Comm erce Stream Data Integration Streaming Data Sources Log as a first-class citizen! Turning the database Inside out! Legacy Data Sources
  • 36. Ref Architecture Service Event Stream Bulk Data Flow Bulk Source Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Modern Data Platform Event Stream Event Stream
  • 37. Reference 1. Stream Processing Concepts and Frameworks 2. Streaming Visualization 3. Building event-driven (Micro)Services with Apache Kafka 4. Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka