SlideShare a Scribd company logo

Day in the life event-driven workshop

•
•
Christina Lin
Christina Lin

This document provides an agenda and overview for an event-driven architecture workshop. The workshop will cover topics including event-driven architecture, change data capture, cloud-native integration, how cloud architectures have evolved, modern application elements like APIs, events and data, container-based application development, and integration patterns. Hands-on labs will allow attendees to work with technologies like Apache Kafka and Red Hat AMQ Streams.

1 of 97
Download to read offline
Event-Driven Architecture Workshop 1 A Day In The Streaming Life
AGENDA 2 Discover Event Driven Architecture, Change Data Capture and Cloud-Native Integration Introduction Walkthrough of the lab environment, including a demonstration of one the labs Demo Presenter will distribute the lab URL and credentials Hands-on Workshop 10:00 11:00
3 How does cloud change things?
4 Modern application architectures Ye Olde Architecture Enlightened postmodernism
5 Modern application architectures APIs Events Data Enterprise Integration Patterns Process and Decisions
6 Waterfall Agile, Incremental, Iterative Development and delivery model
7 Container Workload Key to incremental delivery: Containers
8 What the container is on matters Platform
9 What’s in the container matters Runtime Frameworks {Your business code} OS
10 What’s around the container matters Function- as-a-service Messaging API management Rules Process automation
11 How it’s all engineered together matters {Your Code} Integration Runtime Function- as-a-service Messaging API management Rules Process automation
12 Cloud-Native Community Projects
Red Hat Cloud-Native Dev Platform 13 Our vision is to simplify the creation of cloud-native services and serverless functions with a rich set of components and tools to match the workloads of modern cloud native apps. Cloud-native middleware applications services and service mesh Automate Kubernetes application operations with DevOps in mind Tools and standard processes to increase developer productivity on Kubernetes
Supporting Container and Cloud Native workloads 14 PROCESS AUTOMATIONAPPLICATION RUNTIMES AUTOMATE AND OPTIMIZE BUSINESS PROCESSES Develop, deploy, and manage across cloud and on-premise CORE TOOLS TO BUILD CLOUD NATIVE & MIGRATE EXISTING APPS COMPOSE AND INTEGRATE MICROSERVICES ACROSS AN ENTERPRISE SERVICE NETWORK INTEGRATION Integration with Red Hat Developer, CI/CD tools, & security services Optimized for Red Hat OpenShift & Kubernetes services Support organizations desire for choice and process standardization Emphasis on solution Simplified selling motion Flexible consumption BROKER
15 Designing Modern Applications
Events Use Cases ● Traditional MOM ● Multicloud ● IoT ● Event streaming Data Use Cases ● Data Federation ● Data Virtualization ● Data as APIs ● Change Data Capture EIPs Use Cases ● Intelligent Routing ● Service composition ● Connectivity ● iPaaS ● Data Pipelines APIs Use Cases ● Microservices ● Mobile Apps ● B2B ● Legacy facade Elements of Modern Applications APIs EIPs DataEvents
Events APIs EIPs Data
Introduction: Events EVENT-DRIVENARCHITECTURE 18
What is Event Driven Architecture? 19 Event-Driven Architecture (EDA) is a way of designing applications and services to respond to real-time information based on the sending and receiving of information about individual event notifications
What is an event? 20 Image: https://foto.wuestenigel.com/hand-ready-to-push-domino-pieces/ Event an action or occurrence recognized by software, often originating asynchronously from the external environment, that may be handled by the software
What is an event? 21 Source: Async form of Remote Procedure Call, contains instructions telling recipient what to do, may cause a change of state. QueryCommand Similar to commands, queries expect a response returning the results, but does not cause any change in state. Event Immutable state and value of a particular entity, which occurred during operation among services.
Types of event consumption patterns 22 Source: Events stored durably until read by all registered consumers. Traditional store-and-forward brokers. ReplayableConsumable Events stored durably for specific period of time or storage capacity. Consumers can move back and forth of the stream. Volatile The event needs to be disseminated to all consumers online at time of publication. Not persisted.
Event-driven Microservices EVENT-DRIVENARCHITECTURE 23
Perfect fit for Microservices 24 ● Broadcast ● Parallel Processing ● Modular ● Fault tolerance nature ● real-time responsiveness
25 Event Sourcing Aync Req and Res Change Data Capture Streaming process Event Driven Architecture Components Communication Query Command Event Volatile Consumable Replayable Connects Transformation Routing Channel Cross cluster On demand Self-service
Event-driven architecture use cases 26 Streaming between data centers Reactive notification Command query responsibility segregation (CQRS) Auditing Behavior capture Cache store Complex event processing
27 Zero One Two Three Four There are absolutely no asynchronous communication in the system. There are some adoption of asynchronized communication between applications. Proper streaming and messaging are installed with some level of high availability plan. Ability to provide limited self-service between departments Discoverability of services, endpoints and data schemas. As well as ability to scale freely on cloud platforms. Provide 100% scalable self service for all developer. Some automations are introduce in to continuous delivery Provide observability of the system environment. And allow tracing of events and monitoring performance of application and resource usage. Automations are built-in to all continuous delivery pipelines Event Driven Architecture Maturity Level
Event Stream Processing EVENT-DRIVENARCHITECTURE 28
29 Event-Driven Architecture A core foundation of modern enterprise IT Event streaming Cloud-native Apache Kafka Topics Events
30 Educational Slides on Event Streaming TBD NOT OURS CHANGE IT!!!
What is Apache Kafka? 31 Apache Kafka is a distributed system designed for streams. It is built to be an horizontally-scalable, fault-tolerant, commit log, and allows distributed data streams and stream processing applications.
Apache Kafka ecosystem 32 ● Kafka Core ○ Broker ○ Producer API, Consumer API, Admin API ○ Management tools ● Kafka Connect ● Kafka Streams API ● Mirror Maker / Mirror Maker 2 ● REST Proxy for bridging HTTP and Kafka ● Schema Registry Streams API Producer API Consumer API 3rd party tools Mirror Maker Connect
Running on OpenShift 33 ● Red Hat AMQ streams provides: ○ Container images for Apache Kafka, Connect, Zookeeper and MirrorMaker ○ Kubernetes Operators for managing/configuring Apache Kafka clusters, topics and users ○ Kafka Consumer, Producer and Admin clients, Kafka Streams ● Upstream Community: Strimzi ○ 100% Open source project licensed under Apache License 2.0 ○ Part of the Cloud Native Computing Foundation (CNCF)
Cluster Deployment with AMQ Streams Topic & User operators Zookeeper cluster Kafka cluster Cluster operator Kafka Custom Resource
Cluster Update with AMQ Streams Topic & User operators Zookeeper cluster Kafka cluster Cluster operator Kafka Custom Resource
Kafka Connect ● Framework for transferring data between Kafka and other data systems. ● Facilitate data conversion, scaling, load balancing, fault tolerance. ● Connector plugins are deployed into Kafka connect cluster: ○ Well defined API for creating new connectors (with Sink/Source) ○ Apache Kafka itself includes only FileSink and FileSource plugins (reading records from file and posting them as Kafka messages / writing Kafka messages to files) ○ Many additional plugins are available outside of Apache Kafka 36
Kafka Streams API 37 ● Stream processing framework. ● Streams are Kafka topics (as input and output). ● It’s really just a Java library to include in your application. ● Creates a topology of processing nodes (filter, map, join etc) acting on a stream: ○ Low level processor API ○ High level DSL ○ Using “internal” topics (when re-partitioning is needed or for “stateful” transformations)
Kafka Connect + Kafka Streams API 38 Other System Other System Kafka Application Topic Topic Processing Processing Topic Topic Topic Topic Streams API Kafka Connect Kafka Connect Source Connector Sink Connector
Kafka HTTP Proxy Bridge 39 ● General Available (GA) since AMQ Streams 1.3 ○ Available on RHEL and on OCP ○ On OCP deployed through the Cluster operator ● On RHEL installed and configured manually ○ Can be used to access Kafka using HTTP REST API ○ Tries to maintain compatibility with the other REST Proxies ● But implements only selected features ● AMQP Bridge (part of the same upstream code) is not supported!
Kafka Mirror Maker 40 ● Kafka is latency sensitive ○ Zookeeper require low latency to maintain a quorum ○ Kafka is less latency sensitive, but replications and reliable producers will be slow with high latency networks ● Kafka clusters do not work well when split across multiple datacenters. It is recommended to setup independent cluster in each data center and mirror data. ● Usual solution is to create local clusters and mirror data between them ● Mirroring with Kafka is always asynchronous!
41 Data Center B Kafka Cluster 1 Topic 1 Partition 0 Data Center A Kafka Cluster 2 Mirror Maker Producer Consumer Mirror Maker Topic 1 Partition 0 Topic 2 Consumer Producer Topic 2 Topic 3 Partition 0 Topic 3 Partition 0 Producer Consumer Mirror Maker
42 Data access patterns for modern applications How data patterns evolve in a microservices world Change Data Capture Incorporating data in to event-driven architecture Topics Data
43 Data Architecture Traditional n-tier app DB stays DB moves DB embeds
44 Data Access Patterns DATA VIRTUALIZATION Federates multiple relational and non-relational sources Compatible with existing SQL clients Live CRUD access to back-end data sources Client-driven interaction CHANGE DATA CAPTURE Database changes externalized as events Process data change events as they happen Distributed state updates, caching, data lake, etc. Event-driven interaction
What is Change Data Capture? 45 ● Well-established software design pattern ● A system monitors and captures the changes in data so that other software can respond to those changes ● CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus ● Applications can read these change event streams and access the change events in the order in which they occurred
What is Change Data Capture? 46 Get an event stream with all data and schema changes in your DB Database Apache Kafka ?E E E E E E E Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7 Topic
Change Data Capture Connectors 47 ● Change Data Capture (CDC) connectors for Kafka Connect ○ Component connects to selected database, reads its transaction log and publishes it as Kafka messages ○ Supported databases are MySQL, PostgreSQL, MongoDB, SQL Server ○ The Kafka messages can be send for example in JSON format ● Makes it easy to integrate DB based applications into Kafka ○ No need to write data to DB and send to Kafka ○ Use cases like, for example, microservices integration and data replication
48 Data Replication Replicate data to another Database Feed Analytics system, Data Lake or DWH Feed data to other teams Microservices Excellent fit for microservices architectures: Propagate data between different services without coupling Each service keeps optimized views locally Other Uses Auditing / Historization Update / Invalidate caches Enable full-text search Update CQRS read models
REST API ODAT A JDBC ODBC REST ● Transformation ● Filtering ● Masking ● Authentication ● Authorization ● Query Optimization ● Materialization ● ... Data Virtualization
50 Cloud-native adoption requires integration Enterprise integration solves organizational and technical challenges associated with cloud adoption. iPaaS Hype and adoption continue to grow. Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community. Topics Enterprise Integration Patterns
Organizational Impacts of Cloud Adoption Central IT as Gatekeeper Central IT viewed as bottleneck Developers and LOB taking more ownership Cloud adoption opens additional paths to production Best practices and institutional knowledge lost IT Central IT as Advisor Distribute IT knowledge throughout organization Facilitate self-service and connect participants Integration Strategy Enablement Team (ISET) Central IT viewed as catalyst
Technology Impacts of Cloud Adoption DISTRIBUTED INTEGRATION ❏ Lightweight ❏ Pattern Based ❏ Event Oriented ❏ Community Sourced FLEXIBILITY Fuse Standalone Fuse on OpenShift Fuse Online
Apache Camel Going Strong Ranked #1 ASF project considering commits number ASF Annual Report 2019 3Camel 3.0 Final - Est. Nov 2019 ● 145 releases ● 508 unique contributors ● 2793 stars on Github ● 3639 forks on Github ● 8985 questions on StackOverFlow ● 68 committers ● 35 PMC members ● 764 Followers on Twitter
54 Cloud-native adoption requires integration Enterprise integration solves organizational and technical challenges associated with cloud adoption. iPaaS Hype and adoption continue to grow. Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community. Topics Enterprise Integration Patterns
Self-service integration Self-service API Design & Implementation Data Mapping Connectivity Intelligent Routing Extensibility
Self-service messaging AMQ Console Create an address Select type & name it Choose plan View summary Address is created Namespace view
Kogito Next-gen Cloud-Native Business Automation Cloud-Native Business Automation for building intelligent applications, backed by battle-tested capabilities A continuation of Drools, jBPM and Optaplanner but completely redesigned to be cloud-native!
58 Source: Quotation: Abraham H. Maslow (1966). The Psychology of Science. p. 15. Image: https://en.wikipedia.org/wiki/Abraham_Maslow#/media/File:Abraham_Maslow.jpg I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. “” Abraham Maslow, 1966
59 A Complete Foundation for the Event-Driven Enterprise DESCRIPTION BEST FIT PATTERNS POSSIBLE PATTERNS UNSUITABLE PATTERNS AMQ BROKER Traditional messaging broker w/ Queuing & Pub/Sub Rich feature set inc. JMS 2.0 Best-in-class performance Based on Apache ActiveMQ Artemis CONSUMABLE EVENTS VOLATILE EVENTS COMMAND QUERY REPLAYABLE EVENTS AMQ INTERCONNECT Message router 1:1 (anycast) and 1:many (multicast) Secure messaging backbone for hybrid cloud Based on Apache Qpid Dispatch Router COMMAND QUERY VOLATILE EVENTS CONSUMABLE EVENTS REPLAYABLE EVENTS AMQ STREAMS Enterprise distribution of Apache Kafka Simplified deployment on OpenShift Based on Kafka and Strimzi REPLAYABLE EVENTS CONSUMABLE EVENTS COMMAND QUERY VOLATILE EVENTS TRANSACTED/FILTERED EVENTS AMQ ONLINE Scalable, “self-service” on OpenShift Available self-managed and Red Hat-managed Based on EnMasse, Artemis & Qpid Dispatch Router COMMAND QUERY VOLATILE EVENTS DURABLE EVENTS REPLAYABLE EVENTS
Single Message Transformations 60 Modify events before storing in Kafka Image Source: “Penknife, Swiss Army Knife” by Emilian Robert Vicol , used under CC BY 2.0 ● Lightweight single message inline transformation ● Format conversions ○ Time/date fields ○ Extract new row state ● Aggregate sharded tables to single topic ● Keep compatibility with existing consumers ● Transformation does not interact with external systems
61 KAFKA Connects VS Camel route
62 Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community Emerging Trends Enterprise Integration Patterns
Apache Camel K ● A platform for directly running integrations on Openshift and Kubernetes ● Based on Operator SDK ● Apache-based, community-driven project ● A subproject of Apache Camel started on August 31st, 2018 https://github.com/apache/camel-k
Integration and Knative ChannelSource Service Source Channel Service
Integration and Knative ChannelSource Service
66 Messaging for the hybrid cloud A uniform messaging overlay for the hybrid-cloud Emerging Trends Events
AMQ Interconnect Event Mesh for Hybrid Cloud Event Mesh • Overlay messaging network spanning private and public cloud for seamless inter-service communication • Secure ○ Mutual TLS between routers with dedicated CA ○ No inbound TCP connections to private cloud • Logical addressing, not hosts & ports Use-cases • Expose data securely from private to public cloud (APIs and Events) • Topology-aware request routing • Cloud bursting • Fault-tolerance
See skupper.io
Hands-on Labs 69
70 The storyline 2024
71 International, Inc - Large global company - Designing its digital transformation strategy - API-first approach to ease merger & acquisition - Fundamental pain-point: profits are impaired by pollution on Earth Fleur de Lune - Eco-friendly startup - Core business focused on space agriculture - Staggering growth, especially among Millennials - Features an amazing culture among I.T. workers - Heavy usage of cloud-native integration technologies The storyline
Installation RHPDS OPENSHIFT WORKSHOP 4.3 Ansible CHE Fuse Online AMQ Online AMQ Streams Infinispan Camel K User Projects Shared Projects Solution Patterns
Personas 73
Data Engineer vs Scientist Tooling 74
Personas???? Lab one Lab two Lab three Lab four Developer 100% Developer 100% Developer 100% Developer 100% Architect 100% Architect 70% Architect 70% Architect 50% Data Eng 0% Data Eng 100% Data Eng 10% Data Eng 30%
EVENT-DRIVENARCHITECTURE Lab 1 Event Driven Architecture 76
77 ● Distributed system ● Multiple machines ● Each service is a process ● Lightweight protocols Need to talk to each other, but how? Customers Billing Administration Operations Customer Service Lab One - Event Driven Architecture
Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
Lab One - Event Driven Architecture invoice inventory order i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP HTTP HTTP SSE AMQP AMQP AMQP AMQP HTTP
Lab One - Event Driven Architecture i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP SSE AMQP AMQP AMQP AMQP
Lab One - Event Driven Architecture i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP SSE AMQP AMQP AMQP AMQP
EVENT-DRIVENARCHITECTURE Lab 2 Change Data Capture & Apache Kafka 84
85 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth Kafka ConnectMS SQL Server
86 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Moon Kafka Connect Mirror Maker MS SQL Server
87 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Moon Kafka Connect Mirror Maker HTTP Bridge Dashboard MS SQL Server
EVENT-DRIVENARCHITECTURE Lab 3 Cloud-native EIPs 88
Lab Three - Cloud-Native Integration with EIPs IoT IoT IoT IoT IoT IoT IoT IoT Queue Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm EdgeSimulator Config
Lab Three - Cloud-Native Integration with EIPs Queue streams streams Wiretap Inventory Config Premium Standard Utility CBR Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm Splitter Inventory Store
Lab Three - Cloud-Native Integration with EIPs Shipping Cargo No Cart No mmid Streams Premium Streams Standard Premium Shipping Assignment Standard Shipping Assignment Shipping Cargo No Weight Data Lake (Cache) Data Lake (Cache)
Lab Three - Cloud-Native Integration with EIPs Shipping Cargo No Cart No mmid Shipping Cargo No Weight Data Lake (Cache) Data Lake (Cache) Console Service API Inventory Store Grafana
EVENT-DRIVENARCHITECTURE Lab 4 Event Sourcing & CEP 93
Lab Four - Event Sourcing and CEP Queue Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm Streams Cost Center Cost Center Filter Cost Advisor Calculate Cost Reset offset Replay What if cost increased?
Lab Four - Event Sourcing and CEP Farm CEP Kogito Time window Streams Rules Determine Disaster Events EventsFarm Harvest Events EventsFarm Harvest Every 10 Seconds Events EventsFarm Harvest Events EventsFarm Harvest Every 10 Seconds < 150 every 10 Sec means disaster Streams Disaster Streams Disaster Farm Farm Farm Farm
Environment URL: https://tutorial.blah.redhat.com User: userX Password: openshift 96
linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you 97

More Related Content

Day in the life event-driven workshop

  • 2. AGENDA 2 Discover Event Driven Architecture, Change Data Capture and Cloud-Native Integration Introduction Walkthrough of the lab environment, including a demonstration of one the labs Demo Presenter will distribute the lab URL and credentials Hands-on Workshop 10:00 11:00
  • 4. 4 Modern application architectures Ye Olde Architecture Enlightened postmodernism
  • 5. 5 Modern application architectures APIs Events Data Enterprise Integration Patterns Process and Decisions
  • 8. 8 What the container is on matters Platform
  • 9. 9 What’s in the container matters Runtime Frameworks {Your business code} OS
  • 10. 10 What’s around the container matters Function- as-a-service Messaging API management Rules Process automation
  • 11. 11 How it’s all engineered together matters {Your Code} Integration Runtime Function- as-a-service Messaging API management Rules Process automation
  • 13. Red Hat Cloud-Native Dev Platform 13 Our vision is to simplify the creation of cloud-native services and serverless functions with a rich set of components and tools to match the workloads of modern cloud native apps. Cloud-native middleware applications services and service mesh Automate Kubernetes application operations with DevOps in mind Tools and standard processes to increase developer productivity on Kubernetes
  • 14. Supporting Container and Cloud Native workloads 14 PROCESS AUTOMATIONAPPLICATION RUNTIMES AUTOMATE AND OPTIMIZE BUSINESS PROCESSES Develop, deploy, and manage across cloud and on-premise CORE TOOLS TO BUILD CLOUD NATIVE & MIGRATE EXISTING APPS COMPOSE AND INTEGRATE MICROSERVICES ACROSS AN ENTERPRISE SERVICE NETWORK INTEGRATION Integration with Red Hat Developer, CI/CD tools, & security services Optimized for Red Hat OpenShift & Kubernetes services Support organizations desire for choice and process standardization Emphasis on solution Simplified selling motion Flexible consumption BROKER
  • 16. Events Use Cases ● Traditional MOM ● Multicloud ● IoT ● Event streaming Data Use Cases ● Data Federation ● Data Virtualization ● Data as APIs ● Change Data Capture EIPs Use Cases ● Intelligent Routing ● Service composition ● Connectivity ● iPaaS ● Data Pipelines APIs Use Cases ● Microservices ● Mobile Apps ● B2B ● Legacy facade Elements of Modern Applications APIs EIPs DataEvents
  • 19. What is Event Driven Architecture? 19 Event-Driven Architecture (EDA) is a way of designing applications and services to respond to real-time information based on the sending and receiving of information about individual event notifications
  • 20. What is an event? 20 Image: https://foto.wuestenigel.com/hand-ready-to-push-domino-pieces/ Event an action or occurrence recognized by software, often originating asynchronously from the external environment, that may be handled by the software
  • 21. What is an event? 21 Source: Async form of Remote Procedure Call, contains instructions telling recipient what to do, may cause a change of state. QueryCommand Similar to commands, queries expect a response returning the results, but does not cause any change in state. Event Immutable state and value of a particular entity, which occurred during operation among services.
  • 22. Types of event consumption patterns 22 Source: Events stored durably until read by all registered consumers. Traditional store-and-forward brokers. ReplayableConsumable Events stored durably for specific period of time or storage capacity. Consumers can move back and forth of the stream. Volatile The event needs to be disseminated to all consumers online at time of publication. Not persisted.
  • 24. Perfect fit for Microservices 24 ● Broadcast ● Parallel Processing ● Modular ● Fault tolerance nature ● real-time responsiveness
  • 25. 25 Event Sourcing Aync Req and Res Change Data Capture Streaming process Event Driven Architecture Components Communication Query Command Event Volatile Consumable Replayable Connects Transformation Routing Channel Cross cluster On demand Self-service
  • 26. Event-driven architecture use cases 26 Streaming between data centers Reactive notification Command query responsibility segregation (CQRS) Auditing Behavior capture Cache store Complex event processing
  • 27. 27 Zero One Two Three Four There are absolutely no asynchronous communication in the system. There are some adoption of asynchronized communication between applications. Proper streaming and messaging are installed with some level of high availability plan. Ability to provide limited self-service between departments Discoverability of services, endpoints and data schemas. As well as ability to scale freely on cloud platforms. Provide 100% scalable self service for all developer. Some automations are introduce in to continuous delivery Provide observability of the system environment. And allow tracing of events and monitoring performance of application and resource usage. Automations are built-in to all continuous delivery pipelines Event Driven Architecture Maturity Level
  • 29. 29 Event-Driven Architecture A core foundation of modern enterprise IT Event streaming Cloud-native Apache Kafka Topics Events
  • 30. 30 Educational Slides on Event Streaming TBD NOT OURS CHANGE IT!!!
  • 31. What is Apache Kafka? 31 Apache Kafka is a distributed system designed for streams. It is built to be an horizontally-scalable, fault-tolerant, commit log, and allows distributed data streams and stream processing applications.
  • 32. Apache Kafka ecosystem 32 ● Kafka Core ○ Broker ○ Producer API, Consumer API, Admin API ○ Management tools ● Kafka Connect ● Kafka Streams API ● Mirror Maker / Mirror Maker 2 ● REST Proxy for bridging HTTP and Kafka ● Schema Registry Streams API Producer API Consumer API 3rd party tools Mirror Maker Connect
  • 33. Running on OpenShift 33 ● Red Hat AMQ streams provides: ○ Container images for Apache Kafka, Connect, Zookeeper and MirrorMaker ○ Kubernetes Operators for managing/configuring Apache Kafka clusters, topics and users ○ Kafka Consumer, Producer and Admin clients, Kafka Streams ● Upstream Community: Strimzi ○ 100% Open source project licensed under Apache License 2.0 ○ Part of the Cloud Native Computing Foundation (CNCF)
  • 34. Cluster Deployment with AMQ Streams Topic & User operators Zookeeper cluster Kafka cluster Cluster operator Kafka Custom Resource
  • 35. Cluster Update with AMQ Streams Topic & User operators Zookeeper cluster Kafka cluster Cluster operator Kafka Custom Resource
  • 36. Kafka Connect ● Framework for transferring data between Kafka and other data systems. ● Facilitate data conversion, scaling, load balancing, fault tolerance. ● Connector plugins are deployed into Kafka connect cluster: ○ Well defined API for creating new connectors (with Sink/Source) ○ Apache Kafka itself includes only FileSink and FileSource plugins (reading records from file and posting them as Kafka messages / writing Kafka messages to files) ○ Many additional plugins are available outside of Apache Kafka 36
  • 37. Kafka Streams API 37 ● Stream processing framework. ● Streams are Kafka topics (as input and output). ● It’s really just a Java library to include in your application. ● Creates a topology of processing nodes (filter, map, join etc) acting on a stream: ○ Low level processor API ○ High level DSL ○ Using “internal” topics (when re-partitioning is needed or for “stateful” transformations)
  • 38. Kafka Connect + Kafka Streams API 38 Other System Other System Kafka Application Topic Topic Processing Processing Topic Topic Topic Topic Streams API Kafka Connect Kafka Connect Source Connector Sink Connector
  • 39. Kafka HTTP Proxy Bridge 39 ● General Available (GA) since AMQ Streams 1.3 ○ Available on RHEL and on OCP ○ On OCP deployed through the Cluster operator ● On RHEL installed and configured manually ○ Can be used to access Kafka using HTTP REST API ○ Tries to maintain compatibility with the other REST Proxies ● But implements only selected features ● AMQP Bridge (part of the same upstream code) is not supported!
  • 40. Kafka Mirror Maker 40 ● Kafka is latency sensitive ○ Zookeeper require low latency to maintain a quorum ○ Kafka is less latency sensitive, but replications and reliable producers will be slow with high latency networks ● Kafka clusters do not work well when split across multiple datacenters. It is recommended to setup independent cluster in each data center and mirror data. ● Usual solution is to create local clusters and mirror data between them ● Mirroring with Kafka is always asynchronous!
  • 41. 41 Data Center B Kafka Cluster 1 Topic 1 Partition 0 Data Center A Kafka Cluster 2 Mirror Maker Producer Consumer Mirror Maker Topic 1 Partition 0 Topic 2 Consumer Producer Topic 2 Topic 3 Partition 0 Topic 3 Partition 0 Producer Consumer Mirror Maker
  • 42. 42 Data access patterns for modern applications How data patterns evolve in a microservices world Change Data Capture Incorporating data in to event-driven architecture Topics Data
  • 43. 43 Data Architecture Traditional n-tier app DB stays DB moves DB embeds
  • 44. 44 Data Access Patterns DATA VIRTUALIZATION Federates multiple relational and non-relational sources Compatible with existing SQL clients Live CRUD access to back-end data sources Client-driven interaction CHANGE DATA CAPTURE Database changes externalized as events Process data change events as they happen Distributed state updates, caching, data lake, etc. Event-driven interaction
  • 45. What is Change Data Capture? 45 ● Well-established software design pattern ● A system monitors and captures the changes in data so that other software can respond to those changes ● CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus ● Applications can read these change event streams and access the change events in the order in which they occurred
  • 46. What is Change Data Capture? 46 Get an event stream with all data and schema changes in your DB Database Apache Kafka ?E E E E E E E Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Offset 6 Offset 7 Topic
  • 47. Change Data Capture Connectors 47 ● Change Data Capture (CDC) connectors for Kafka Connect ○ Component connects to selected database, reads its transaction log and publishes it as Kafka messages ○ Supported databases are MySQL, PostgreSQL, MongoDB, SQL Server ○ The Kafka messages can be send for example in JSON format ● Makes it easy to integrate DB based applications into Kafka ○ No need to write data to DB and send to Kafka ○ Use cases like, for example, microservices integration and data replication
  • 48. 48 Data Replication Replicate data to another Database Feed Analytics system, Data Lake or DWH Feed data to other teams Microservices Excellent fit for microservices architectures: Propagate data between different services without coupling Each service keeps optimized views locally Other Uses Auditing / Historization Update / Invalidate caches Enable full-text search Update CQRS read models
  • 49. REST API ODAT A JDBC ODBC REST ● Transformation ● Filtering ● Masking ● Authentication ● Authorization ● Query Optimization ● Materialization ● ... Data Virtualization
  • 50. 50 Cloud-native adoption requires integration Enterprise integration solves organizational and technical challenges associated with cloud adoption. iPaaS Hype and adoption continue to grow. Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community. Topics Enterprise Integration Patterns
  • 51. Organizational Impacts of Cloud Adoption Central IT as Gatekeeper Central IT viewed as bottleneck Developers and LOB taking more ownership Cloud adoption opens additional paths to production Best practices and institutional knowledge lost IT Central IT as Advisor Distribute IT knowledge throughout organization Facilitate self-service and connect participants Integration Strategy Enablement Team (ISET) Central IT viewed as catalyst
  • 52. Technology Impacts of Cloud Adoption DISTRIBUTED INTEGRATION ❏ Lightweight ❏ Pattern Based ❏ Event Oriented ❏ Community Sourced FLEXIBILITY Fuse Standalone Fuse on OpenShift Fuse Online
  • 53. Apache Camel Going Strong Ranked #1 ASF project considering commits number ASF Annual Report 2019 3Camel 3.0 Final - Est. Nov 2019 ● 145 releases ● 508 unique contributors ● 2793 stars on Github ● 3639 forks on Github ● 8985 questions on StackOverFlow ● 68 committers ● 35 PMC members ● 764 Followers on Twitter
  • 54. 54 Cloud-native adoption requires integration Enterprise integration solves organizational and technical challenges associated with cloud adoption. iPaaS Hype and adoption continue to grow. Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community. Topics Enterprise Integration Patterns
  • 55. Self-service integration Self-service API Design & Implementation Data Mapping Connectivity Intelligent Routing Extensibility
  • 56. Self-service messaging AMQ Console Create an address Select type & name it Choose plan View summary Address is created Namespace view
  • 57. Kogito Next-gen Cloud-Native Business Automation Cloud-Native Business Automation for building intelligent applications, backed by battle-tested capabilities A continuation of Drools, jBPM and Optaplanner but completely redesigned to be cloud-native!
  • 58. 58 Source: Quotation: Abraham H. Maslow (1966). The Psychology of Science. p. 15. Image: https://en.wikipedia.org/wiki/Abraham_Maslow#/media/File:Abraham_Maslow.jpg I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. “” Abraham Maslow, 1966
  • 59. 59 A Complete Foundation for the Event-Driven Enterprise DESCRIPTION BEST FIT PATTERNS POSSIBLE PATTERNS UNSUITABLE PATTERNS AMQ BROKER Traditional messaging broker w/ Queuing & Pub/Sub Rich feature set inc. JMS 2.0 Best-in-class performance Based on Apache ActiveMQ Artemis CONSUMABLE EVENTS VOLATILE EVENTS COMMAND QUERY REPLAYABLE EVENTS AMQ INTERCONNECT Message router 1:1 (anycast) and 1:many (multicast) Secure messaging backbone for hybrid cloud Based on Apache Qpid Dispatch Router COMMAND QUERY VOLATILE EVENTS CONSUMABLE EVENTS REPLAYABLE EVENTS AMQ STREAMS Enterprise distribution of Apache Kafka Simplified deployment on OpenShift Based on Kafka and Strimzi REPLAYABLE EVENTS CONSUMABLE EVENTS COMMAND QUERY VOLATILE EVENTS TRANSACTED/FILTERED EVENTS AMQ ONLINE Scalable, “self-service” on OpenShift Available self-managed and Red Hat-managed Based on EnMasse, Artemis & Qpid Dispatch Router COMMAND QUERY VOLATILE EVENTS DURABLE EVENTS REPLAYABLE EVENTS
  • 60. Single Message Transformations 60 Modify events before storing in Kafka Image Source: “Penknife, Swiss Army Knife” by Emilian Robert Vicol , used under CC BY 2.0 ● Lightweight single message inline transformation ● Format conversions ○ Time/date fields ○ Extract new row state ● Aggregate sharded tables to single topic ● Keep compatibility with existing consumers ● Transformation does not interact with external systems
  • 61. 61 KAFKA Connects VS Camel route
  • 62. 62 Serverless and Knative Integration concerns begin to surface as first class concepts in Kubernetes community Emerging Trends Enterprise Integration Patterns
  • 63. Apache Camel K ● A platform for directly running integrations on Openshift and Kubernetes ● Based on Operator SDK ● Apache-based, community-driven project ● A subproject of Apache Camel started on August 31st, 2018 https://github.com/apache/camel-k
  • 64. Integration and Knative ChannelSource Service Source Channel Service
  • 66. 66 Messaging for the hybrid cloud A uniform messaging overlay for the hybrid-cloud Emerging Trends Events
  • 67. AMQ Interconnect Event Mesh for Hybrid Cloud Event Mesh • Overlay messaging network spanning private and public cloud for seamless inter-service communication • Secure ○ Mutual TLS between routers with dedicated CA ○ No inbound TCP connections to private cloud • Logical addressing, not hosts & ports Use-cases • Expose data securely from private to public cloud (APIs and Events) • Topology-aware request routing • Cloud bursting • Fault-tolerance
  • 71. 71 International, Inc - Large global company - Designing its digital transformation strategy - API-first approach to ease merger & acquisition - Fundamental pain-point: profits are impaired by pollution on Earth Fleur de Lune - Eco-friendly startup - Core business focused on space agriculture - Staggering growth, especially among Millennials - Features an amazing culture among I.T. workers - Heavy usage of cloud-native integration technologies The storyline
  • 72. Installation RHPDS OPENSHIFT WORKSHOP 4.3 Ansible CHE Fuse Online AMQ Online AMQ Streams Infinispan Camel K User Projects Shared Projects Solution Patterns
  • 74. Data Engineer vs Scientist Tooling 74
  • 75. Personas???? Lab one Lab two Lab three Lab four Developer 100% Developer 100% Developer 100% Developer 100% Architect 100% Architect 70% Architect 70% Architect 50% Data Eng 0% Data Eng 100% Data Eng 10% Data Eng 30%
  • 77. 77 ● Distributed system ● Multiple machines ● Each service is a process ● Lightweight protocols Need to talk to each other, but how? Customers Billing Administration Operations Customer Service Lab One - Event Driven Architecture
  • 78. Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
  • 79. Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
  • 80. Lab One - Event Driven Architecture invoice inventory order HTTP HTTP HTTP
  • 81. Lab One - Event Driven Architecture invoice inventory order i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP HTTP HTTP SSE AMQP AMQP AMQP AMQP HTTP
  • 82. Lab One - Event Driven Architecture i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP SSE AMQP AMQP AMQP AMQP
  • 83. Lab One - Event Driven Architecture i-invoice i-inventory i-order AMQONLINE events Incomingorders/notifications HTTP SSE AMQP AMQP AMQP AMQP
  • 84. EVENT-DRIVENARCHITECTURE Lab 2 Change Data Capture & Apache Kafka 84
  • 85. 85 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth Kafka ConnectMS SQL Server
  • 86. 86 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Moon Kafka Connect Mirror Maker MS SQL Server
  • 87. 87 Lab Two - Change Data Capture (CDC) Enterprise | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Earth | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Moon Kafka Connect Mirror Maker HTTP Bridge Dashboard MS SQL Server
  • 89. Lab Three - Cloud-Native Integration with EIPs IoT IoT IoT IoT IoT IoT IoT IoT Queue Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm EdgeSimulator Config
  • 90. Lab Three - Cloud-Native Integration with EIPs Queue streams streams Wiretap Inventory Config Premium Standard Utility CBR Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm Splitter Inventory Store
  • 91. Lab Three - Cloud-Native Integration with EIPs Shipping Cargo No Cart No mmid Streams Premium Streams Standard Premium Shipping Assignment Standard Shipping Assignment Shipping Cargo No Weight Data Lake (Cache) Data Lake (Cache)
  • 92. Lab Three - Cloud-Native Integration with EIPs Shipping Cargo No Cart No mmid Shipping Cargo No Weight Data Lake (Cache) Data Lake (Cache) Console Service API Inventory Store Grafana
  • 94. Lab Four - Event Sourcing and CEP Queue Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Marshmello.. Farm Streams Cost Center Cost Center Filter Cost Advisor Calculate Cost Reset offset Replay What if cost increased?
  • 95. Lab Four - Event Sourcing and CEP Farm CEP Kogito Time window Streams Rules Determine Disaster Events EventsFarm Harvest Events EventsFarm Harvest Every 10 Seconds Events EventsFarm Harvest Events EventsFarm Harvest Every 10 Seconds < 150 every 10 Sec means disaster Streams Disaster Streams Disaster Farm Farm Farm Farm
  • 97. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you 97