SlideShare a Scribd company logo

„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

•
•
Guido Schmutz
Guido Schmutz

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. In this session an architecture with a central log structured storage is presented where anybody can store and subscribe for events. This can be implemented using frameworks such as Kafka, Storm, Samza and Spark Streaming.

Related slideshows

Data Democratization at Nubank
Data Democratization at Nubank Data Democratization at Nubank
Data Democratization at Nubank
 
Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning
Data Preparation vs. Inline Data Wrangling in Data Science and Machine LearningData Preparation vs. Inline Data Wrangling in Data Science and Machine Learning
Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning
 
"Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about...
"Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about..."Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about...
"Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about...
 
1 of 30
Download to read offline
2015 © Trivadis BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURGI.BR. HAMBURG MÜNCHEN STUTTGART WIEN 2014 © Trivadis „Enterprise Event Bus“ Unified Log (Event) Processing Architecture DOAG DevCamp 2015 Guido Schmutz Trivadis AG April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 1
2015 © Trivadis A little story of a “real-life” customer situation Traditional system interact with its clients and does its work Implemented using legacy technologies (i.e. PL/SQL) New requirement: • Offer notification service to notify customerwhen goods are shipped • Subscription and inform over different channels • Existing technology doesn’tfit May 2015 Blueprints for the analysis of social media 2 delivery Logistic System Oracle Mobile Apps Sensor ship sort 2 Rich (Web) Client Apps DB schedule Logic (PL/SQL) delivery
2015 © Trivadis A little story of a “real-life” customer situation May 2015 Blueprints for the analysis of social media 3 delivery Logistic System Oracle Mobile Apps Sensor ship sort 3 Rich (Web) Client Apps DB schedule Notification Logic (PL/SQL) Logic (Java) delivery SMS Email … Events are “owned” by traditional application (as well as the channels they are transported over)
2015 © Trivadis A little story of a “real-life” customer situation Rule Engine implemented in Java and invoked from OSB message flow Notification system informed via queue Higher Latency introduced (good enough in this case) integrate in order to get the information! Oracle Service Bus was already there May 2015 Blueprints for the analysis of social media 4 delivery Logistic System Oracle Oracle Service Bus Mobile Apps Sensor AQship sort 4 Rich (Web) Client Apps DB schedule Filter Notification Logic (PL/SQL) JMS Rule Engine (Java) Logic (Java) delivery shipdelivery delivery true SMS Email …
2015 © Trivadis A little story of a “real-life” customer situation May 2015 Blueprints for the analysis of social media 5 delivery Logistic System Oracle Oracle Service Bus Mobile Apps Sensor AQship sort 5 Rich (Web) Client Apps DB schedule Filter Notification Logic (PL/SQL) JMS Rule Engine (Java) Logic (Java) delivery shipdelivery delivery true SMS Email … Treat events as first-class citizens Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !! Event (stream) processing can be introduced and by that latency reduced!
2015 © Trivadis Why Stream Processing? Response latency Stream Processing Milliseconds to minutes RPC Synchronous Later. Possibly much later. April 2015 Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared 6
2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 7
2015 © Trivadis What is Stream Processing? April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 8 Event Stream event Collecting event Persist (Queue) Event Stream event Collecting event Processing event Processing result result Event Stream event Collecting/ Processing result
2015 © Trivadis What is Stream Processing? Infrastructure for continuous data processing Computational model can be as general as MapReduce but with the ability to produce low-latency results Data collected continuously is naturally processed continuously aka. Event Processing / Complex Event Processing (CEP) April 2015 Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared 9
2015 © Trivadis How to design a Streaming Processing System? It usually starts very simple … just one data pipeline April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 10 Event Stream ConsumereventApplication
2015 © Trivadis New Event Stream sources are added … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 11 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream event event event event Application 2nd Application 3rd Application Nth Application
2015 © Trivadis New Processors are interested in the events … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 12 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumerevent event event event Application 2nd Application 3rd Application Nth Application
2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 13 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumer 3rd Consumer Nth Consumer event event event event Application 2nd Application 3rd Application Nth Application
2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 14 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumer 3rd Consumer Nth Consumer event event event event Application 2nd Application 3rd Application Nth Application
2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 15 New Customers Operational Logs Click Stream Meter Readings event event event event CDC Collector Log Collector Click Stream Collector Senor Collector Hadoop/Data Warehouse Recommendation System Log Search Fraud Detection
2015 © Trivadis Decouple event streams from consumers April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 16 „Unified Log“ Remember Enterprise Service Bus (ESB) ? Enterprise Event Bus Event Stream ProcessorEvent Stream Source New Customers Operational Logs Click Stream Meter Readings CDC Collector Log Collector Click Stream Collector Senor Collector Hadoop/Data Warehouse Recommendation System Log Search Fraud Detection What is the idea of a Unified Log?
2015 © Trivadis Unified Log – What is it? By Unified Log, we do not mean this …. 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-admin/images/date-button.gif HTTP/1.1" 200 111 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 200 13593 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=349-20805 HTTP/1.1" 200 101114 137.229.78.245 - - [02/Jul/2012:13:22:28 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30747 137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "POST /wp-admin/post.php HTTP/1.1" 302 - 137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "GET /wp-admin/post.php?post=387&action=edit&message=1 HTTP/1.1" 200 73160 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/css/editor.css?ver=3.4.1 HTTP/1.1" 304 - 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 304 - 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30809 … but this • a structured log (records are numbered beginning with 0 based on order they are written) • aka. commit log or journal April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 17 0 1 2 3 4 5 6 7 8 9 10 11 1st record Next record written
2015 © Trivadis Central Unified Log for (real-time) subscription Take all the organization’s data (events) and put it into a central log for subscription Properties of the Unified Log: • Unified: “Enterprise”, single deployment • Append-Only:events are appended,no update in place => immutable • Ordered: each event has an offset, which is unique within a shard • Fast: should be able to handle thousands ofmessages /sec • Distributed: lives on a cluster of machines April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 18 0 1 2 3 4 5 6 7 8 9 10 11 reads writes Collector Consumer System A (time = 6) Consumer System B (time = 10) reads
2015 © Trivadis Unified Log Processing Architecture Stream processing allows for computing feeds off other feeds Derived feeds are no different than original feeds they are computed off Single deploymentof “Unified Log” logically different feeds April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 19 Meter Readings Collector Enrich / Transform Aggregate by Minute Raw Meter Readings Meter & Customer Meter by Customer by Minute Customer Aggregate by Minute Meter by Minute Persist Meter by Minute Persist Raw Meter Readings ….
2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 20
2015 © Trivadis Apache Kafka - Overview • A distributed publish-subscribe messaging system • Designed for processing of real time activity stream data (logs, metrics collections,social media streams, …) • Initially developed at LinkedIn, now part of Apache • Does not follow JMS Standards and does not use JMS API • Kafka maintains feeds of messages in topics April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 21 Kafka Cluster Consumer Consumer Consumer Producer Producer Producer 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 Anatomy of a topic: Partition 0 Partition 1 Partition 2 Writes old new
2015 © Trivadis Apache Kafka - Performance Kafka at LinkedIn Up to 2 million writes/sec on 3 cheap machines § Using 3 producers on 3 different machines April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 22 10+ billion writes per day 172k messages per second (average) 55+ billion messages per day to real-time consumers http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
2015 © Trivadis Apache Kafka - Partition offsets Offset: messages in the partitions are each assigned a unique (per partition) and sequential id called the offset • Consumers track their pointers via (offset, partition, topic) tuples April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 23 Consumer group C1
2015 © Trivadis Apache Kafka – two Options for Log Cleanup April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 24 1. Retaining a window of data • Ideal for event data • Window can be defined in time (days) or space (GBs) – defaults to 1 week 2. Retain a complete log (log compaction) • Ideal for keyed data • Keep a space-efficientcomplete log of changes • Log compaction runs in the background • Ensures that always at least the last known value for each message key within the log of data is retained
2015 © Trivadis Unified Log Alternatives • Amazon Kinesis (http://aws.amazon.com/kinesis/) • Confluent (http://confluent.io/) • Redis Pub/Sub (http://redis.io/topics/pubsub) • Kestrel (http://robey.github.io/kestrel/) • ZeroMQ (http://zeromq.org/) • RabbitMQ (http://www.rabbitmq.com/) • Oracle GoldenGate (http://bit.ly/g-gate) • JMS compliant Server • Apache ActiveMQ (http://activemq.apache.org/) • Weblogic JMS (http://www.oracle.com/technetwork/middleware/weblogic/overview/index.html ) • IBM Websphere MQ (http://www-03.ibm.com/software/products/de/ibm-mq) • … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 25
2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 26
2015 © Trivadis Unified Log Processing Architecture in Trivadis CRA April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 27 Tweets Filter and Unify Persist Tweet Filtered Tweets Split Text Words Count over Time Count by Minute Analyze Sentiment Tweet & Sentiment Remove Stopwords Tweet Tweets Consumer Twitter Filter Stream Sensor Layer Distribution Layer Speed Layer Kafka Storm Cassandra Elasticsearch
2015 © Trivadis Unified Log Processing Architecture in Trivadis CRA April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 28 Tweets Filter and Unify Persist Tweet Filtered Tweets Split Text Words Count over Time Count by Minute Analyze Sentiment Tweet & Sentiment Remove Stopwords Tweet Tweets Consumer Twitter Filter Stream Sensor Layer Distribution Layer Splitter Kafka Spout Word Remover Splitter Word Remover Shuffle Fields Kafka Kafka Word Remover Storm Topology Speed Layer Kafka Storm Cassandra Elasticsearch
2015 © Trivadis Weitere Informationen... April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 29
2015 © Trivadis BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURGI.BR. HAMBURG MÜNCHEN STUTTGART WIEN Fragen und Antworten... 2013 © Trivadis April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

More Related Content

„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

  • 1. 2015 © Trivadis BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURGI.BR. HAMBURG MÜNCHEN STUTTGART WIEN 2014 © Trivadis „Enterprise Event Bus“ Unified Log (Event) Processing Architecture DOAG DevCamp 2015 Guido Schmutz Trivadis AG April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 1
  • 2. 2015 © Trivadis A little story of a “real-life” customer situation Traditional system interact with its clients and does its work Implemented using legacy technologies (i.e. PL/SQL) New requirement: • Offer notification service to notify customerwhen goods are shipped • Subscription and inform over different channels • Existing technology doesn’tfit May 2015 Blueprints for the analysis of social media 2 delivery Logistic System Oracle Mobile Apps Sensor ship sort 2 Rich (Web) Client Apps DB schedule Logic (PL/SQL) delivery
  • 3. 2015 © Trivadis A little story of a “real-life” customer situation May 2015 Blueprints for the analysis of social media 3 delivery Logistic System Oracle Mobile Apps Sensor ship sort 3 Rich (Web) Client Apps DB schedule Notification Logic (PL/SQL) Logic (Java) delivery SMS Email … Events are “owned” by traditional application (as well as the channels they are transported over)
  • 4. 2015 © Trivadis A little story of a “real-life” customer situation Rule Engine implemented in Java and invoked from OSB message flow Notification system informed via queue Higher Latency introduced (good enough in this case) integrate in order to get the information! Oracle Service Bus was already there May 2015 Blueprints for the analysis of social media 4 delivery Logistic System Oracle Oracle Service Bus Mobile Apps Sensor AQship sort 4 Rich (Web) Client Apps DB schedule Filter Notification Logic (PL/SQL) JMS Rule Engine (Java) Logic (Java) delivery shipdelivery delivery true SMS Email …
  • 5. 2015 © Trivadis A little story of a “real-life” customer situation May 2015 Blueprints for the analysis of social media 5 delivery Logistic System Oracle Oracle Service Bus Mobile Apps Sensor AQship sort 5 Rich (Web) Client Apps DB schedule Filter Notification Logic (PL/SQL) JMS Rule Engine (Java) Logic (Java) delivery shipdelivery delivery true SMS Email … Treat events as first-class citizens Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !! Event (stream) processing can be introduced and by that latency reduced!
  • 6. 2015 © Trivadis Why Stream Processing? Response latency Stream Processing Milliseconds to minutes RPC Synchronous Later. Possibly much later. April 2015 Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared 6
  • 7. 2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 7
  • 8. 2015 © Trivadis What is Stream Processing? April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 8 Event Stream event Collecting event Persist (Queue) Event Stream event Collecting event Processing event Processing result result Event Stream event Collecting/ Processing result
  • 9. 2015 © Trivadis What is Stream Processing? Infrastructure for continuous data processing Computational model can be as general as MapReduce but with the ability to produce low-latency results Data collected continuously is naturally processed continuously aka. Event Processing / Complex Event Processing (CEP) April 2015 Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared 9
  • 10. 2015 © Trivadis How to design a Streaming Processing System? It usually starts very simple … just one data pipeline April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 10 Event Stream ConsumereventApplication
  • 11. 2015 © Trivadis New Event Stream sources are added … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 11 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream event event event event Application 2nd Application 3rd Application Nth Application
  • 12. 2015 © Trivadis New Processors are interested in the events … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 12 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumerevent event event event Application 2nd Application 3rd Application Nth Application
  • 13. 2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 13 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumer 3rd Consumer Nth Consumer event event event event Application 2nd Application 3rd Application Nth Application
  • 14. 2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 14 Event Stream Consumer 2nd Event Stream 3rd Event Stream nth Event Stream 2nd Consumer 3rd Consumer Nth Consumer event event event event Application 2nd Application 3rd Application Nth Application
  • 15. 2015 © Trivadis … and the solution becomes the problem April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 15 New Customers Operational Logs Click Stream Meter Readings event event event event CDC Collector Log Collector Click Stream Collector Senor Collector Hadoop/Data Warehouse Recommendation System Log Search Fraud Detection
  • 16. 2015 © Trivadis Decouple event streams from consumers April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 16 „Unified Log“ Remember Enterprise Service Bus (ESB) ? Enterprise Event Bus Event Stream ProcessorEvent Stream Source New Customers Operational Logs Click Stream Meter Readings CDC Collector Log Collector Click Stream Collector Senor Collector Hadoop/Data Warehouse Recommendation System Log Search Fraud Detection What is the idea of a Unified Log?
  • 17. 2015 © Trivadis Unified Log – What is it? By Unified Log, we do not mean this …. 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-admin/images/date-button.gif HTTP/1.1" 200 111 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 200 13593 137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=349-20805 HTTP/1.1" 200 101114 137.229.78.245 - - [02/Jul/2012:13:22:28 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30747 137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "POST /wp-admin/post.php HTTP/1.1" 302 - 137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "GET /wp-admin/post.php?post=387&action=edit&message=1 HTTP/1.1" 200 73160 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/css/editor.css?ver=3.4.1 HTTP/1.1" 304 - 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 304 - 137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30809 … but this • a structured log (records are numbered beginning with 0 based on order they are written) • aka. commit log or journal April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 17 0 1 2 3 4 5 6 7 8 9 10 11 1st record Next record written
  • 18. 2015 © Trivadis Central Unified Log for (real-time) subscription Take all the organization’s data (events) and put it into a central log for subscription Properties of the Unified Log: • Unified: “Enterprise”, single deployment • Append-Only:events are appended,no update in place => immutable • Ordered: each event has an offset, which is unique within a shard • Fast: should be able to handle thousands ofmessages /sec • Distributed: lives on a cluster of machines April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 18 0 1 2 3 4 5 6 7 8 9 10 11 reads writes Collector Consumer System A (time = 6) Consumer System B (time = 10) reads
  • 19. 2015 © Trivadis Unified Log Processing Architecture Stream processing allows for computing feeds off other feeds Derived feeds are no different than original feeds they are computed off Single deploymentof “Unified Log” logically different feeds April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 19 Meter Readings Collector Enrich / Transform Aggregate by Minute Raw Meter Readings Meter & Customer Meter by Customer by Minute Customer Aggregate by Minute Meter by Minute Persist Meter by Minute Persist Raw Meter Readings ….
  • 20. 2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 20
  • 21. 2015 © Trivadis Apache Kafka - Overview • A distributed publish-subscribe messaging system • Designed for processing of real time activity stream data (logs, metrics collections,social media streams, …) • Initially developed at LinkedIn, now part of Apache • Does not follow JMS Standards and does not use JMS API • Kafka maintains feeds of messages in topics April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 21 Kafka Cluster Consumer Consumer Consumer Producer Producer Producer 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 Anatomy of a topic: Partition 0 Partition 1 Partition 2 Writes old new
  • 22. 2015 © Trivadis Apache Kafka - Performance Kafka at LinkedIn Up to 2 million writes/sec on 3 cheap machines § Using 3 producers on 3 different machines April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 22 10+ billion writes per day 172k messages per second (average) 55+ billion messages per day to real-time consumers http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
  • 23. 2015 © Trivadis Apache Kafka - Partition offsets Offset: messages in the partitions are each assigned a unique (per partition) and sequential id called the offset • Consumers track their pointers via (offset, partition, topic) tuples April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 23 Consumer group C1
  • 24. 2015 © Trivadis Apache Kafka – two Options for Log Cleanup April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 24 1. Retaining a window of data • Ideal for event data • Window can be defined in time (days) or space (GBs) – defaults to 1 week 2. Retain a complete log (log compaction) • Ideal for keyed data • Keep a space-efficientcomplete log of changes • Log compaction runs in the background • Ensures that always at least the last known value for each message key within the log of data is retained
  • 25. 2015 © Trivadis Unified Log Alternatives • Amazon Kinesis (http://aws.amazon.com/kinesis/) • Confluent (http://confluent.io/) • Redis Pub/Sub (http://redis.io/topics/pubsub) • Kestrel (http://robey.github.io/kestrel/) • ZeroMQ (http://zeromq.org/) • RabbitMQ (http://www.rabbitmq.com/) • Oracle GoldenGate (http://bit.ly/g-gate) • JMS compliant Server • Apache ActiveMQ (http://activemq.apache.org/) • Weblogic JMS (http://www.oracle.com/technetwork/middleware/weblogic/overview/index.html ) • IBM Websphere MQ (http://www-03.ibm.com/software/products/de/ibm-mq) • … April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 25
  • 26. 2015 © Trivadis Agenda 1. Designing Stream Processing Solutions 2. Implementing the Enterprise Event Bus (Unified Log) 3. Unified Log (Event) Processing Architecture in Action April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 26
  • 27. 2015 © Trivadis Unified Log Processing Architecture in Trivadis CRA April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 27 Tweets Filter and Unify Persist Tweet Filtered Tweets Split Text Words Count over Time Count by Minute Analyze Sentiment Tweet & Sentiment Remove Stopwords Tweet Tweets Consumer Twitter Filter Stream Sensor Layer Distribution Layer Speed Layer Kafka Storm Cassandra Elasticsearch
  • 28. 2015 © Trivadis Unified Log Processing Architecture in Trivadis CRA April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 28 Tweets Filter and Unify Persist Tweet Filtered Tweets Split Text Words Count over Time Count by Minute Analyze Sentiment Tweet & Sentiment Remove Stopwords Tweet Tweets Consumer Twitter Filter Stream Sensor Layer Distribution Layer Splitter Kafka Spout Word Remover Splitter Word Remover Shuffle Fields Kafka Kafka Word Remover Storm Topology Speed Layer Kafka Storm Cassandra Elasticsearch
  • 29. 2015 © Trivadis Weitere Informationen... April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 29
  • 30. 2015 © Trivadis BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURGI.BR. HAMBURG MÜNCHEN STUTTGART WIEN Fragen und Antworten... 2013 © Trivadis April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture