SlideShare a Scribd company logo

Kafka as an event store - is it good enough?

•
•
Guido Schmutz
Guido Schmutz

Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?

1 of 58
Download to read offline
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH http://guidoschmutz.wordpress.comgschmutz Kafka as an Event Store – is it Good Enough? Guido Schmutz W-JAX Munich – 6.11.2019
gschmutz Agenda 1. How do we build applications traditionally? 2. CQRS & Event Sourcing 3. What exactly is an Event Store? 4. Implementing Event Store 5. Summary
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 173rd edition
gschmutz How do we build applications traditionally?
gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model
gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Impedance Mismatch • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity
gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity SELECT acc.id, acc.account_number, acc.account_type.id, acct.name, cus.first_name, cus.last_name, addr.street, addr.city FROM account_t acc , account_type_t acct , customer_t cus , cust_adr_t cusa , address_t addr WHERE acc.account_type_id = acct.account_type_id AND adc.customer_id = cus.customer_id AND cusa.customer_id = cus.customer_id AND cusa.address_id = addr.address_id AND cusa.type = 'MAIN'
gschmutz Microservices - using Layered Architecture Microservices … • are responsible for their data • might use NoSQL instead of RDBMS • often still use traditional approach to persistence • “Data silos” do no longer support database join • keep synchronous communication to a minimum Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Account API Account Account Logic Product Microservice { } Product API Product Product Logic Finance App Finance UI UI Logic GUI REST REST REST sync request/response async, event pub/sub
gschmutz Events Distribute to all handlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result Three mechanisms through which services can interact Commands Route to single handler Use consistent hashing Provide Result Adapted from Axon IQ
gschmutz Domain Driven Design (DDD) – Concepts • Domain Objects – hold the state of the application • Entity – Domain Objects with an identity • Value Object – an immutable type that is distinguishable only by the state of its properties and has no identity • Aggregate - A cluster of domain objects that can be treated as a single unit • Aggregate Root – one object of aggregate is root object. Any reference from outside goes through aggregate root Aggregate Root Account Aggregate Customer Aggregate Aggregate Root
gschmutz Microservices with Event-driven communication This is Event Streaming and not really Event Sourcing Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic REST REST REST Event Hub (pub/sub) Customer Mat View sync request/response async, event pub/sub Finance App Finance UI UI Logic GUI
gschmutz CQRS & Event Sourcing
gschmutz Command Query Responsibility Segregation (CQRS) Optimize for write and read differently API is split between • commands - trigger changes in state • queries - provide read access to the state Still using CRUD pattern, but separates ”R” from CRUD Might involve eventual consistency between write and read model Data Storage Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic CDC command query project read insert update delete 1 2 3 4
gschmutz Local CQRS vs. System Wide CQRS Local CQRS System-Wide CQRS Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic CDC command query project read insert update delete Write Model Read Model (read-only) Service Command API App UI Projection Handler UI Logic CDC command project insert update delete Service Write Model command Command API insert update delete
gschmutz Event Sourcing – Persist state-changing events and not state # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000
gschmutz Event Sourcing persists the state of an aggregate as a sequence of state-changing events Each event describes a state change that occurred to the aggregate in the past new event is appended to the list of events an aggregate’s current state is reconstructed by replaying the events => a.k.a ”rehydration” Rehydration also needed for queries Event Store ServiceApp UI UI Logic Command API & Handler Event Handler(s) Service Subscribe publish publish apply (append) REST Data Storage trigger replycommand command 1 2 3 4 5 5
gschmutz Event Sourcing - ”Rehydrate” State 1. Create an empty Aggregate object 2. Read all events stored for that Aggregate from event store 3. Apply each event to the Aggregate object in the correct order AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 Account <empty> Account id: 123 accountType: Savings balance: 0 Account id: 123 accountType: Savings balance: 3000 transactions: [+1000, +2000] Account id: 123 accountType: Savings balance: 2500 transactions:[+1000, +2000, -500] Account id: 123 accountType: Savings balance: 1000 transactions: [+1000] applyTo applyTo applyTo applyTo
gschmutz Event Sourcing – Write Path CreateAccount command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) AccountCreated id: 123 accountType: Savings 10 # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} Account Aggregate <empty>
gschmutz Event Sourcing – Write Path DepositMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 10 20 Account Aggregate id: 123 accountType: Savings balance: 0
gschmutz Event Sourcing – Write Path DepositMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 10 20 100 Account Aggregate id: 123 accountType: Savings balance: 1000
gschmutz Event Sourcing – Write Path WithdrawMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000 Account Aggregate id: 123 accountType: Savings balance: 3000
gschmutz Event Sourcing - Potential Benefits 1. Subscribe to changes from other Aggregates 2. Examine a historical record of every change that has ever been applied on the model 3. Use the event store data for trend, forcast and other business analytics 4. Consider “what if” questions by replaying events to Aggregates which have experimental enhancements 5. Patch errors by adding ”correction” events (if it is legally allowed) 6. Perform “undo” and “redo” operations by replying varying sets of Events
gschmutz Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern Combines best of Event Sourcing and CQRS Project events published by Event Store into Read Model (Materialized Views) Write Model and Read Model might only support eventual consistency AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler command query read 1 Event Store publish apply (append) trigger reply 2 3 4 5 publish project 5 6
gschmutz Snapshot Optimization in Event Sequence # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} 5 2000 A32B3DE AccountSnapshot { id: 123, accountType: Savings, amount: 2500} 6 3000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} Snapshots for optimizing rehydration
gschmutz What is an Event Store?
gschmutz Event Store Capabilities 1. Append Events efficiently 2. Read aggregate’s events in order 3. Full Sequential Read (over all aggregates) 4. Consistent writes 5. Event versioning 6. Subscribable event stream 7. Correction events (O) 8. Ingestion & event time, bi- temporal (O) 9. Adhoc-Query on event store (O) 10.Snapshot Optimization (O) 11.High-Availability and Reliability (O)
gschmutz How many Event Stores do we need ? { } API State Logic REST Event Store { } API State Logic REST Event Store Microservice { } API State Logic REST Event Store Microservice { } API State Logic REST Microservice { } API State Logic REST Event Store Event Microservice { } API State Logic REST OR Microservice Microservice
gschmutz Implementing an Event Store
gschmutz Event Store Implementations • Event Store (https://eventstore.org/) – by Greg Young • Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ • Axon DB (https://axoniq.io/) - by Axon IQ • Eventuate (https://eventuate.io/) – by Eventuate.io • Serialized (https://serialized.io/) – by Serialized.io • Build your own …. • Apache Kafka ???
gschmutz Implementing an Event Store: using Kafka Broker
gschmutz Apache Kafka – A Streaming Platform Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
gschmutz No SPoF, highly available Consumer polls for new messages based on offset Apache Kafka – A Streaming Platform horizontally scalable, guaranteed order
gschmutz Kafka as an Event Store 1. One, single-partitioned Kafka topic per Aggregate 2. One, partitioned Kafka topic per Aggregate Type 3. One single, highly partitioned Kafka topic for all Aggregate Types Should you put several Event Types in the same Kafka topic?: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
gschmutz 1) One, single-partitioned Kafka topic per Aggregate Instance This will guarantee that the events are stored in order Reading state of an aggregate is as simple as reading a topic from offset 0 Not really feasible as there will be just too many topics needed Kafka Customer Aggregate Account Aggregate
gschmutz 2) One, partitioned Kafka topic per Aggregate Type • Required number of partitions is dependent on number of aggregate instances • Events are produced with aggregate-id as the key • guarantees that events are stored in order • For reading state of an aggregate, all data of all aggregate instances have to be scanned => slow • Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
gschmutz 3) One single, highly partitioned Kafka topic for all Aggregate Types • Required number of partitions is dependent on number of aggregate types * instances • Events are produced with aggregate-id as the key • guarantees that events are stored in order • For reading state of an aggregate, all data of all aggregate types & instances have to be scanned => really slow • Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
gschmutz Kafka as an Event Store # Capability Kafka Broker 1 Append events efficiently yes 2 Read aggregate’s events in order not efficiently 3 Full sequential Read yes 4 Consistent Writes no 5 Event Versioning yes (if Avro is used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header 9 Snapshot Optimization (O) no 10 Ad-Hoc Query on Events (O) no 11 High-Availability and Reliability (O) yes
gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run part of our own Event Store implementation add a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Other App Consumer
gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run our own Event Store implementation adding a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Other App Consumer
gschmutz Event StoreEvent Store Two solutions for avoiding «dual write» Write Event first then consume it to write it to database Write through database (CDC, outbox design pattern) Application { } API Database Biz Logic REST Event Hub Other App Biz Logic Application { } API Database REST Biz Logic CDC Event Hub CDC Connector Other App Biz Logic Publish
gschmutz Implementing an Event Store: using Axon Framework
gschmutz Axon • Spring Boot with Axon Framework for Application • MongoDB for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • Kafka Connect / Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Event Sourcing with Axon Account Events Account Command Account Aggregate Account Command Response Account App Event Store Account Customer Projection Command Handler Event Handler Account Query Projection Handler Query Handler Account Query Account Query Response Customer Event https://github.com/gschmutz/various-demos/tree/master/event-sourcing
gschmutz Event Sourcing with Axon - Aggregate @Aggregate public class AccountAggregate{ @AggregateIdentifier private String id; private BigDecimal balance; private String forCustomerId; private String accountType; @CommandHandler ... @EventSourcingHandler ...
gschmutz Event Sourcing with Axon - Command Handler @CommandHandler public AccountAggregate(AccountCreateCommand command) { Assert.hasLength(command.getForCustomerId(), "CustomerId must have a value"); Assert.hasLength(command.getAccountType(), "AccountType must have a value"); ... apply(new AccountCreatedEvent(command.getId(), command.getForCustomerId(), command.getAccountType(), new BigDecimal("0"))); }
gschmutz Event Sourcing with Axon – Command Handler @CommandHandler public void on(WithdrawMoneyCommand command) { Assert.isTrue(command.getAmount() > 0, "Amount should be a positive number"); if(command.getAmount().compareTo(this.balance) > 0 ) { throw new InsufficientBalanceException( "Insufficient balance. Trying to withdraw:" + command.getAmount() + ", but current balance is: " + this.balance); } apply(new MoneyWithdrawnEvent(command.getId(), command.getAmount())); }
gschmutz Event Sourcing with Axon – Event Handler @EventSourcingHandler public void handle(AccountCreatedEvent event) { id = event.getId(); forCustomerId = event.getForCustomerId(); accountType = event.getAccountType(); balance = event.getBalance(); } @EventSourcingHandler public void handle(MoneyWithdrawnEvent event) { balance = balance.subtract(event.getAmount()); }
gschmutz Event Sourcing with Axon – Projection Handler public class AccountQueryController { @Autowired private AccountRepository accRepo; @EventHandler public void on(AccountCreatedEvent event,@Timestamp Instant instant) { Account account = new Account(event.getId(),event.getBalance(), event.getAccHolder(),event.getAccHolderName(), instant.toString()); accRepo.insert(account); } @EventHandler public void on(MoneyDepositedEvent event,@Timestamp Instant instant) { Account account = accRepo.findByAccountNo(event.getId()); account.setBalance(account.getBalance().add(event.getAmount())); account.setLastUpdated(instant.toString()); accRepo.save(account); }
gschmutz Axon with Axon DB • Spring Boot with Axon Framework for Application • Axon DB for Event Store and Event Bus • Spring Boot for Projection Handler • Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Axon as an Event Store # Capability Axon Framework Axon Framework & Axon DB 1 Append events efficiently yes yes 2 Read aggregate’s events in order yes yes 3 Full sequential Read yes yes 4 Consistent Writes yes yes 5 Event Versioning yes yes 6 Subscribeable Event Stream yes yes 7 Correction events (O) no no 8 Event time & ingestion time, aka. Bi-temporal (O) no no 9 Snapshot Optimization (O) yes yes 10 Ad-Hoc Query on Events (O) yes yes 11 High-Availability and Reliability (O) possible yes
gschmutz Implementing an Event Store: using Kafka and Kafka Streams
gschmutz Apache Kafka – A Streaming Platform Source Connector Sink Connector trucking_ driver KSQL Engine Kafka Streams Kafka Broker
gschmutz Kafka & Kafka Streams • Kafka Streams with State for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • No reply of events, current snapshot is held in state store AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Account Event Handler Event Sourcing with Kafka Streams Account Created Money Deposited Money Withdrawn Command Account Command Handler Command Response Account API Account Snapshot Account Snapshot Customer Snapshot Account Customer API Account Customer Projection Account Customer Projector Account Customer Rec persist https://github.com/gschmutz/various-demos/tree/master/event-sourcing
gschmutz Kafka & Kafka Streams as an Event Store # Capability Kafka & Kafka Streams 1 Append events efficiently yes 2 Read aggregate’s events in order no (snapshot state only holds current snapshot) 3 Full sequential Read no 4 Consistent Writes yes (only one event per aggregate in flight) 5 Event Versioning yes (if Avro used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no 9 Snapshot Optimization (O) yes (snapshot state only) 10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …) 11 High-Availability and Reliability (O) yes
gschmutz Summary
gschmutz Summary • Event Sourcing and CQRS might be more natural to business people than IT => we are used to work with “CRUD based persistence” • Event Sourcing provides history and logging for free • Kafka Broker alone is really “just” Event Streaming, not Event Sourcing • Axon Framework supports the implementation of Event Sourcing applications with Pluggable Event Store and Event Bus implementations • Axon DB implements an Event Store and an Event Bus • Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming fashion” with current snapshot
Kafka as an event store - is it good enough?

More Related Content

Kafka as an event store - is it good enough?

  • 1. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH http://guidoschmutz.wordpress.comgschmutz Kafka as an Event Store – is it Good Enough? Guido Schmutz W-JAX Munich – 6.11.2019
  • 2. gschmutz Agenda 1. How do we build applications traditionally? 2. CQRS & Event Sourcing 3. What exactly is an Event Store? 4. Implementing Event Store 5. Summary
  • 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 173rd edition
  • 4. gschmutz How do we build applications traditionally?
  • 5. gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model
  • 6. gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Impedance Mismatch • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity
  • 7. gschmutz Data Access Layer Monolithic System - using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity SELECT acc.id, acc.account_number, acc.account_type.id, acct.name, cus.first_name, cus.last_name, addr.street, addr.city FROM account_t acc , account_type_t acct , customer_t cus , cust_adr_t cusa , address_t addr WHERE acc.account_type_id = acct.account_type_id AND adc.customer_id = cus.customer_id AND cusa.customer_id = cus.customer_id AND cusa.address_id = addr.address_id AND cusa.type = 'MAIN'
  • 8. gschmutz Microservices - using Layered Architecture Microservices … • are responsible for their data • might use NoSQL instead of RDBMS • often still use traditional approach to persistence • “Data silos” do no longer support database join • keep synchronous communication to a minimum Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Account API Account Account Logic Product Microservice { } Product API Product Product Logic Finance App Finance UI UI Logic GUI REST REST REST sync request/response async, event pub/sub
  • 9. gschmutz Events Distribute to all handlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result Three mechanisms through which services can interact Commands Route to single handler Use consistent hashing Provide Result Adapted from Axon IQ
  • 10. gschmutz Domain Driven Design (DDD) – Concepts • Domain Objects – hold the state of the application • Entity – Domain Objects with an identity • Value Object – an immutable type that is distinguishable only by the state of its properties and has no identity • Aggregate - A cluster of domain objects that can be treated as a single unit • Aggregate Root – one object of aggregate is root object. Any reference from outside goes through aggregate root Aggregate Root Account Aggregate Customer Aggregate Aggregate Root
  • 11. gschmutz Microservices with Event-driven communication This is Event Streaming and not really Event Sourcing Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic REST REST REST Event Hub (pub/sub) Customer Mat View sync request/response async, event pub/sub Finance App Finance UI UI Logic GUI
  • 13. gschmutz Command Query Responsibility Segregation (CQRS) Optimize for write and read differently API is split between • commands - trigger changes in state • queries - provide read access to the state Still using CRUD pattern, but separates ”R” from CRUD Might involve eventual consistency between write and read model Data Storage Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic CDC command query project read insert update delete 1 2 3 4
  • 14. gschmutz Local CQRS vs. System Wide CQRS Local CQRS System-Wide CQRS Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic CDC command query project read insert update delete Write Model Read Model (read-only) Service Command API App UI Projection Handler UI Logic CDC command project insert update delete Service Write Model command Command API insert update delete
  • 15. gschmutz Event Sourcing – Persist state-changing events and not state # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000
  • 16. gschmutz Event Sourcing persists the state of an aggregate as a sequence of state-changing events Each event describes a state change that occurred to the aggregate in the past new event is appended to the list of events an aggregate’s current state is reconstructed by replaying the events => a.k.a ”rehydration” Rehydration also needed for queries Event Store ServiceApp UI UI Logic Command API & Handler Event Handler(s) Service Subscribe publish publish apply (append) REST Data Storage trigger replycommand command 1 2 3 4 5 5
  • 17. gschmutz Event Sourcing - ”Rehydrate” State 1. Create an empty Aggregate object 2. Read all events stored for that Aggregate from event store 3. Apply each event to the Aggregate object in the correct order AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 Account <empty> Account id: 123 accountType: Savings balance: 0 Account id: 123 accountType: Savings balance: 3000 transactions: [+1000, +2000] Account id: 123 accountType: Savings balance: 2500 transactions:[+1000, +2000, -500] Account id: 123 accountType: Savings balance: 1000 transactions: [+1000] applyTo applyTo applyTo applyTo
  • 18. gschmutz Event Sourcing – Write Path CreateAccount command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) AccountCreated id: 123 accountType: Savings 10 # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} Account Aggregate <empty>
  • 19. gschmutz Event Sourcing – Write Path DepositMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 10 20 Account Aggregate id: 123 accountType: Savings balance: 0
  • 20. gschmutz Event Sourcing – Write Path DepositMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 10 20 100 Account Aggregate id: 123 accountType: Savings balance: 1000
  • 21. gschmutz Event Sourcing – Write Path WithdrawMoney command Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000 Account Aggregate id: 123 accountType: Savings balance: 3000
  • 22. gschmutz Event Sourcing - Potential Benefits 1. Subscribe to changes from other Aggregates 2. Examine a historical record of every change that has ever been applied on the model 3. Use the event store data for trend, forcast and other business analytics 4. Consider “what if” questions by replaying events to Aggregates which have experimental enhancements 5. Patch errors by adding ”correction” events (if it is legally allowed) 6. Perform “undo” and “redo” operations by replying varying sets of Events
  • 23. gschmutz Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern Combines best of Event Sourcing and CQRS Project events published by Event Store into Read Model (Materialized Views) Write Model and Read Model might only support eventual consistency AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler command query read 1 Event Store publish apply (append) trigger reply 2 3 4 5 publish project 5 6
  • 24. gschmutz Snapshot Optimization in Event Sequence # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} 5 2000 A32B3DE AccountSnapshot { id: 123, accountType: Savings, amount: 2500} 6 3000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} Snapshots for optimizing rehydration
  • 25. gschmutz What is an Event Store?
  • 26. gschmutz Event Store Capabilities 1. Append Events efficiently 2. Read aggregate’s events in order 3. Full Sequential Read (over all aggregates) 4. Consistent writes 5. Event versioning 6. Subscribable event stream 7. Correction events (O) 8. Ingestion & event time, bi- temporal (O) 9. Adhoc-Query on event store (O) 10.Snapshot Optimization (O) 11.High-Availability and Reliability (O)
  • 27. gschmutz How many Event Stores do we need ? { } API State Logic REST Event Store { } API State Logic REST Event Store Microservice { } API State Logic REST Event Store Microservice { } API State Logic REST Microservice { } API State Logic REST Event Store Event Microservice { } API State Logic REST OR Microservice Microservice
  • 29. gschmutz Event Store Implementations • Event Store (https://eventstore.org/) – by Greg Young • Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ • Axon DB (https://axoniq.io/) - by Axon IQ • Eventuate (https://eventuate.io/) – by Eventuate.io • Serialized (https://serialized.io/) – by Serialized.io • Build your own …. • Apache Kafka ???
  • 30. gschmutz Implementing an Event Store: using Kafka Broker
  • 31. gschmutz Apache Kafka – A Streaming Platform Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  • 32. gschmutz No SPoF, highly available Consumer polls for new messages based on offset Apache Kafka – A Streaming Platform horizontally scalable, guaranteed order
  • 33. gschmutz Kafka as an Event Store 1. One, single-partitioned Kafka topic per Aggregate 2. One, partitioned Kafka topic per Aggregate Type 3. One single, highly partitioned Kafka topic for all Aggregate Types Should you put several Event Types in the same Kafka topic?: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
  • 34. gschmutz 1) One, single-partitioned Kafka topic per Aggregate Instance This will guarantee that the events are stored in order Reading state of an aggregate is as simple as reading a topic from offset 0 Not really feasible as there will be just too many topics needed Kafka Customer Aggregate Account Aggregate
  • 35. gschmutz 2) One, partitioned Kafka topic per Aggregate Type • Required number of partitions is dependent on number of aggregate instances • Events are produced with aggregate-id as the key • guarantees that events are stored in order • For reading state of an aggregate, all data of all aggregate instances have to be scanned => slow • Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
  • 36. gschmutz 3) One single, highly partitioned Kafka topic for all Aggregate Types • Required number of partitions is dependent on number of aggregate types * instances • Events are produced with aggregate-id as the key • guarantees that events are stored in order • For reading state of an aggregate, all data of all aggregate types & instances have to be scanned => really slow • Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
  • 37. gschmutz Kafka as an Event Store # Capability Kafka Broker 1 Append events efficiently yes 2 Read aggregate’s events in order not efficiently 3 Full sequential Read yes 4 Consistent Writes no 5 Event Versioning yes (if Avro is used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header 9 Snapshot Optimization (O) no 10 Ad-Hoc Query on Events (O) no 11 High-Availability and Reliability (O) yes
  • 38. gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run part of our own Event Store implementation add a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Other App Consumer
  • 39. gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run our own Event Store implementation adding a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Other App Consumer
  • 40. gschmutz Event StoreEvent Store Two solutions for avoiding «dual write» Write Event first then consume it to write it to database Write through database (CDC, outbox design pattern) Application { } API Database Biz Logic REST Event Hub Other App Biz Logic Application { } API Database REST Biz Logic CDC Event Hub CDC Connector Other App Biz Logic Publish
  • 41. gschmutz Implementing an Event Store: using Axon Framework
  • 42. gschmutz Axon • Spring Boot with Axon Framework for Application • MongoDB for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • Kafka Connect / Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 43. gschmutz Event Sourcing with Axon Account Events Account Command Account Aggregate Account Command Response Account App Event Store Account Customer Projection Command Handler Event Handler Account Query Projection Handler Query Handler Account Query Account Query Response Customer Event https://github.com/gschmutz/various-demos/tree/master/event-sourcing
  • 44. gschmutz Event Sourcing with Axon - Aggregate @Aggregate public class AccountAggregate{ @AggregateIdentifier private String id; private BigDecimal balance; private String forCustomerId; private String accountType; @CommandHandler ... @EventSourcingHandler ...
  • 45. gschmutz Event Sourcing with Axon - Command Handler @CommandHandler public AccountAggregate(AccountCreateCommand command) { Assert.hasLength(command.getForCustomerId(), "CustomerId must have a value"); Assert.hasLength(command.getAccountType(), "AccountType must have a value"); ... apply(new AccountCreatedEvent(command.getId(), command.getForCustomerId(), command.getAccountType(), new BigDecimal("0"))); }
  • 46. gschmutz Event Sourcing with Axon – Command Handler @CommandHandler public void on(WithdrawMoneyCommand command) { Assert.isTrue(command.getAmount() > 0, "Amount should be a positive number"); if(command.getAmount().compareTo(this.balance) > 0 ) { throw new InsufficientBalanceException( "Insufficient balance. Trying to withdraw:" + command.getAmount() + ", but current balance is: " + this.balance); } apply(new MoneyWithdrawnEvent(command.getId(), command.getAmount())); }
  • 47. gschmutz Event Sourcing with Axon – Event Handler @EventSourcingHandler public void handle(AccountCreatedEvent event) { id = event.getId(); forCustomerId = event.getForCustomerId(); accountType = event.getAccountType(); balance = event.getBalance(); } @EventSourcingHandler public void handle(MoneyWithdrawnEvent event) { balance = balance.subtract(event.getAmount()); }
  • 48. gschmutz Event Sourcing with Axon – Projection Handler public class AccountQueryController { @Autowired private AccountRepository accRepo; @EventHandler public void on(AccountCreatedEvent event,@Timestamp Instant instant) { Account account = new Account(event.getId(),event.getBalance(), event.getAccHolder(),event.getAccHolderName(), instant.toString()); accRepo.insert(account); } @EventHandler public void on(MoneyDepositedEvent event,@Timestamp Instant instant) { Account account = accRepo.findByAccountNo(event.getId()); account.setBalance(account.getBalance().add(event.getAmount())); account.setLastUpdated(instant.toString()); accRepo.save(account); }
  • 49. gschmutz Axon with Axon DB • Spring Boot with Axon Framework for Application • Axon DB for Event Store and Event Bus • Spring Boot for Projection Handler • Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 50. gschmutz Axon as an Event Store # Capability Axon Framework Axon Framework & Axon DB 1 Append events efficiently yes yes 2 Read aggregate’s events in order yes yes 3 Full sequential Read yes yes 4 Consistent Writes yes yes 5 Event Versioning yes yes 6 Subscribeable Event Stream yes yes 7 Correction events (O) no no 8 Event time & ingestion time, aka. Bi-temporal (O) no no 9 Snapshot Optimization (O) yes yes 10 Ad-Hoc Query on Events (O) yes yes 11 High-Availability and Reliability (O) possible yes
  • 51. gschmutz Implementing an Event Store: using Kafka and Kafka Streams
  • 52. gschmutz Apache Kafka – A Streaming Platform Source Connector Sink Connector trucking_ driver KSQL Engine Kafka Streams Kafka Broker
  • 53. gschmutz Kafka & Kafka Streams • Kafka Streams with State for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • No reply of events, current snapshot is held in state store AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 54. gschmutz Account Event Handler Event Sourcing with Kafka Streams Account Created Money Deposited Money Withdrawn Command Account Command Handler Command Response Account API Account Snapshot Account Snapshot Customer Snapshot Account Customer API Account Customer Projection Account Customer Projector Account Customer Rec persist https://github.com/gschmutz/various-demos/tree/master/event-sourcing
  • 55. gschmutz Kafka & Kafka Streams as an Event Store # Capability Kafka & Kafka Streams 1 Append events efficiently yes 2 Read aggregate’s events in order no (snapshot state only holds current snapshot) 3 Full sequential Read no 4 Consistent Writes yes (only one event per aggregate in flight) 5 Event Versioning yes (if Avro used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no 9 Snapshot Optimization (O) yes (snapshot state only) 10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …) 11 High-Availability and Reliability (O) yes
  • 57. gschmutz Summary • Event Sourcing and CQRS might be more natural to business people than IT => we are used to work with “CRUD based persistence” • Event Sourcing provides history and logging for free • Kafka Broker alone is really “just” Event Streaming, not Event Sourcing • Axon Framework supports the implementation of Event Sourcing applications with Pluggable Event Store and Event Bus implementations • Axon DB implements an Event Store and an Event Bus • Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming fashion” with current snapshot