Developing Event-driven Microservices with Event Sourcing & CQRS (gotoams)
•
19 likes•4,598 views
The document discusses how to develop event-driven microservices using event sourcing and Command Query Responsibility Segregation (CQRS). It describes event sourcing, where events are stored instead of the current application state. Services update their own state by subscribing to events. CQRS is used to separate read models from write models so that queries can be optimized with materialized views of aggregated data. Implementing queries in an event-sourced application involves using separate query-side microservices to update and retrieve materialized views based on domain events from the event store.
Report
Share
Developing Event-driven Microservices with Event Sourcing & CQRS (gotoams)
- 1. @crichardson Developing event-driven microservices with event sourcing and CQRS Chris Richardson Author of POJOs in Action Founder of the original CloudFoundry.com @crichardson chris@chrisrichardson.net http://plainoldobjects.com http://microservices.io
- 2. @crichardson Presentation goal Show how Event Sourcing and Command Query Responsibility Segregation (CQRS) are a great way to implement microservices
- 4. @crichardson About Chris Founder of a startup that’s creating a platform for developing event-driven microservices: http://eventuate.io/ Consultant helping organizations improve how they architect and deploy applications using cloud, micro services, polyglot applications, NoSQL, ... Creator of http://microservices.io
- 6. @crichardson Agenda Why build event-driven microservices? Overview of event sourcing Designing microservices with event sourcing Implementing queries in an event sourced application
- 7. @crichardson Tomcat Traditional monolithic architecture Browser/ Client WAR/EAR RDBMS Customers Accounts Transfers Banking UI develop test deploy Simple Load balancer scale Spring MVC Spring Hibernate ... HTML REST/JSON ACID
- 8. @crichardson But large and/or complex monolithic applications = Trouble!
- 9. @crichardson Apply the scale cube X axis - horizontal duplication Z axis -data partitioning Y axis - functional decomposition Scale by splitting sim ilar things Scale by splitting different things
- 10. @crichardson Today: use a microservice, polyglot architecture Banking UI Account Management Service MoneyTransfer Management Service Account Database MoneyTransfer Database Standalone services Sharded SQLNoSQL DB
- 11. @crichardson But this results in distributed data management problems
- 12. @crichardson Example: Money transfer Account Management Service MoneyTransfer Management Service Account Database MoneyTransfer Database Account #1 Money Transfer Account #2 No ACID No 2PC
- 13. @crichardson Use an event-driven Services publish events when state changes Services subscribe to events and update their state Maintain eventual consistency across multiple aggregates (in multiple datastores) Synchronize replicated data
- 14. @crichardson MoneyTransferService MoneyTransfer fromAccountId = 101 toAccountId = 202 amount = 55 state = INITIAL MoneyTransfer fromAccountId = 101 toAccountId = 202 amount = 55 state = DEBITED MoneyTransfer fromAccountId = 101 toAccountId = 202 amount = 55 state = COMPLETED Eventually consistent money transfer Message Bus AccountService transferMoney() Publishes: Subscribes to: Subscribes to: publishes: MoneyTransferCreatedEvent AccountDebitedEvent DebitRecordedEvent AccountCreditedEvent MoneyTransferCreatedEvent DebitRecordedEvent AccountDebitedEvent AccountCreditedEvent Account id = 101 balance = 250 Account id = 202 balance = 125 Account id = 101 balance = 195 Account id = 202 balance = 180
- 16. @crichardson Update and publish using 2PC Guaranteed atomicity BUT Need a distributed transaction manager Database and message broker must support 2PC Impacts reliability Not fashionable 2PC is best avoided
- 17. @crichardson Use data store as message queue Use datastore as a message queue Txn #1: Update database: new entity state & event Txn #2: Consume event Txn #3: Mark event as consumed Eventually consistent mechanism (used by eBay) See BASE: An Acid Alternative, http://bit.ly/ebaybase BUT Tangled business logic and event publishing code Difficult to implement when using a NoSQL database :-(
- 18. @crichardson Agenda Why build event-driven microservices? Overview of event sourcing Designing microservices with event sourcing Implementing queries in an event sourced application
- 19. @crichardson Event sourcing For each aggregate in your domain model: Identify (state-changing) domain events Define Event classes For example, Account: AccountOpenedEvent, AccountDebitedEvent, AccountCreditedEvent ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent
- 20. @crichardson Persists events NOT current state Account balance open(initial) debit(amount) credit(amount) AccountOpened Event table AccountCredited AccountDebited 101 450 Account table X 101 101 101 901 902 903 500 250 300
- 21. @crichardson Replay events to recreate state Account balance AccountOpenedEvent(balance) AccountDebitedEvent(amount) AccountCreditedEvent(amount) Events
- 22. @crichardson Before: update state + publish events Two actions that must be atomic Single action that can be done atomically Now: persist (and publish) events
- 23. @crichardson Request handling in an event-sourced application HTTP Handler Event Store pastEvents = findEvents(entityId) Account new() applyEvents(pastEvents) newEvents = processCmd(SomeCmd) saveEvents(newEvents) Microservice A
- 24. @crichardson Event Store publishes events - consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Aggregate NoSQL materialized view update() update() Microservice B
- 25. @crichardson Event store implementations Home-grown/DIY geteventstore.com by Greg Young My event store - http://bit.ly/trialeventuate
- 26. @crichardson Optimizing using snapshots Most aggregates have relatively few events BUT consider a 10-year old Account many transactions Therefore, use snapshots: Periodically save snapshot of aggregate state Typically serialize a memento of the aggregate Load latest snapshot + subsequent events
- 27. @crichardson Hybrid OO/Functional style example aggregate
- 28. @crichardson OO = State + Behavior balance Account processCommand(cmd : Command) : Seq[Events] applyEvent(event : Event) : Account State Behavior
- 29. @crichardson Aggregate traits Map Command to Events Apply event returning updated Aggregate Used by Event Store to reconstitute aggregate
- 30. @crichardson Account - command processing Prevent overdraft
- 31. @crichardson Account - applying events Immutable
- 34. @crichardson FP = Separation of State and Behavior Account balance AccountAggregate processCommand(Account, Command) : Seq[Events] applyEvent(Account, Event) : Account State Behavior
- 37. @crichardson FP-style event store Enables inference of T, and EV Tells ES how to instantiate aggregate and apply events
- 38. @crichardson Business benefits of event sourcing Built-in, reliable audit log Enables temporal queries Publishes events needed by big data/predictive analytics etc. Preserved history More easily implement future requirements
- 39. @crichardson Technical benefits of event sourcing Solves data consistency issues in a Microservice/NoSQL- based architecture: Atomically save and publish events Event subscribers update other aggregates ensuring eventual consistency Event subscribers update materialized views in SQL and NoSQL databases (more on that later) Eliminates O/R mapping problem
- 40. @crichardson Drawbacks of event sourcing Weird and unfamiliar Events = a historical record of your bad design decisions Handling duplicate events can be tricky Application must handle eventually consistent data Event store only directly supports PK-based lookup (more on that later)
- 41. @crichardson Agenda Why build event-driven microservices? Overview of event sourcing Designing microservices with event sourcing Implementing queries in an event sourced application
- 42. @crichardson Use the familiar building blocks of DDD Entity Value object Services Repositories Aggregates With some differences
- 43. @crichardson Partition a bounded context’s domain model into Aggregates
- 44. Aggregate design Graph consisting of a root entity and one or more other entities and value objects Each core business entity = Aggregate: e.g. customer, Account, Order, Product, …. Reference other aggregate roots via primary key Often contains partial copy of other aggregates’ data Order OrderLine Item quantity productId productName productPrice customerId Address street city …
- 45. @crichardson Aggregate granularity is important Transaction = processing one command by one aggregate No opportunity to update multiple aggregates within a transaction If an update must be atomic (i.e. no compensating transaction) then it must be handled by a single aggregate e.g. scanning boarding pass at security checkpoint or when entering jetway
- 47. @crichardson Identify the state changing events for each Aggregate
- 48. @crichardson Designing domain events Naming Past tense to reflect that something occurred Ideally specific: AccountOpened/Debited/Credited Sometimes vague: FooUpdated Event attributes Id - TimeUUID Other attributes - from command, required to persist entity Event enrichment ProductAddedToCart(productId) vs. ProductAddedCart(productInfo) Extra data to support event consumers
- 49. @crichardson The anatomy of a microservice Event Store HTTP Request HTTP Adapter Event Adapter Cmd Cmd Events Events Xyz Adapter Xyz Request microservice Aggregate
- 50. @crichardson Asynchronous Spring MVC controller Scala Future => Spring MVC DeferredResult
- 51. @crichardson MoneyTransferService DSL concisely specifies: 1.Creates MoneyTransfer aggregate 2.Processes command 3.Applies events 4.Persists events
- 52. @crichardson Handling events published by Accounts 1.Load MoneyTransfer aggregate 2.Processes command 3.Applies events 4.Persists events
- 53. @crichardson Agenda Why build event-driven microservices? Overview of event sourcing Designing microservices with event sourcing Implementing queries in an event sourced application
- 54. @crichardson Let’s imagine that you want to display an account and it’s recent transactions...
- 55. @crichardson Displaying balance + recent credits and debits We need to do a “join: between the Account and the corresponding MoneyTransfers (Assuming Debit/Credit events don’t include other account, ...) BUT Event Store = primary key lookup of individual aggregates, ... Use Command Query Responsibility Segregation
- 56. @crichardson Command Query Responsibility Segregation (CQRS) Command-side Commands Aggregate Event Store Events Query-side Queries (Denormalized) View Events
- 57. @crichardson Query-side microservices Event Store Updater - microservice View Updater Service Events Reader - microservice HTTP GET Request View Query Service View Store e.g. MongoDB Neo4J CloudSearch update query
- 58. @crichardson Persisting account balance and recent transactions in MongoDB { id: "298993498", balance: 100000, transfers : [ {"transferId" : "4552840948484", "fromAccountId" : 298993498, "toAccountId" : 3483948934, "amount" : 5000}, ... ], changes: [ {"changeId" : "93843948934", "transferId" : "4552840948484", "transactionType" : "AccountDebited", "amount" : 5000}, ... ] } Denormalized = efficient lookup MoneyTransfers that update the account The debits and credits Current balance
- 59. @crichardson Persisting account info using MongoDB... class AccountInfoUpdateService (accountInfoRepository : AccountInfoRepository, mongoTemplate : MongoTemplate) extends CompoundEventHandler { @EventHandlerMethod def created(de: DispatchedEvent[AccountOpenedEvent]) = … @EventHandlerMethod def recordDebit(de: DispatchedEvent[AccountDebitedEvent]) = … @EventHandlerMethod def recordCredit(de: DispatchedEvent[AccountCreditedEvent]) = … @EventHandlerMethod def recordTransfer(de: DispatchedEvent[MoneyTransferCreatedEvent]) = … }
- 60. @crichardson Persisting account info using NodeJS and MongoDB... this.handlers[accountEvents.entityTypeName][accountEvents.AccountOpenedEvent] = function (event, callback){ accountViewUpdaterService.createAccount(event, callback) }; this.handlers[accountEvents.entityTypeName][accountEvents.AccountDebitedEvent] = function (event, callback) { accountViewUpdaterService.saveAccountChange(event, -1, callback); }; exports.saveAccountChange = function(event, delta, callback){ … var update = { $inc: { balance: amount * delta }, $push: { changes: ci }, $set: { version: changeId } }; var options = { multi: true }; db.AccountModel.update(conditions, update, options, callback); };
- 61. Other kinds of views AWS Cloud Search Text search as-a-Service View updater batches aggregates to index View query service does text search AWS DynamoDB NoSQL as-a-Service On-demand scalable - specify desired read/write capacity Document and key-value data models Useful for denormalized, UI oriented views
- 62. Benefits and drawbacks of CQRS Benefits Necessary in an event-sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
- 63. @crichardson Summary Event sourcing solves key data consistency issues with: Microservices Partitioned SQL/NoSQL databases Apply strategic DDD to identify microservices Apply tactical DDD to design individual services Use CQRS to implement materialized views for queries