Designing a reactive real-time data platform: Architecture and Infrastructure Challenges
•
1 like•585 views
Alex covers the key architectural decisions and challenges encountered during the design and implementation of a reactive data platform that is able to fulfill several business use cases, ranging from real-time data ingestion to analysis, at massive scales. Pluralsight's current implementation consists of discrete microservices running on top of the JVM, using a mixture of Scala, Akka, Spark, and Kafka. Alex shares Pluralsight's event-driven, reactive design that leverages REST, Hypermedia, and several open source frameworks and platforms, including Spring, Hadoop, YARN, Kafka, Storm, and Spark. Topics include: - The major architectural components of the platform - Leveraging Kafka's distributed commit log for message persistence and event sourcing. - Integrating real-time and batch data pipelines using Spark - Designing event-driven protocols in Scala using Akka - Asynchronous remote communication between services and actors - Binary message serialization protocols - Data-integration challenges
Report
Share
Designing a reactive real-time data platform: Architecture and Infrastructure Challenges
- 1. ARCHITECTURE AND INFRASTRUCTURE http://linkedin.com/in/alexvsilva @thealexsilva DESIGNING A REACTIVE REAL-TIME DATA PLATFORM
- 2. Who am I? - DATA Platform Architect at Pluralsight - Rackspace - WDW
- 3. TECHNOLOGY LEARNING PLATFORM What shou ld Ilearn?Where Sho uld IStart?Who can help me?Whatdid I learn? • Online technology learning platform • Subscription model • Data-driven PLURALSIGHT
- 5. Development TIME became the bottleneck
- 7. HOW DO WE FIX THAT?
- 11. RESPONSIVE
- 12. ELASTIC
- 13. asynchronous share nothing location transparency divide and conquer
- 14. RESILENT MORE THAN “JUST” FAULT TOLERANCE
- 21. Software systems are complex systems.
- 22. “Complex systems run in degraded mode.” “Complex systems run as broken systems.” Richard Cook
- 25. MESSAGE DRIVEN
- 26. asynchronous FAILURES AS MESSAGES location transparency ISOLATION
- 27. Messages and events SAVE THIS! SOMEBODY LOGGED IN! FactsTopic Events ARE… Past AddressableSpecific Messages ARE…
- 28. Data platform at pluralsight
- 30. HYDRA INGEST Ingestion + Replication PORTAL Schema Manager “The Log” HYDRA STREAMS Streaming + Replication
- 32. AKKA
- 34. Akka challenges Remoting Type Safety Debugging Release Cycles
- 35. WHAT’s AN ACTOR? behavior state MAILBOX CHILD ACTORS SUPERVISOR STRATEGY
- 36. ACTOR Refs are reactive!
- 37. ACTOR Refs Vs. ACTOR PATHS
- 38. akka.tcp://Hydra@localhost:9001/user/service/ingestor_registry Protocol Address ActorSystem Path Actor paths enable location transparency akka.tcp://Hydra/user/service/ingestor_registry Protocol ActorSystem Path
- 41. akka { actor { deployment { /services-manager/handler_registry/segment_handler { router = round-robin-pool optimal-size-exploring-resizer { enabled = on action-interval = 5s downsize-after-underutilized-for = 2h } } /services-manager/kafka_producer { router = round-robin-pool resizer { lower-bound = 5 upper-bound = 50 messages-per-resize = 500 } } } } }
- 42. Sending messages on akka VS
- 43. Hydra ingest Data capture at scale
- 44. mitigate message loss DATA format is secondary automated replication schema driven
- 45. ENFORCE METADATA AT INGESTION TIME
- 46. DATA PIPELINES DATA QUALITY DATA REPLICATIONDATA DISCOVERY
- 47. Metadata is a first class citizen
- 48. Why avro? Schema evolution Smaller data footprint Json friendly Strong community support Existing tools
- 49. INGESTION PROTOCOL OVERVIEW BRINGING REACTIVE PRINCIPLES TO THE MIX
- 50. HYDRA REQUEST PAYLOAD { "name":"John", "age":30, "cars":[ "Ford", "BMW" ] } kafka-topic = PersonTopic validation = Strict avro-schema = Person.avsc METADATA + = HYDRA REQUEST Anything, really
- 51. HYDRA REPLICATION PROTOCOL HYDRA REQUEST INGESTORS Publish Akka Actors (remote) Transport Transports Akka Actors (remote) Kafka Postgres Elastic Search Inspect metadata and decide
- 53. WHY DIFFERENT PHASES? Divide and conquer isolation Small asynchronous tasks recovery
- 56. HYDRA Message delivery guarantees ALSO METADATA DRIVEN AT-LEAST-ONCE SEMANTICS AKKA PERSISTENT ACTOR hydra-delivery-strategy
- 57. kafka
- 58. A Messaging system based on distributed log semantics Scalable Fault tolerant Stateful Strong ordering High concurrency
- 59. BROKER BROKER BROKER(User, 0) Topic: User (User, 0) (User, 0) READS/WRITES FROM/TO Leader only REPLICATION PROTOCOL
- 60. Replication is about RESILIENCY BROKER BROKER BROKER BROKER
- 61. Looks like A GLOBALLY ORDERED QUEUE BROKER APPLICATION APPLICATION CONSUMER APPLICATION
- 62. THE LOG is a linear structure Old New Messages are added here
- 63. Consumers have a position Only sequential access Read to offset and SCAN Old New Consumer 1 Consumer 2
- 64. MESSAGES CAN BE REPLAYED FOR AS LONG AS THEY EXIST IN THE LOG Old New Consumer 1 Consumer 2
- 65. A DISTRIBUTED REPLICATION PROTOCOL Rewind and Replay
- 66. Hydra STREAMS STREAMING FEDERATION LAYER
- 68. Continuously updating datasets Max(viewed_time) from clip_views where location=‘CA’ over 1 day window
- 69. Similar features as a database JOINAGREGGATE FILTER VIEW
- 73. Why spark? Support for many different data formats Structured streaming Failover and lifecycle management Medium latency Unified api
- 74. EVENT STREAM / LOG MATERIALIZEDVIEWS/CACHE HADOOP ETL SERVICE TRANSF Writes to Replicates to • Reproducible • Stays in sync
- 75. Why kafka streams? application that can run anywhere Medium data volumes Kafka specific Low latency Basic tasks
- 76. IT WORKS FOR MICROSERVICES TOO HYDRA Sends BROKER stores (at a minimum) INGESTION Customer HYDRA STREAM DISPATCH { } /dsls submits POSTs Invoices Returns joins/normalizes streams
- 77. Hydra SPARK
- 78. What is it? Abstraction layer on top of SPARK datasets Models data flows Sources and operations Based on a custom dsl Api-driven
- 80. WE ARE ON GITHUB! github.com/pluralsight/hydra github.com/pluralsight/hydra-spark
- 82. Thank You!