Designing a Scalable Data Platform
•
1 like•596 views
With the advent of reliable streaming technologies, real-time data pipelines have become a crucial component of any robust data initiative today. Compared to a traditional Hadoop-centric data hub, these real-time stacks provide high-levels of system availability and data integrity coupled with very low latency queries without incurring the overhead of inflexible schemas and batch analysis lag. Alex Silva demonstrates how to use Kafka, Spark Streaming, Akka, and Hadoop to orchestrate a real-time stack and explains how data flows through this system. This real-time data platform combines a mix of open source technologies and home-grown services aimed at providing a full end-to-end solution, starting from flexible data-ingestion protocols to fast data analysis and queries. Topics include: External message providers, which connect to the platform through a data-ingestion service modeled as a robust actor system using Akka and Scala Routing different backend systems, including Kafka and Druid Spark Streaming, which is used to perform real-time complex analytical and scientific processing on the data Exporting data for future processing into Hadoop Querying and visualization Photo of Alex Silva
Report
Share
Designing a Scalable Data Platform
- 1. DESIGNING A SCALABLE DATA PLATFORM USING AKKA, SPARK STREAMING AND KAFKA Alex Silva
- 8. BUT HOW DO WE START?
- 9. REAL-TIME VIDEO DATA STREAMING AND ANALYSIS
- 13. REACTIVE
- 14. The Reactive Manifesto Responsive Elastic Resilient Message Driven
- 16. Why Akka? Reactive Scalable Fault Tolerant Load Management Both up and out Location Transparency
- 17. Challenges with Akka Learning Curve Type Safety Debugging Dead Letters
- 19. Producers vs Consumers Producer Producer Kafka Cluster Broker 2 Topic 1 Partition 1 Broker 1 Topic 1 Partition 0 Broker 3 Topic 1 Partition 3 Producer Producer Producer
- 20. Kafka: A CP or CA System? CONSISTENCY AVAILABILITY PARTITION TOLERANCE INCONSISTENCY UNAVAILABILITY PARTITIONING INTOLERANNCE
- 22. MODULES
- 24. HYDRA INGESTION Actor Hierarchy Supervision Kafka Gateway Message Protocol
- 25. MESSAGE HANDLERS
- 26. Handler Registry Monitors registered handlers for errors/stops Brodcasts messages Handler Lifecycle
- 27. < META > { } /ingest Coordinator Registry Handlers Ingestion Flow
- 28. Ingestion Actors: Coordinator Supervises ingestion at the request level Coordinates protocol flow Reports errors and metrics
- 30. HEY GUYS! CHECK THIS OUT! HUH?! NICE!! BRING IT!! NAH… Publish JoinJoin Ingestion Protocol: Publish Handler Registry Message handlers
- 31. Ingestion Protocol: Validation HOW DOES IT LOOK? Validate BAD! Invalid GOOD! Valid Ingestion Coordinator Message handlers
- 32. Ingestion Protocol: Invalid Message Ingestion Coordinator Error Reporter GOT A BAD ONE ReportError Ingest
- 34. abstract class BaseMessageHandler extends Actor with ActorConfigSupport with ActorLogging with IngestionFlow with ProducerSupport with MessageHandler { ingest { case Initialize => { //nothing required by default } case Publish(request) => { log.info(s"Publish message was not handled by ${self}. Will not join.") } case Validate(request) => { sender ! Validated } case Ingest(request) => { log.warning("Ingest message was not handled by ${self}.") sender ! HandlerCompleted } case Shutdown => { //nothing required by default } case Heartbeat => { Health.get(self).getChecks } } }
- 35. Elasticity
- 37. akka { actor { deployment { /services-manager/handler_registry/segment_handler { router = round-robin-pool optimal-size-exploring-resizer { enabled = on action-interval = 5s downsize-after-underutilized-for = 2h } } /services-manager/kafka_producer { router = round-robin-pool resizer { lower-bound = 5 upper-bound = 50 messages-per-resize = 500 } } } } }
- 38. akka { actor { deployment { /services-manager/handler_registry/segment_handler { router = round-robin-pool optimal-size-exploring-resizer { enabled = on action-interval = 5s downsize-after-underutilized-for = 2h } } } provider = "akka.cluster.ClusterRefActorProvider" } cluster { seed-nodes = ["akka.tcp://Hydra@127.0.0.1:2552","akka.tcp://hydra@172.0.0.1:2553"] } }
- 39. Message Driven
- 41. SAVE THIS! SOMEBODY LOGGED IN! Events are not addressed to a specific recipient. Message vs. Events
- 43. Message Serialization Happens at the message-level Binds Messages to Producers Pluggable
- 44. trait KafkaMessage[K, P] { val timestamp = System.currentTimeMillis def key: K def payload: P def retryOnFailure: Boolean = true } case class JsonMessage(key: String, payload: JsonNode) extends KafkaMessage[String, JsonNode] object JsonMessage { val mapper = new ObjectMapper() def apply(key: String, json: String) = { val payload: JsonNode = mapper.readTree(json) new JsonMessage(key, payload) } } case class AvroMessage(val schema: SchemaHolder, key: String, json: String) extends KafkaMessage[String, GenericRecord] { def payload: GenericRecord = { val converter: JsonConverter[GenericRecord] = new JsonConverter[GenericRecord](schema.schema) converter.convert(json) } }
- 46. Fault Tolerance
- 47. “The ability of something to return to its original shape, after it has been pulled, stretched, pressed, or bent.” Merriam-Webster Resiliency
- 53. Resilient Protocols Message Loss Message Reordering Message Duplication
- 57. override val supervisorStrategy = OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) { case _: ActorInitializationException => akka.actor.SupervisorStrategy.Stop case _: FailedToSendMessageException => Restart case _: ProducerClosedException => Restart case _: NoBrokersForPartitionException => Escalate case _: KafkaException => Escalate case _: ConnectException => Escalate case _: Exception => Escalate } val kafkaProducerSupervisor = BackoffSupervisor.props( Backoff.onFailure( kafkaProducerProps, childName = actorName[KafkaProducerActor], minBackoff = 3.seconds, maxBackoff = 30.seconds, randomFactor = 0.2 ))
- 58. class KafkaProducerActor extends Actor with LoggingAdapter with ActorConfigSupport with NotificationSupport[KafkaMessage[Any, Any]] { import KafkaProducerActor._ implicit val ec = context.dispatcher override def preRestart(cause: Throwable, message: Option[Any]) = { //send it to itself again after the exponential delays, no Ack from Kafka message match { case Some(rp: RetryingProduce) => { notifyObservers(KafkaMessageNotDelivered(rp.msg)) val nextBackOff = rp.backOff.nextBackOff val retry = RetryingProduce(rp.topic, rp.msg) retry.backOff = nextBackOff context.system.scheduler.scheduleOnce(nextBackOff.waitTime, self, retry) } case Some(produce: Produce) => { notifyObservers(KafkaMessageNotDelivered(produce.msg)) if (produce.msg.retryOnFailure) { context.system.scheduler.scheduleOnce(initialDelay, self, RetryingProduce(produce.topic, produce.msg)) } } } } }
- 59. The Error Kernel Pattern Error Kernel Per Request No Processing Delegation Ingestion Errors Time outs
- 63. Location Transparency: Akka Remoting Peer-to-Peer Serialization Delivery Reliability Latency
- 64. Message Delivery in Hydra What does guaranteed delivery mean? At most once semantics Can be made stronger
- 65. Reliable Proxy
- 66. @throws(classOf[Exception]) override def init: Future[Boolean] = Future { val useProxy = config.getBoolean(“message.proxy”,false) val ingestorPath = config.getRequiredString("ingestor.path") ingestionActor = if (useProxy) context.actorSelection(ingestorPath) else context.actorOf(ReliableIngestionProxy.props(ingestorPath)) val cHeaders = config.getOptionalList("headers") topic = config.getRequiredString("kafka.topic") headers = cHeaders match { case Some(ch) => List( ch.unwrapped.asScala.map { header => { val sh = header.toString.split(":") RawHeader(sh(0), sh(1)) } }: _* ) case None => List.empty[HttpHeader] } true }
- 67. Conductors Webhooks What’s streaming into Hydra today?
- 68. What’s streaming into Hydra in the next few days? Internal ETL Many others…
- 69. HYDRA SPARK Kafka Dispatching Spark Job Server Streaming Analytics
- 70. 1 2 3 4 5 6 7 8 9 10 … 90 91 9 2 9 3 9 4 9 5 9 6 9 7 9 8 9 9 10 0 Time 321 Offsets Dispatching
- 71. 1 2 3 4 5 6 7 8 9 10 … 90 91 9 2 9 3 9 4 9 5 9 6 9 7 9 8 9 9 10 0 1 2 3 Time Offsets Dispatching
- 72. ImpalaHive Dispatchers HDFS RDBMS Kafka Cassandra
- 73. Dispatching Adhoc Tables { } /dispatch
- 74. WHAT ABOUT ORDERING? Post-Dispatch Ordering Single Partition Topic
- 75. Spark Defined via hydra configuration and bootstrapped Both batch and streaming contexts Ad-hoc via job server
- 76. Spark Job Server Spark as a REST Service Spark SQL, Streaming, etc. Async and Sync Job APIs Jar Management Context Management
- 79. Job Manager Endpoint Configuration Job repository Tracking Lifecycle management GET /jobs?limit=N - Lists the last N jobs POST /jobs - Starts a new job; ‘sync=true’ to wait GET /jobs/<jobId> - Gets the result or status of a job DELETE /jobs/<jobId> - Kills the job GET /jobs/<jobId>/config - Gets the job configuration
- 80. Creating Spark Jobs #Ad-hoc jobs through hydra - Run-once jobs with transient curl --data-binary @/etc/local/hydra/video-segment-fx.jar localhost:9091/jars/segment curl -d "kafka.topic=segment" 'localhost:9091/jobs?appName=segment&classPath=hydra.SegmentJob&sync=false' { "status": "STARTED", "result": { "jobId": "3156120b-f001-56cf-d22a-b40ebf0a9af1", "context": "f5ed0ec1-hydra.spark.analytics.segment.SegmentJob" } }
- 81. Persistent Context Jobs #Required for related jobs #Create a new context curl -X POST 'localhost:9091/contexts/video-032116-ctx?num-cpu-cores=10&memory-per- node=512m' OK #Verify creation curl localhost:9091/contexts ["video-032116-ctx"] #Run job using the context curl -d "kafka.topic=segment" 'localhost:9091/jobs? appName=segment&classPath=hydra.SegmentJob&sync=true&context=video-032116-ctx' { "result":{ "active-sessions":24476221 } }
- 82. Dispatcher Jobs curl -X POST localhost:9091/dispatch { "dispatch":{ "kafka.topic":"youbora", "offsets":{ "start":"2016-03-15T01:18:01", "end":"2016-03-15T03:18:01" }, "avro.schema":"http://172.16.10.220:8085/schemas/youbora", "hdfs-serializer":{ "output.path":"/tmp/hydra/youbora/${c:offsets.start}" } } } localhost
- 83. Dispatcher Jobs curl -X POST localhost:9091/dispatch { "dispatch":{ "kafka.topic":"youbora", "offsets":{ "start":"2016-03-15T01:18:01", "end":"2016-03-15T03:18:01" }, "avro.schema":"http://172.16.10.220:8085/schemas/youbora", "hdfs-serializer":{ "url":"jdbc:postgresql://localhost/test", "driver":"org.postgresql.Driver", "username":"${c:dispatchers.yb_pg_username}", "password":"${c:dispatchers.yb_pg_password}", "sql":"insert into video_raw values ('${total}','${type}','${subtype}','${user_id}','${ip}','$ {country}','${city}','${start}')" } } } localhost
- 84. Job Status #Job Results/Status curl localhost:9091/jobs/3156120b-f001-56cf-d22a-b40ebf0a9af1 { "duration":"16.264 secs", "classPath":"hydra.spark.analytics.segment.SegmentJob", "startTime":"2016-03-26T01:18:01.256Z", "context":"f5ed0ec1-hydra.spark.analytics.segment.SegmentJob", "result":{ "active-sessions":24476221 }, "status":"FINISHED", "jobId":"3156120b-f001-56cf-d22a-b40ebf0a9af1" }
- 85. HOW IT LOOKS LIKE IN THE WILD
- 86. Ingestion Hydra CoreIngestors HTTP Spark (Batch and Streaming) Hydra CoreDispatchers HTTP RDBMS HDFS Conductors Hydra CoreConductors HTTP Persistence :: Kafka Hydra CorePersistence HTTP AKKA Remoting 3 2 2
- 87. AWS Model vCPU Mem (GiB) SSD Storage (GB) m3.medium 1 3.75 1 x 4 m3.large 2 7.5 1 x 32 m3.xlarge 4 15 2 x 40 m3.2xlarge 8 30 2 x 80
- 88. 0 500 1000 1500 2000 2500 Dec-15 Jan-16 Jan-16 Jan-16 1-Feb 3/1/16 Average Ingestions Per Second Requests
- 89. 9,730 lines of Scala code Production Platform Since Jan 2016 C.I. through Jenkins and Salt Some Facts
- 90. What’s next?
- 91. Remote Handlers Ingestion Hydra CoreIngestors HTTP Handler <<REMOTE>> RegisterHandle Handler Registry Ingestion Protocol Create ZNode
- 95. Location Transparency in Hydra Distributed by Default Akka Remoting Configuration-Driven Peer-to-Peer