Kafka on Kubernetes
•
0 likes•166 views
Victor Gamov from Confluent presented 'Streams must fFlow: Developing fault tolerant stream processing application with Kafka Streams and Kubernetes’ at Montreal's very first Cloud Native Day, which took place on June 11, 2019.
Report
Share
Kafka on Kubernetes
- 1. @gamussa | #ImCloudNative | @ConfluentINc Kafka on Kubernetes: Does it really have to be «The Hard Way»? June, 2019 / Montreal, CA, 2019 @gamussa | #ImCloudNative | @ConfluentINc
- 2. @gamussa | #ImCloudNative | @ConfluentINc 2
- 3. @gamussa | #ImCloudNative | @ConfluentINc 3 I build highly scalable Hello World apps
- 4. Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸 🖼 👬 Tag @gamussa With #ImCloudNative
- 5. @gamussa | #ImCloudNative | @ConfluentINc 5 A company is build on DATA FLOWS but All we have is DATA STORES
- 6. @gamussa | #ImCloudNative | @ConfluentINc 6 Pre-Streaming
- 7. @gamussa | #ImCloudNative | @ConfluentINc 7
- 8. @gamussa | #ImCloudNative | @ConfluentINc 8 New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time
- 9. @gamussa | #ImCloudNative | @ConfluentINc 9 Evolution of #devkafkaops Shell scripts ansible/chef Docker Kubernetes
- 10. @gamussa | #ImCloudNative | @ConfluentINc 10 https://twitter.com/claesmogren/status/1108456688175079424
- 11. @gamussa | #ImCloudNative | @ConfluentINc 11 🙋
- 12. @gamussa | #ImCloudNative | @ConfluentINc 12 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? 🙋
- 13. @gamussa | #ImCloudNative | @ConfluentINc 13 kafkaesque world of Kafka on Kubernetes
- 14. @gamussa | #ImCloudNative | @ConfluentINc 14 Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades #devkafkaops
- 15. @gamussa | #ImCloudNative | @ConfluentINc 15 But I just want to deploy kafka
- 16. @gamussa | #ImCloudNative | @ConfluentINc 16 Workloads Deployment
- 17. @gamussa | #ImCloudNative | @ConfluentINc 17 Helm Charts
- 18. @gamussa | #ImCloudNative | @ConfluentINc 18 Helm Charts
- 19. @gamussa | #ImCloudNative | @ConfluentINc 19 Helm Charts
- 20. @gamussa | #ImCloudNative | @ConfluentINc 20 https://cnfl.io/helm_video
- 21. @gamussa | #ImCloudNative | @ConfluentINc 21 Helm Charts is just a GO Templates. How Charts help with rolling update?
- 22. @gamussa | #ImCloudNative | @ConfluentINc 22 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
- 23. @gamussa | #ImCloudNative | @ConfluentINc 23 We will use StatefulSets with OrderedReady
- 24. @gamussa | #ImCloudNative | @ConfluentINc 24 We need SRE / Operator knowledge to manage the platform. You need Operator!
- 25. @gamussa | #ImCloudNative | @ConfluentINc 25
- 26. @gamussa | #ImCloudNative | @ConfluentINc 26 Show me your Operator
- 27. @gamussa | #ImCloudNative | @ConfluentINc 27 Rider Driver 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes KLyfft - Kafka app on Kubernetes
- 28. @gamussa | #ImCloudNative | @ConfluentINc 28 Demo
- 29. @gamussa | #ImCloudNative | @ConfluentINc 29 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND
- 30. @gamussa | #ImCloudNative | @ConfluentINc 30 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
- 31. @gamussa | #ImCloudNative | @ConfluentINc 31 Be like Justin!
- 32. @gamussa | #ImCloudNative | @ConfluentINc 32 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
- 33. @gamussa | #ImCloudNative | @ConfluentINc 33 Want to see advanced deployment? vs.
- 34. @gamussa | #ImCloudNative | @ConfluentINc 34 http://gamov.dev/ksummit_nyc19_k8s
- 35. @gamussa | #ImCloudNative | @ConfluentINc 35 GA Plans● We are in private Preview Release is over! ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon
- 36. @@gamussa | #ImCloudNative | @ConfluentINc Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes
- 37. 37