More Related Content
What's hot
What's hot (20)
Similar to How we manage thousands of clusters with minimal effort
Similar to How we manage thousands of clusters with minimal effort (20)
More from LibbySchulze
More from LibbySchulze (20)
Recently uploaded
Recently uploaded (20)
How we manage thousands of clusters with minimal effort
- 1. !"#$%&%# !"#$#%$&'(')%$*+",-'(.-$ "/$01,-*%2-$#3*+$&3(3&'1$%//"2*$$ Hardik Dodiya, Samarth Deyagond 22/06/2021 Open Source and Community
- 2. Meet your speakers Hardik Dodiya Software Developer @ Gardener on Metal Samarth Deyagond Software Developer @ Gardener
- 3. Motivation OSS Project Gardener: Managed Kubernetes-as-a-Service at Scale • primarily runs in our own infra & also possible providers • is homogeneous in nature (same versions/features) • is fully automated/managed (zero manual ops) • is highly scalable, even beyond a single cluster • is customizable (offers all the configuration knobs)
- 4. Founder @ Botanist Quest (Hypothetical Application) Product Manager @ Botanist Quest Who are we? Botanist Quest Hardik & Samarth
- 5. Control plane Nodes Botanist Quest Botanist Quest Botanist Quest What do we have so far?
- 6. How we plan to host Botanist Quest
- 7. WAIT . . . q Are we ready to scale? q Multi cloud support? q Homogeneity & transparency? q Also employ our own infrastructure.
- 8. Well yeah . .
- 11. q Excessive under-utilized control-plane machines. q Operational complexity q Cluster management complexity HOLD ON
- 12. Kubeception . . . Deploy control-plane as a workload of other cluster ? ü Under utilized dedicated Control plane machines. ü Control plane & workload need not be bound to each other. ü Control plane components are full fledged applications
- 14. Control plane of cluster #1 in its own namespace Control plane of cluster #2 in its own namespace Management cluster
- 15. ü Excessive under-utilized control-plane machines. q Lifecycle of the control-planes. q Lifecycle of the underlying machines WAIT…
- 16. Cluster CRD Machine CRD Solve it the Kubernetes way …
- 17. Machine CRD creates machines for workload cluster Management cluster Nodes of the cluster where customers host their workload Cluster CRD Machine CRD Cluster Controller Manager
- 18. Let’s prototype it then…
- 19. Demo Time
- 20. ü Excessive under-utilized control plane machines ü Lifecycle management of the control-planes. ü Lifecycle of the underlying cluster machines q Cross-region latency q Number of clusters per management cluster HOLD ON
- 22. Well, let me think …
- 23. ü Excessive under-utilized control plane machines ü Lifecycle management of the control-planes. ü Lifecycle of the underlying cluster machines ü Cross-region latency q Management of the management clusters? HOLD ON
- 24. Phase 1 Cluster with dedicated control plane
- 25. Cluster Controller Manager Management cluster Cluster nodes decoupled with their control plane Cluster nodes decoupled with their control plane Cluster nodes decoupled with their control plane Cluster nodes decoupled with their control plane Cluster nodes decoupled with their control plane Phase 2 Deploying control planes as workload in the management cluster
- 26. Phase 3 Provisioning management clusters across different locations
- 27. Deploying control planes of management clusters as a workload on Super Management Cluster Management Cluster nodes hosting control Planes of clusters Super Management Cluster hosting control Planes of management clusters Cluster Controller Manager Management Cluster CRD
- 28. ü Excessive under-utilized control plane machines ü Lifecycle management of the control-planes. ü Lifecycle of the underlying cluster machines ü Cross-region latency ü Management of Management Clusters Awesome…
- 29. Can we now already manage thousands of clusters effortlessly?
- 30. Create cluster object •In super management cluster using cluster CRD Assign the management cluster •To host the control plane of the worker cluster Control plane gets deployed in management cluster •Cluster Controller Manager does the needfull. Cluster controller manager maintains the state •Cluster controller manager creates & manages the machines/nodes of the worker cluster The flow so far…
- 31. Samarth: FacewithRaisedEyebrow Hardik: ThinkingExpression Todo: White board Logo - Chalking - Discussing Cluster creation Pod creation
- 32. Kube- APIServer Kube- scheduler Kube- controller- manager Kubelet Kubelet Node Node pods déjà vu…
- 35. ü Excessive under-utilized control plane machines ü Lifecycle management of the control-planes. ü Lifecycle of the underlying cluster machines ü Cross-region latency ü Management of Management Clusters ü Full control & transparency for systems of record q Extensibility One other thing…
- 36. Batteries included but swappable… Cloud Providers Cloud Providers Gardener Container Runtimes Network plugins Operating Systems Golang Interface definition Interface Implementation AWS- extension- controller GCP- extension- controller XYZ- extension- controller
- 37. Seed hosting Control Planes of shoot Super Management Cluster hosting Control Planes of Seed Gardener Controller Manager Seed Cluster CRD
- 38. Demo Time
- 39. Horizontal Vertical Pod Autoscaler Etcd backup and restore Automatic Seed provisioning Scales control-plane vertically and horizontally Point in time recovery of ETCD with delta snapshots Automatically add more seeds with higher cluster demand Autoscaling of all the clusters ClusterAutoscaler autoscales seed, shoot and super- management clusters
- 40. Adopters
- 41. • External Contributions • - Find the approximate number of external contributions.
- 42. Relation with Cluster API . . . Kubeadm- controlplane-provider Gardener-controlplane- provider To be implemented …
- 43. TODOSamarthGIF: - Scrolling plenty of clusters in the dev landscape credits: https://twitter.com/timebertt
- 44. Join the Community Slack GitHub YouTube Twitter Blog News Community Adopters Documentation
- 45. v Scan to join the Community Samarth Deyagond https://github.com/AxiomSamarth Hardik Dodiya https://github.com/hardikdr