Policy-based Resource Placement
- 1. Policy-based Resource Placement ...across Hybrid-Cloud Federations of Kubernetes Clusters Irfan UR Rehman (Huawei) @irfanurrehman Torin Sandall (Styra) @sometorin
- 2. Federation: Overview federation-apiserver federation-controller app app Federation Control Plane Federated Deployment etcd Federated Clusters
- 3. Federation: Placement federation-apiserver federation-controller kind: ReplicaSet metadata: annotations: replica-set-preferences: | {“clusters”: { “us-west-1”: {“weight”:1}, “asia-se-2”: {“weight”:1} }} ● Placement can be controlled per-resource via annotations ● 2 annotations supported: federation.kubernetes.io/replica-set-preferences federation.alpha.kubernetes.io/cluster-selector ● federation-controller evaluates annotations to produce final placement ReplicaSet created 1 2 3 4
- 4. Policy-based Placement federation-apiserver federation-controller ● Resource placement is a “policy-rich” problem space ○ Legal regulation, cost, technical constraints, internal conventions, etc. ● Goal: give admins greater control and flexibility ○ Automated & programmable ○ Expressiveness ○ Leverage context ○ Ease management ● Decouple developer intent from admin policy ○ Avoid duplication ○ Prevent (and detect) violations ○ Abstract policy implementation ● Policy Engine decides which clusters app runs on ○ Pluggable ○ Simple interface opa
- 5. Architecture federation-apiserver federation-controller opaadmission controller kind: ReplicaSet metadata: name: my-app annotations: customer: acmecorp ● Admission Controller inside federation-apiserver queries Policy Engine when resources are created or updated ● Admission Controller implements “fail-closed” model in case query fails. POST /v1/data/k8s/placement input: resource: metadata: name: my-app annotations: customer: acmecorp ... HTTP/1.1 200 OK result: annotations: replica-set-preferences: clusters: us-west-1: {weight: 1} asia-se-3: {weight: .5} Example Query
- 6. Architecture host cluster pod federation-apiserver federation-controller opa admission controller etcd sidecar external state policy definition
- 7. Example Apps labelled with customer name. Customers are associated with a jurisdiction. Furthermore, apps may be labelled with criticality. If low then public cloud clusters may be used, otherwise, on-prem clusters must be used.
- 8. Visibility & Remediation federation-apiserveropa sidecar which of customer x’s apps are deployed in EU? which customers are affected if cluster X is no longer PCI certified? policy clusters, deployments, services, ...
- 9. Conflicts ● Developers could accidentally specify conflicting intent (result: empty set) ● Developers could explicitly request invalid clusters (result: error) ● Resolve conflicts within policy engine whenever possible ○ Policy is the only place where all intent is known kind: ReplicaSet metadata: annotations: customer: acmetel-US criticality: low replica-set-preferences: clusters: - us-west-2 - eu-central-1 ... not_allowed[cluster] { requested_clusters[cluster] not allowed_clusters[cluster] } errors[“invalid cluster(s)”] { not_allowed != set() }
- 10. Future Work ● Improve policy management ○ Current: policies stored as ConfigMaps in the federation-apiserver ○ Future: policies represented as first-class API objects ○ Cleaner mechanism for reporting policy enforcement status ■ Installed, errors, etc. ● Demonstrate new use cases ○ Cost-based policies ■ Replicate external data representing resource pricing (e.g., cpu, memory, etc.) ■ Pick clusters based on pricing data ■ Cluster inter-connect may be expensive federation-apiserver federation-controller opa
- 11. ● Kubernetes Federation enables hybrid-cloud deployments for a variety of use cases ● Resource placement is a policy-rich problem that must address important business requirements ● Policy solution should empower admins with greater control and flexibility Conclusion federation-apiserver federation-controller opa
- 12. Thank You! SIG-Federation Open Policy Agent (OPA) github.com/open-policy-agent/opa