Kubernetes and OpenStack at Scale

Editor's Notes

1. Goals were: Push system to it’s limit, incl. Ensuring we can reproduce work done in the community with kubernetes upstream incl. SIG Scalability (will come to this in a minute) Identify config changes and best practices to increase capacity and performance Document and file issues upstream and send patches where applicable
2. Saturation test for OpenShift’s HAProxy-based network ingress tier Overlay2 graph driver and SELinux support from kernel v4.9 Persistent volume scalability and performance using Red Hat’ Container-Native Storage (CNS) product (gluster based) Saturation test for OpenShift’s integrated container registry and CI/CD pipeline
3. Primary metrics include: Max cores per cluster Max pods per core Management overhead per node Management overhead per cluster Derived metrics include: Max cores per node Max pods per machine Max machines per cluster Max pods per cluster End-to-end pod startup time Scheduler throughput Max cluster saturation time Pre-pulled images because high degree of variability introduced (network throughput, size of image, etc.) between images that are unrelated to k8s performance.
4. Why IaaS and PaaS Exposition versus Consumption Current state (VMs) versus future state (BM) Culture/people challenges (developer versus operations, who is driving) Isolation concerns Scaling concerns OpenStack Open source cloud computing platform for building massively scalable clouds. Kubernetes Open source system for automating deployment, scaling and management of containerized applications. Provides framework for building distributed platforms. Kubernetes container management/orchestration Red Hat is the biggest contributor outside of Google How did Red Hat end up on the Kubernetes horse? We bet on a simple idea: that an open source community is the best place to build the future of application orchestration, and that only an open source community could successfully integrate the diverse range of capabilities necessary to succeed.
5. OpenShift An integrated infrastructure platform to run, orchestrate, monitor and scale containers. Built around Kubernetes and Docker. OpenShift application platform Acquired Makara in Nov 2010 OpenShift Origin launched in Apr 2012 Docker Open Source Mar 2013 First Kubernetes commit on github Jun 2014 OpenShift v3 re-architected around Docker and Kubernetes Jun 2015 building on operational experiences obtained by OpenShift Online team with v2. LDK!
6. Sandwhich: Your applications OpenShift masters, nodes, registry Infrastructure services (LBaaS, Neutron, Nova, Cinder, etc.) Architectural tenets: Technical independence: Ensure that containers are defined such that they remain independent of the underlying infrastructure. Containers must continue to be portable across host environments. Contextual awareness: Allow containers to easily take advantage of OpenStack shared services beyond compute (i.e. networking and storage). To do this, Red Hat Atomic Enterprise (and other Red Hat container offerings) must be context aware. Avoid redundancy: Limit redundancies where possible to minimize performance and other resource hits. This includes limiting the number of layers between the container and the hardware. Simplified management: Simplify management by delivering a holistic, integrated view across platforms. Currently contextual awareness comes via the cloud provider implementation (all or nothing) Expect to see increased experimentation with using services piecemeal/a la carte (e.g. Cinder) Storage: Container hosts consume OpenStack storage Tenant isolation Application storage managed by Kubernetes Stateful applications Containerized distributed storage services Networking: Use OpenShift-SDN to have full application isolation but get double encapsulation when using Neutron with GRE or VXLAN tunnels. Tenant isolation via OpenStack SDN using Kuryr eventually Use Flannel with host-gw backend to avoid double encapsulation. Load Balancing provided by LBaaS V1 by default. Other options: External load balancer (recommended for production) Dedicated load balancer node - create a dedicated node for HAProxy. Good for demo/test but no HA. None - if using single master node. Authenticate OpenShift users using LDAP.
7. Re-validate Kubernetes SIG Scalability findings on equivalent OpenShift Container Platform release.
8. The CNCF cluster is made up of 1000 nodes deployed at Switch, Las Vegas by Intel for the use of the CNCF community. We were using ~300 NVMe storage will come in handy later
9. Not product supported application_performance: JMeter-based performance testing of applications hosted on OpenShift. applications_scalability: Performance and scalability testing of the OpenShift web UI. conformance: Wrappers to run a subset of e2e/conformance tests in an SVT environment (work in progress) image_provisioner: Ansible playbooks for building AMI and qcow2 images with OpenShift rpms and Docker images baked in. networking: Performance tests for the OpenShift SDN and kube-proxy. openshift_performance: Performance tests for container build parallelism, projects and persistent storage (EBS, Ceph, Gluster and NFS) openshift_scalability: Home of the infamous "cluster-loader", details in openshift_scalability/README.md reliability: Run tests over long periods of time (weeks), cycle object quantity up and down.
10. Why both? For the foreseeable future we envisage there will be baremetal, virtualized, containerized workloads Current state is most people we see are running containers in VMs. Cultural/people issues: Easiest way to get going without rocking the organization wide IT boat in some cases Concerns about potential for breakout (contrast to QEMU and use of similar constructs there) Scale issues: # of pods per node (currently 250 and rising), workload dependent. Availability: Ability to live migrate VMs, not impossible to live migrate a container but also not really the way things should work long term.
11. The Overcloud usually consists of nodes in predefined roles such as Controller nodes, Compute nodes, and different storage node types. Each of these default roles contains a set of services defined in the core Heat template collection on the director node. However, the architecture of the core Heat templates provides a method to: Create custom roles Add and remove services from each role Storage Layout Each storage node includes 2 SSDs and 10 SAS disks. Passed NVMe to VMs for Container Native Storage (Gluster) Ceph performs significantly better when deployed with write-journals on SSDs. Created two write-journals on the SSDs and allocated 5 of the spinning disks to each SSD. In all, we had 90 Ceph OSDs, equating to 158 TB of available disk space. Image Upload Converted to RAW for upload to glance Use snapshot/boot-from-volume flow Consumed ~ 700MB per VM VM pool in Ceph this time around ~1.5 TB for 2048 VMs versus 22 TB last time for 1,000 VMs. Reduced I/O and time to boot VMs, < 15 mins for the 2048 VMs. Ceph’s role in this environment is to provide boot-from-volume service for our VMs (via Cinder).
12. Routing tier consists of nodes running HAProxy for ingress into the cluster. Identified that there are (on average), a large number of low-throughput cluster ingress connections from clients (i.e. web browsers) to HAProxy versus a small number of high-throughput connections. Already some changes in this space based on previous iterations: Default connection limit of 2000 leaves plenty of room on commonly available CPU cores for additional connections. Thus, bumped the default connection limit to 20,000 in OpenShift 3.5 out of the box. If you have other needs to customize the configuration for HAProxy, our networking folks have made it significantly easier — as of OpenShift 3.4, the router pod now uses a configmap, making tweaks to the config that much simpler.
13. Load generator configured via passing in ConfigMaps Queries Kubernetes API for list of routes. Builds list of test targets dynamically Zoomed in on a particularly representative workload mix Combination of HTTP with keepalive and TLS terminated at the edge. Chose this because it represents how most OpenShift production deployments are used - serving large numbers of web applications for internal and external use, with a range of security postures.
14. Graph shows throughput test with a Y-axis of Requests Per Second, higher is better. nbproc refers to number of HAProxy processes spawned. Sched_migration_cost is a kernel tunable that weights processes when deciding if/how the kernel should load balance them amongst available cores. What we learned: CPU affinity matters. But why are certain cores nearly 2x faster? This is because HAProxy is now hitting the CPU cache more often due to NUMA/PCI locality with the network adapter. Increasing nbproc helps throughput. nbproc=2 is ~2x faster than nbproc=1, BUT we get no more boost from going to 4 cores, and in fact nbproc=4 is slower than nbproc=2. This is because there were 4 cores in this guest, and 4 busy HAProxy threads left no room for the OS to do its thing (like process interrupts). Can improve performance over 20% from baseline with no changes other than sched_migration_cost. By increasing it by a factor of 10, we keep HAProxy on the CPU longer, and increase our likelihood of CPU cache hits by doing so. This is a common technique amongst the low-latency networking crowd, and is in fact recommended tuning in our Low Latency Performance Tuning Guide for RHEL7.
15. Provides full multi-tenancy Encapsulation comes with tradeoffs in CPU cycles to wrap/unwrap packets Can be mitigated via VXLAN offloading with commonly available NICs incl. Those in CNCF cluster. Pluggable, so like OpenStack you can use other SDN solutions where integration has been done Also expect to use Kuryr in future Allows it to be used on any public/private footprint incl. OpenStack
16. RFC2544 - Benchmarking Methodology for Network Interconnect Devices Discusses and defines a number of tests that may be used to describe the performance characteristics of a network interconnecting device. Also describes specific formats for reporting the results of the tests. As you would expect, adding more streams for same payload provides a notable increase. Difference between baremetal/baremetal+pod and vm/vm+pod only becomes pronounced at largest payload size. Bonus tuning: Large clusters with over 1000 routes or nodes require increasing the default kernel arp cache size. We’ve increased it by a factor of 8x, and are including that tuning out of the box in OpenShift 3.5.
17. Reasons: Maturity Supportability Security POSIX compliance Overlay/Overlay2 Density improvements gained by page cache sharing are very important for certain environments where there is significant overlap in base image content. Overlay2 w/ SELinux in Linux kernel 4.9
18. Rate limited pod creation using “tuningset” w/ cluster-loader Each of the 6 bumps is a batch of 40 pods. Before it moves to the next batch, cluster-loader makes sure the previous batch is in running state. In this way avoid crushing the API server with requests, and can examine the system’s profiles at each plateau. The savings in terms of memory is reasonable (again, this is a “perfect world” scenario and your mileage may vary).
19. The reduction in disk operations below is due to subsequent container starts leveraging the kernel’s page cache rather than having to repeatedly fetch base image content from storage: Overall found overlay2 to be very stable, and it becomes even more interesting with the addition of SELinux support.
20. Deployed in pods, scheduled like any application Used Kubernetes dynamic provisioning to expose volumes to applications. Marked unschedulable to control variability.
21. Roughly 6 seconds from submit to the PVC going into “Bound” state. This number does not vary when CNS is deployed on bare metal or virtualized. Not pictured here are our tests verifying that several other persistent volume providers respond in a very similar timeframe.
22. Filed bugs across Kubernetes, Docker, OpenStack, Ceph, Kernel, OpenvSwitch, Golang, Gluster