Replacing iptables with eBPF in Kubernetes with Cilium

USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.

In this session, we’ll review how previous efforts, including Netfilter, Berkley Packet Filter (BPF), Open vSwitch (OVS), and TC, approached the problem of extensibility. We’ll show you an open source solution available within the Red Hat Enterprise Linux kernel, where extending and merging some of the existing concepts leads to an extensible framework that satisfies the networking needs of datacenter and cloud virtualization.

SOSCON 2019.10.17 What are the methods for packet processing on Linux? And how fast are each packet processing methods? In this presentation, we will learn how to handle packets on Linux (User space, socket filter, netfilter, tc), and compare performance with analysis of where each packet processing is done in the network stack (hook point). Also, we will discuss packet processing using XDP, an in-kernel fast-path recently added to the Linux kernel. eXpress Data Path (XDP) is a high-performance programmable network data-path within the Linux kernel. The XDP is located at the lowest level of access through SW in the network stack, the point at which driver receives the packet. By using the eBPF infrastructure at this hook point, the network stack can be expanded without modifying the kernel. Daniel T. Lee (Hoyeon Lee) @danieltimlee Daniel T. Lee currently works as Software Engineer at Kosslab and contributing to Linux kernel BPF project. He has interest in cloud, Linux networking, and tracing technologies, and likes to analyze the kernel's internal using BPF technology.

This document provides an introduction to eBPF and XDP. It discusses the history of BPF and how it evolved into eBPF. Key aspects of eBPF covered include the instruction set, JIT compilation, verifier, helper functions, and maps. XDP is introduced as a way to program the data plane using eBPF programs attached early in the receive path. Example use cases and performance benchmarks for XDP are also mentioned.

This document discusses XDP (eXpress Data Path), a high-performance network data path that allows programs to run on the receive path of a network interface card. XDP enables packet processing using eBPF programs before packets reach the Linux networking stack. The document provides an overview of XDP and its performance advantages over other packet processing methods. It also discusses XDP's current status and support in the Linux kernel as well as example use cases and benchmarks.

- The document discusses Linux network stack monitoring and configuration. It begins with definitions of key concepts like RSS, RPS, RFS, LRO, GRO, DCA, XDP and BPF. - It then provides an overview of how the network stack works from the hardware interrupts and driver level up through routing, TCP/IP and to the socket level. - Monitoring tools like ethtool, ftrace and /proc/interrupts are described for viewing hardware statistics, software stack traces and interrupt information.

Cilium is an open source project which provides networking, security and load balancing for containers by using eBPF and XDP technologies in the Linux kernel. It provides eBPF and XDP features to CRI-O, Docker and Kubernetes. This presentation shows an overview on Cilium, explains the concepts behind it and then provide the project update, as it reached the 1.0 milestone last year. The video from talk at FOSDEM 2019: https://video.fosdem.org/2019/H.2214/cilium_overview_and_updates.webm

This document summarizes a presentation about Cilium and eBPF. Cilium provides cloud native networking and security using eBPF. eBPF allows programs to run securely in the Linux kernel for networking, security, and observability. Cilium offers networking features like Kubernetes services, cluster mesh for multi-cluster connectivity, and platform integration. It also provides security using identity-based policies and API authorization. Observability features include flow visibility and service maps. Cilium can be used as a service mesh or with Tetragon for prevention capabilities without proxies.

Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.

This talk demonstrates that programmability and performance does not require user space networking, it can be achieved in the kernel by generating BPF programs and leveraging the existing kernel subsystems. We will demo an early prototype which provides fast IPv6 & IPv4 connectivity to containers, container labels based security policy with avg cost O(1), and debugging and monitoring based on the per-cpu perf ring buffer. We encourage a lively discussion on the approach taken and next steps.

The document provides an overview of eBPF maps and how they can be used to share data between eBPF programs running in the kernel and userspace applications. It describes how maps are created via the BPF syscall using the BPF_MAP_CREATE command. It also explains how keys and values can be looked up, updated, and deleted from maps using commands like BPF_MAP_LOOKUP_ELEM, BPF_MAP_UPDATE_ELEM, and BPF_MAP_DELETE_ELEM. Finally, it lists the different types of eBPF maps available.

Container runtimes cause Linux to return to its original purpose: to serve applications interacting directly with the kernel. At the same time, the Linux kernel is traditionally difficult to change and its development process is full of myths. A new efficient in-kernel programming language called eBPF is changing this and allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself.

Cilium is open source software for transparently securing the network connectivity between application services deployed using Linux container management platforms like Docker and Kubernetes. At the foundation of Cilium is a new Linux kernel technology called BPF, which enables the dynamic insertion of powerful security visibility and control logic within Linux itself. Because BPF runs inside the Linux kernel itself, Cilium security policies can be applied and updated without any changes to the application code or container configuration.

The document compares eBPF, XDP and DPDK for packet inspection. It describes the speaker's experience using these tools to build a virtual machine that can handle 10Gbps of traffic and drop packets to mitigate DDoS attacks. It details how eBPF and XDP were able to achieve higher packet drop rates than iptables or a custom module. While DPDK could drop traffic at line rate, it required specialized hardware and expertise. Ultimately, XDP provided the best balance of performance, driver support and programmability using eBPF to drop millions of packets per second.

The document discusses how Cilium can accelerate Envoy and Istio by using eBPF/XDP to provide transparent acceleration of network traffic between Kubernetes pods and sidecars without any changes required to applications or Envoy. Cilium also provides features like service mesh datapath, network security policies, load balancing, and visibility/tracing capabilities. BPF/XDP in Cilium allows for transparent TCP/IP acceleration during the data phase of communications between pods and sidecars.

Netronome's half-day tutorial on host data plane acceleration at ACM SIGCOMM 2018 introduced attendees to models for host data plane acceleration and provided an in-depth understanding of SmartNIC deployment models at hyperscale cloud vendors and telecom service providers. Presenter Bios Jakub Kicinski is a long term Linux kernel contributor, who has been leading the kernel team at Netronome for the last two years. Jakub’s major contributions include the creation of BPF hardware offload mechanisms in the kernel and bpftool user space utility, as well as work on the Linux kernel side of OVS offload. David Beckett is a Software Engineer at Netronome with a strong technical background of computer networks including academic research with DDoS. David has expertise in the areas of Linux architecture and computer programming. David has a Masters Degree in Electrical, Electronic Engineering at Queen’s University Belfast and continues as a PhD student studying Emerging Application Layer DDoS threats.

CNI, the Container Network Interface, is a standard API between container runtimes and container network implementations. These slides are from the Cloud Native Computing Foundation's Webinar, and explain what CNI is, how you use it, and what lies ahead on the roadmap.

1. The document discusses OpenStack Neutron and Open vSwitch (OVS), describing their architecture and configuration. It explains that Neutron uses OVS to provide virtual networking and switching capabilities between virtual machines. 2. Key components of the Neutron-OVS architecture include the Neutron server, OVS agents on compute nodes, and the OVS daemon that implements the switch in the kernel and userspace. 3. The document also provides examples of configuring an OVS bridge and ports for virtual networking in OpenStack.

Extended BPF (eBPF) provides a mechanism for running custom programs inside the Linux kernel that can be used for filtering network packets, monitoring system activity, and more. eBPF programs are written in a restricted subset of C and compiled to bytecode that is verified by the kernel for safety before being run. The BCC toolkit makes it easier to write and load eBPF programs. The IO Visor project aims to further develop eBPF and provide tools and use cases for networking, security, and system tracing applications.

NDIV is a young, very simple, yet efficient network traffic diverter. Its purpose is to help build network applications that intercept packets at line rate with a very low processing overhead. A first example application is a stateless HTTP server reaching line rate on all packet sizes. Willy Tarreau, HaproxyTech

When it comes to networking inside Kubernetes, selecting the correct networking solution may be one of the most important decisions you may face. This is especially true if you are trying to run a Kubernetes cluster in production.  Therefore it's beneficial to have a good understanding of different CNI options out there and most importantly how these networking options are different from each other. This presentation goes over packet by packet-level details of how the network plumbing is happening with different CNI plugins including, Flannel, Calico & Cilium. 

Vector Packet Technologies such as DPDK and FD.io/VPP revolutionized software packet processing initially for discrete appliances and then for NFV use cases. Container based VNF deployments and it's supporting NFV infrastructure is now the new frontier in packet processing and has number of strong advocates among both traditional Comms Service Providers and in the Cloud. This presentation will give an overview of how DPDK and FD.io/VPP project are rising to meet the challenges of the Container dataplane. The discussion will provide an overview of the challenges, recent new features and what is coming soon in this exciting new area for the software dataplane, in both DPDK and FD.io/VPP! About the speaker: Ray Kinsella has been working on Linux and various other open source technologies for about twenty years. He is recently active in open source communities such as VPP and DPDK but is a constant lurker in many others. He is interested in the software dataplane and optimization, virtualization, operating system design and implementation, communications and networking.

This document provides an introduction to TCP/IP networking. It discusses the TCP/IP network architecture including the client-server model and layers. It also covers naming and addressing schemes, common protocols like TCP, UDP, IP, and Ethernet. Packet formats and programming interfaces are described. Finally, it discusses protocol analysis tools like Wireshark that can be used to observe network traffic.

Scaling Kubernetes to Support 50,000 Services The challenges of scaling Kubernetes include high API server load from managing a large number of services, pods and endpoints. Solutions tested include batch processing requests to reduce QPS, and restructuring iptables rules into a search tree for faster routing. IPVS was also tested and showed significantly better performance than iptables for large scale deployments, with constant latency for adding rules and higher network bandwidth. With these techniques, Kubernetes has been scaled to support over 50,000 services.

In this slide, we discuss the concept of IPTABLES/EBTABLES and then show how they work in a simple docker environment. In order to track the packet flow in those containers communication, we use the LOG module in IPTABLES/EBTABLE to track the information.

The document discusses using eBPF filters to optimize Netlink performance in SONiC by filtering unnecessary Netlink messages. It proposes: 1. Using eBPF/CBPF socket filtering to drop unwanted Netlink messages in the kernel before they are sent to applications. 2. Implementing filters using either eBPF assembly or Clang/LLVM for easier development and debugging. 3. Developing a customized eBPF library for SONiC with predefined filter rules and actions to simplify application-specific filtering.

Project Calico is an open-source networking project that provides layer 3 networking for scalable datacenter deployments using a more efficient implementation than traditional overlays. Calico is able to secure network interfaces on hosts using the same security policy model used for workloads. It supports building components like Calicoctl and Calico/node, and defines two types of endpoints - host endpoints for static interfaces and workload endpoints for dynamically managed interfaces. To run Calico and secure host interfaces, basic connectivity and policy is created, host endpoint objects are created in etcd for each interface, and additional security policies can be applied.

The Container Network Interface (CNI) is a simple specification for connecting containers to an arbitrary network. It promises interoperability between diverse networking technologies and container orchestration engines. Since its release two years ago, the CNI standard has grown in adoption. It is now a cross-industry effort, with contributors from CoreOS, RedHat, Google, Microsoft, and WeaveWorks, for example. CNI is used by the Kubernetes, CloudFoundry, and Mesos container orchestration engines. After a brief overview of the project, this talk will cover recent and coming developments in the CNI. As a specification, the CNI must balance the desire for new features with that of stability. I’ll cover the implications of that need for balance, design considerations, changes in the CNI spec, and the new use cases made possible.

The Container Network Interface (CNI) is a simple specification for connecting containers to an arbitrary network. It promises interoperability between diverse networking technologies and container orchestration engines. Since its release two years ago, the CNI standard has grown in adoption. It is now a cross-industry effort, with contributors from CoreOS, RedHat, Google, Microsoft, and WeaveWorks, for example. CNI is used by the Kubernetes, CloudFoundry, and Mesos container orchestration engines. After a brief overview of the project, this talk will cover recent and coming developments in the CNI. As a specification, the CNI must balance the desire for new features with that of stability. I’ll cover the implications of that need for balance, design considerations, changes in the CNI spec, and the new use cases made possible.

The presentation from Joseph Muli and Rajesh Dutta from Xebia on "How eBPF boost up Kubernetes service networking performance" - as presented on the 13th of April, 2023 at the Site Reliability Engineering NL MeetUp.

From the bottom-up approach to introduction the iptables, including the architecture of iptables/ebtables and the some usage of iptables.

We present a new open source project which provides IPv6 networking for Linux Containers by generating programs for each individual container on the fly and then runs them as JITed BPF code in the kernel. By generating and compiling the code, the program is reduced to the minimally required feature set and then heavily optimised by the compiler as parameters become plain variables. The upcoming addition of the Express Data Plane (XDP) to the kernel will make this approach even more efficient as the programs will get invoked directly from the network driver.

This document discusses Linux rumpkernel and the LKL (Linux Kernel Library). It introduces LKL as a library that allows running unmodified Linux kernel code in various configurations like application libraries and microkernels. LKL transforms a monolithic kernel code into a reusable library called liblkl by outsourcing machine-dependent code and keeping application and kernel code untouched. It provides different interfaces for applications to interact with the LKL kernel, including direct syscalls, hijacking the host library, or extending an alternative libc. Various usages of LKL are also presented, such as running a network stack in userspace (NUSE), building unikernels, and doing network simulation with ns-3 using the Linux network stack.

Container technologies use namespaces and cgroups to provide isolation between processes and limit resource usage. Docker builds on these technologies using a client-server model and additional features like images, containers, and volumes to package and run applications reliably and at scale. Kubernetes builds on Docker to provide a platform for automating deployment, scaling, and operations of containerized applications across clusters of hosts. It uses labels and pods to group related containers together and services to provide discovery and load balancing for pods.

Kubernetes networking allows pods to communicate with each other and services to load balance traffic to pods. The document discusses Kubernetes networking concepts including the network model, pod networking using CNI plugins like Flannel, and different service types such as ClusterIP, NodePort, and Ingress. It provides examples of exposing a Kubernetes service using hostNetwork, hostPort, and NodePort and how network policies are implemented using iptables.

The document describes how to configure a Linux machine as a router to connect two subnets. It provides instructions to enable IP forwarding and configure the network interfaces using temporary and permanent methods. The summary is: - Enable IP forwarding and configure the network interfaces of two Ethernet cards using ifconfig to set up routing temporarily - Use netconf to configure the interfaces and routing permanently by editing settings, accepting changes, and rebooting to confirm the configuration persists - Install traffic generator programs on end stations to test routing of UDP and TCP packets between subnets going through the router