Embedded Recipes 2018 - Finding sources of Latency In your system - Steven Rostedt

The webinar discussed accelerating P4 and eBPF programs on Netronome SmartNIC hardware. It covered the Linux kernel infrastructure like TC and XDP that supports offloading eBPF programs. It also explained how the NFP architecture is optimized for network flow processing with its multi-core design and memory hierarchy. The webinar demonstrated how eBPF programs can be translated to run efficiently on the NFP hardware by handling maps and applying optimizations.

The document provides an overview of the initialization process in the Linux kernel from start_kernel to rest_init. It lists the functions called during this process organized by category including functions for initialization of multiprocessor support (SMP), memory management (MM), scheduling, timers, interrupts, and architecture specific setup. The setup_arch section focuses on x86 architecture specific initialization functions such as reserving memory regions, parsing boot parameters, initializing memory mapping and MTRRs.

The document discusses Linux networking architecture and covers several key topics in 3 paragraphs or less: It first describes the basic structure and layers of the Linux networking stack including the network device interface, network layer protocols like IP, transport layer, and sockets. It then discusses how network packets are managed in Linux through the use of socket buffers and associated functions. The document also provides an overview of the data link layer and protocols like Ethernet, PPP, and how they are implemented in Linux.

This document summarizes a presentation on static partitioning virtualization for RISC-V. It discusses the motivation for embedded virtualization, an overview of static partitioning hypervisors like Jailhouse and Xen, and the Bao hypervisor. It then provides an overview of the RISC-V hypervisor specification and extensions, including implemented features. It evaluates the performance overhead and interrupt latency of a prototype RISC-V hypervisor implementation with and without interference mitigations like cache partitioning.

Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.

This document discusses using Linux perf_events (perf) profiling tools to analyze Java performance on Linux. It describes how perf can provide complete visibility into Java, JVM, GC and system code but that Java profilers have limitations. It presents the solution of using perf to collect mixed-mode flame graphs that include Java method names and symbols. It also discusses fixing issues with broken Java stacks and missing symbols on x86 architectures in perf profiles.

The arrival of flash storage introduced a radical change in performance profiles of direct attached devices. At the time, it was obvious that Linux I/O stack needed to be redesigned in order to support devices capable of millions of IOPs, and with extremely low latency. In this talk we revisit the changes the Linux block layer in the last decade or so, that made it what it is today - a performant, scalable, robust and NUMA-aware subsystem. In addition, we cover the new NVMe over Fabrics support in Linux. Sagi Grimberg Sagi is Principal Architect and co-founder at LightBits Labs.

Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).

Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg. There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes. This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.

Talk for USENIX LISA 2016 by Brendan Gregg. "Linux 4.x Tracing Tools: Using BPF Superpowers The Linux 4.x series heralds a new era of Linux performance analysis, with the long-awaited integration of a programmable tracer: Enhanced BPF (eBPF). Formally the Berkeley Packet Filter, BPF has been enhanced in Linux to provide system tracing capabilities, and integrates with dynamic tracing (kprobes and uprobes) and static tracing (tracepoints and USDT). This has allowed dozens of new observability tools to be developed so far: for example, measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more. These lead to performance wins large and small, especially when instrumenting areas that previously had zero visibility. Tracing superpowers have finally arrived. In this talk I'll show you how to use BPF in the Linux 4.x series, and I'll summarize the different tools and front ends available, with a focus on iovisor bcc. bcc is an open source project to provide a Python front end for BPF, and comes with dozens of new observability tools (many of which I developed). These tools include new BPF versions of old classics, and many new tools, including: execsnoop, opensnoop, funccount, trace, biosnoop, bitesize, ext4slower, ext4dist, tcpconnect, tcpretrans, runqlat, offcputime, offwaketime, and many more. I'll also summarize use cases and some long-standing issues that can now be solved, and how we are using these capabilities at Netflix."

This talk discusses Linux profiling using perf_events (also called "perf") based on Netflix's use of it. It covers how to use perf to get CPU profiling working and overcome common issues. The speaker will give a tour of perf_events features and show how Netflix uses it to analyze performance across their massive Amazon EC2 Linux cloud. They rely on tools like perf for customer satisfaction, cost optimization, and developing open source tools like NetflixOSS. Key aspects covered include why profiling is needed, a crash course on perf, CPU profiling workflows, and common "gotchas" to address like missing stacks, symbols, or profiling certain languages and events.

USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.

The motivation of hypervisor based CPUFreq is to enable the one of the main PM use-cases (Dynamic voltage and frequency scaling) in virtualized system powered by Xen hypervisor. Rationale behind this activity is that CPU virtualization is done by hypervisor and the guest OS doesn't actually know anything about physical CPUs because it is running on virtual CPUs. In this talk Oleksandr will briefly describe the possible approach of generic CPUFreq in Xen on ARM, the advantages and disadvantages of having DVFS support on ARM boards powered by Xen hypervisor and share results of his CPUFreq PoC which implies power consumption measurements with and without CPUFreq enabled on R-Car Gen3 based board as an example.

Video: https://www.youtube.com/watch?v=JRFNIKUROPE . Talk for linux.conf.au 2017 (LCA2017) by Brendan Gregg, about Linux enhanced BPF (eBPF). Abstract: A world of new capabilities is emerging for the Linux 4.x series, thanks to enhancements that have been included in Linux for to Berkeley Packet Filter (BPF): an in-kernel virtual machine that can execute user space-defined programs. It is finding uses for security auditing and enforcement, enhancing networking (including eXpress Data Path), and performance observability and troubleshooting. Many new open source tools that have been written in the past 12 months for performance analysis that use BPF. Tracing superpowers have finally arrived for Linux! For its use with tracing, BPF provides the programmable capabilities to the existing tracing frameworks: kprobes, uprobes, and tracepoints. In particular, BPF allows timestamps to be recorded and compared from custom events, allowing latency to be studied in many new places: kernel and application internals. It also allows data to be efficiently summarized in-kernel, including as histograms. This has allowed dozens of new observability tools to be developed so far, including measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more. This talk will summarize BPF capabilities and use cases so far, and then focus on its use to enhance Linux tracing, especially with the open source bcc collection. bcc includes BPF versions of old classics, and many new tools, including execsnoop, opensnoop, funcccount, ext4slower, and more (many of which I developed). Perhaps you'd like to develop new tools, or use the existing tools to find performance wins large and small, especially when instrumenting areas that previously had zero visibility. I'll also summarize how we intend to use these new capabilities to enhance systems analysis at Netflix.

This document provides an overview of the data structures and functions used to implement ethernet drivers in the Linux kernel. It discusses the net_device and sk_buff structures that represent network interfaces and packets. It also describes how the driver interacts with the kernel via polling, interrupts, and NAPI to handle reception and transmission of frames. Finally, it provides an example of the key components needed for a simple ethernet driver, including initialization, setup, open/close, transmission, and reception functions.

Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).

The document describes a biolatency tool that traces block device I/O latency using eBPF. It discusses how the tool was originally written in the bcc framework using C/BPF, but has since been rewritten in the bpftrace framework using a simpler one-liner script. It provides examples of the bcc and bpftrace implementations of biolatency.

Trying to figure out why your application is responding late can be difficult, especially if it is because of interference from the operating system. This talk will briefly go over how to write a C program that can analyze what in the Linux system is interfering with your application. It will use trace-cmd to enable kernel trace events as well as tracing lock functions, and it will then go over a quick tutorial on how to use libtracecmd to read the created trace.dat file to uncover what is the cause of interference to you application.

In this PowerPoint, learn how a security policy can be your first line of defense. Servers running AIX and other operating systems are frequent targets of cyberattacks, according to the Data Breach Investigations Report. From DoS attacks to malware, attackers have a variety of strategies at their disposal. Having a security policy in place makes it easier to ensure you have appropriate controls in place to protect mission-critical data.

"Session ID: BUD17-309 Session Name: IRQ prediction - BUD17-309 Speaker: Daniel Lezcano Track: Power Management ★ Session Summary ★ The CPUidle is one component of the power management framework. It behaves in an opportunistic way when there is nothing to do on the system, by trying to predict the next CPU wake up and select an idle state. But the current design has some weaknesses as it mixes the different sources of wakeup, resulting in an already non-deterministic situation getting worse. This presentation will describe the issues faced with the current approach and will show another approach to predict the next wake up event with a better accuracy, leading, under some circumstances, to 100% right predictions. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/bud17/bud17-309/ Presentation: https://www.slideshare.net/linaroorg/bud17309-irq-prediction Video: https://youtu.be/krFAyi1ietI --------------------------------------------------- ★ Event Details ★ Linaro Connect Budapest 2017 (BUD17) 6-10 March 2017 Corinthia Hotel, Budapest, Erzsébet krt. 43-49, 1073 Hungary --------------------------------------------------- Keyword: IRQ, power-management, CPUidle http://www.linaro.org http://connect.linaro.org --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://twitter.com/linaroorg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961"

In the context of high-performance computing (HPC), the Operating System Noise (osnoise) refers to the interference experienced by an application due to activities inside the operating system. In the context of Linux, NMIs, IRQs, softirqs, and any other system thread can cause noise to the application. Moreover, hardware-related jobs can also cause noise, for example, via SMIs. HPC users and developers that care about every microsecond stolen by the OS need not only a precise way to measure the osnoise but mainly to figure out who is stealing cpu time so that they can pursue the perfect tune of the system. These users and developers are the inspiration of Linux's osnoise tracer. The osnoise tracer runs an in-kernel loop measuring how much time is available. It does it with preemption, softirq and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise tracer takes note of the entry and exit point of any source of interferences. When the noise happens without any interference from the operating system level, the tracer can safely point to a hardware-related noise. In this way, osnoise can account for any source of interference. The osnoise tracer also adds new kernel tracepoints that auxiliaries the user to point to the culprits of the noise in a precise and intuitive way. At the end of a period, the osnoise tracer prints the sum of all noise, the max single noise, the percentage of CPU available for the thread, and the counters for the noise sources, serving as a benchmark tool.

A short, introductory talk to the world of debuggers. During the talk, we write a simple debugger application in Rust. Video at: https://www.youtube.com/watch?v=qS51kIHWARM

This document discusses interrupts, exceptions, and system calls in operating systems. It begins by explaining that the OS is event-driven and only executes when there is an interrupt, trap, or system call. It then describes the different event types - interrupts, exceptions, and how hardware interrupts are handled using interrupt controllers like the 8259 PIC and APIC. It discusses interrupt vectors, interrupt descriptor tables, and how the CPU switches to the interrupt handler. Finally, it covers system calls and exceptions as a type of software interrupt, providing an example of the write system call.

This document provides an overview of servers, processes, and system administration. It discusses servers as machines made up of components like RAM, CPU, and I/O. It then covers these components and their capacities, as well as processes and how they interact with servers through system calls. Hands-on examples are provided to demonstrate monitoring servers and investigating processes using tools like top, lsof, strace, and vmstat.

Modern operating systems are complex beasts, responsible for sharing hardware resources between many competing programs. For low-latency systems, sometimes it's necessary to subvert the OS to grab back the resources your program needs. In this talk, we will explore what is actually going on when you run a program, how much time it actually gets on CPU, and strategies to help make your code run as fast as possible. By the end of this talk, you will know how to tune your software and the linux kernel to get the most from your hardware, and more importantly, how to validate that your changes have worked.

While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies. This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.

This document discusses time measurement in embedded systems. It explains that most embedded applications need to measure time for functions like auto power off, playing media, and measuring elapsed time. It also discusses real-time systems and the difference between hard and soft real-time constraints. Examples of needing real-time measurement include missile control and autonomous vehicles. The document then talks about different ways to measure time on embedded platforms like Arduino and ESP32 using millis(), micros(), and interrupts. It provides examples of using delays or interrupts to trigger a task repeatedly every second. Finally, it provides homework questions involving measuring elapsed time and triggering tasks using delays or interrupts on Arduino or C/C++.

Ftrace is the official tracer of the Linux kernel. It has been apart of Linux since 2.6.31, and has grown tremendously ever since. Ftrace’s name comes from its most powerful feature: function tracing. But the ftrace infrastructure is much more than that. It also encompasses the trace events that are used by perf, as well as kprobes that can dynamically add trace events that the user defines. This talk will focus on learning how the kernel works by using the ftrace infrastructure. It will show how to see what happens within the kernel during a system call; learn how interrupts work; see how ones processes are being scheduled, and more. A quick introduction to some tools like trace-cmd and KernelShark will also be demonstrated. Steven Rostedt, VMware

Talk for YOW! by Brendan Gregg. "Systems performance studies the performance of computing systems, including all physical components and the full software stack to help you find performance wins for your application and kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (ftrace, bcc/BPF, and bpftrace/BPF), advice about what is and isn't important to learn, and case studies to see how it is applied. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud. "

The PREEMPT_RT patch turns Linux into a hard Real-Time designed operating system. But it takes more than just a kernel to make sure you can meet all your requirements. This talk explains all aspects of the system that is being used for a mission critical project that must be considered. Creating a Real-Time environment is difficult and there is no simple solution to make sure that your system is capable to fulfill its needs. One must be vigilant with all aspects of the system to make sure there are no surprises. This talk will discuss most of the “gotchas” that come with putting together a Real-Time system. You don’t need to be a developer to enjoy this talk. If you are curious to know how your computer is an unpredictable mess you should definitely come to this talk. Steven Rostedt - Red Hat

Complexity is increasing. Trust eroding. In the wake of Spectre and Meltdown, when it seems that things cannot get any darker for processor security, the last light goes out. This talk will demonstrate what everyone has long feared but never proven: there are hardware backdoors in x86 processors, and they’re buried deeper than we ever imagined possible. In this talk, we walk through how we discovered a privilege escalation backdoor in a family of x86 CPUs, that allows an unprivileged user, on an unmodified system, to circumvent all processor security checks and escalate from ring 3 to ring 0 – permitting an unprivileged, arbitrary userland program to directly modify and execute code inside of the kernel, regardless of the operating system, security patches, antivirus, firmware, etc. Speakers: Christopher Domas, Cyber Security Researcher

The document discusses analyzing database systems using a 3D method for performance analysis. It introduces the 3D method, which looks at performance from the perspectives of the operating system (OS), Oracle database, and applications. The 3D method provides a holistic view of the system that can help identify issues and direct solutions. It also covers topics like time-based analysis in Oracle, how wait events are classified, and having a diagnostic framework for quick troubleshooting using tools like the Automatic Workload Repository report.

This document discusses concurrency in SystemC simulations. It explains that SystemC uses events and processes to model concurrent systems. There are two main types of processes: threads and methods. Threads can wait for events using wait() and methods use next_trigger() to establish dynamic sensitivity. Events have no duration and are used to trigger processes. Notifying an event using notify() moves waiting processes to the ready queue. The SystemC kernel is event-driven and executes ready processes in non-deterministic order.

A general overview and some in depth about how operation systems schedule tasks and how much does it really cost?

The document discusses process management in operating systems. It covers control blocks, interrupts, process states, scheduling algorithms like FIFO, SJF, SRTF, Round Robin and priority scheduling. It also discusses queuing, multiprogramming vs time sharing and scheduling criteria like CPU utilization, throughput, turnaround time and waiting time. Scheduling can be long, medium or short term and algorithms include priority queues and multilevel feedback queues.

In this intermediate training session we look at resolving issues that may occur with replication (Tungsten Replicator) and clustering (Tungsten Clustering). This training is for all engineers using Continuent Tungsten solutions, as the tools we use will apply to any of our products and topologies. Some MySQL replication knowledge is assumed. AGENDA - How to identify an issue when scanning logs - Tools to help identify issues - Identifying replication lag / diagnosing replication latency - Resolving common replication issues - Resolving common clustering issues

This presentation covers the general concepts about real-time systems, how Linux kernel works for preemption, the latency in Linux, rt-preempt, and Xenomai, the real-time extension as the dual kernel approach.

This document discusses the benefits of contributing device drivers and other code to the Linux kernel upstream first instead of maintaining private branches. It outlines Chrome OS's strategy of picking long-term stable kernels and contributing hardware support to upstream. While challenging, working upstream provides many benefits like free code reviews, bugfixes, and easier updates. The document shows that Chrome OS contributions have helped increase upstream support for vendors' devices and that there is still work remaining to upstream GPU and other drivers. It demos mainlining the Samsung Chromebook Plus by using the open source Panfrost GPU driver instead of a vendor driver.

As the name would suggest, a Non-Maskable Interrupt (NMI) is an interrupt-like feature that is unaffected by the disabling of classic interrupts. In Linux, NMIs are involved in some features such as performance event monitoring, hard-lockup detector, on demand state dumping, etc… Their potential to fire when least expected can fill the most seasoned kernel hackers with dread. AArch64 (aka arm64 in the Linux tree) does not provide architected NMIs, a consequence being that features benefiting from NMIs see their use limited on AArch64. However, the Arm Generic Interrupt Controller (GIC) supports interrupt prioritization and masking, which, among other things, provides a way to control whether or not a set of interrupts can be signaled to a CPU. This talk will cover how, using the GIC interrupt priorities, we provide a way to configure some interrupts to behave in an NMI-like manner on AArch64. We’ll discuss the implementation, some of the complications that ensued and also some of the benefits obtained from it. Julien Thierry

At a rate of almost 9 changes per hour (24/7), the Linux kernel is definitely a scary beast. Bugs are introduced on a daily basis and, through the use of multiple code analyzers, *some* of them are detected and fixed before they hit mainline. Over the course of the last few years, Gustavo has been fixing such bugs and many different issues in every corner of the Linux kernel. Recently, he was in charge of leading the efforts to globally enable -Wimplicit-fallthrough; which appears by default in Linux v5.3. This presentation is a report on all the stuff Gustavo has found and fixed in the kernel with the support of the Core Infrastructure Initiative. Gustavo A.R. Silva

In I.T. we all use all kinds of metrics. Operations teams rely heavily on these, especially when things go south. These metrics are sometimes overrated. Let’s dive into a few real life stories together. Aurélien Rougemont

This document discusses the current state and future plans for Linux kernel documentation. It notes that documentation has transitioned from DocBook to Sphinx/RST, improving formatting and integration. There are now over 3,000 documentation files and many kerneldoc comments. Future plans include converting remaining text files, updating ancient documents, improving organization by topic, integrating documentation better, and enhancing the documentation toolchain. The goal is to improve documentation to better help kernel developers, users and the community.