Performance Analysis: new tools and concepts from the cloud

Video: https://youtu.be/eO94l0aGLCA?t=3m37s . Talk by Brendan Gregg for ACM Applicative 2016 "System Methodology - Holistic Performance Analysis on Modern Systems Traditional systems performance engineering makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. For modern systems, we can choose the metrics, and can choose ones we need to support new holistic performance analysis methodologies. These methodologies provide faster, more accurate, and more complete analysis, and can provide a starting point for unfamiliar systems. Methodologies are especially helpful for modern applications and their workloads, which can pose extremely complex problems with no obvious starting point. There are also continuous deployment environments such as the Netflix cloud, where these problems must be solved in shorter time frames. Fortunately, with advances in system observability and tracers, we have virtually endless custom metrics to aid performance analysis. The problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems. System methodologies provide a starting point for analysis, as well as guidance for quickly moving through the metrics to root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, and more. This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. The focus is on single systems (any operating system), including single cloud instances, and quickly locating performance issues or exonerating the system. Many methodologies will be discussed, along with recommendations for their implementation, which may be as documented checklists of tools, or custom dashboards of supporting metrics. In general, you will learn to think differently about your systems, and how to ask better questions."

Tracing Summit 2014, Düsseldorf. What can Linux learn from DTrace: what went well, and what didn't go well, on its path to success? This talk will discuss not just the DTrace software, but lessons from the marketing and adoption of a system tracer, and an inside look at how DTrace was really deployed and used in production environments. It will also cover ongoing problems with DTrace, and how Linux may surpass them and continue to advance the field of system tracing. A world expert and core contributor to DTrace, Brendan now works at Netflix on Linux performance with the various Linux tracers (ftrace, perf_events, eBPF, SystemTap, ktap, sysdig, LTTng, and the DTrace Linux ports), and will summarize his experiences and suggestions for improvements. He has also been contributing to various tracers: recently promoting ftrace and perf_events adoption through articles and front-end scripts, and testing eBPF.

CMP325 talk for AWS re:Invent 2017, by Brendan Gregg. " At Netflix we make the best use of AWS EC2 instance types and features to create a high performance cloud, achieving near bare metal speed for our workloads. This session will summarize the configuration, tuning, and activities for delivering the fastest possible EC2 instances, and will help other EC2 users improve performance, reduce latency outliers, and make better use of EC2 features. We'll show how we choose EC2 instance types, how we choose between EC2 Xen modes: HVM, PV, and PVHVM, and the importance of EC2 features such SR-IOV for bare-metal performance. SR-IOV is used by EC2 enhanced networking, and recently for the new i3 instance type for enhanced disk performance as well. We'll also cover kernel tuning and observability tools, from basic to advanced. Advanced performance analysis includes the use of Java and Node.js flame graphs, and the new EC2 Performance Monitoring Counter (PMC) feature released this year."

Talk from SREcon2016 by Brendan Gregg. Video: https://www.usenix.org/conference/srecon16/program/presentation/gregg . "There's limited time for performance analysis in the emergency room. When there is a performance-related site outage, the SRE team must analyze and solve complex performance issues as quickly as possible, and under pressure. Many performance tools and techniques are designed for a different environment: an engineer analyzing their system over the course of hours or days, and given time to try dozens of tools: profilers, tracers, monitoring tools, benchmarks, as well as different tunings and configurations. But when Netflix is down, minutes matter, and there's little time for such traditional systems analysis. As with aviation emergencies, short checklists and quick procedures can be applied by the on-call SRE staff to help solve performance issues as quickly as possible. In this talk, I'll cover a checklist for Linux performance analysis in 60 seconds, as well as other methodology-derived checklists and procedures for cloud computing, with examples of performance issues for context. Whether you are solving crises in the SRE war room, or just have limited time for performance engineering, these checklists and approaches should help you find some quick performance wins. Safe flying."

This document discusses using Linux perf_events (perf) profiling tools to analyze Java performance on Linux. It describes how perf can provide complete visibility into Java, JVM, GC and system code but that Java profilers have limitations. It presents the solution of using perf to collect mixed-mode flame graphs that include Java method names and symbols. It also discusses fixing issues with broken Java stacks and missing symbols on x86 architectures in perf profiles.

This document summarizes a presentation on flame graphs for profiling CPU and memory performance on FreeBSD. It introduces flame graphs as a way to visualize stack profiles to easily compare performance across systems. Examples are given profiling MySQL workload CPU usage on two hosts to identify a 30% performance difference. Commands are provided to generate flame graphs from DTrace profiles of CPU stack sampling and page faults.

Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg. There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes. This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.

My talk for BayLISA, Oct 2013, launching the Systems Performance book. Operating system performance analysis and tuning leads to a better end-user experience and lower costs, especially for cloud computing environments that pay by the operating system instance. This book covers concepts, strategy, tools and tuning for Unix operating systems, with a focus on Linux- and Solaris-based systems. The book covers the latest tools and techniques, including static and dynamic tracing, to get the most out of your systems.

Slides from a recent performance meetup at Netflix. A look into how we measure and tune performance for our clients and the streaming service.

Your AMI is one of the core foundations for running applications and services effectively on Amazon EC2. In this session, you'll learn how to optimize your AMI, including how you can measure and diagnose system performance and tune parameters for improved CPU and network performance. We'll cover application-specific examples from Netflix on how optimized AMIs can lead to improved performance.

This document provides a performance engineer's predictions for computing performance trends in 2021 and beyond. The engineer discusses trends in processors, memory, disks, networking, runtimes, kernels, hypervisors, and observability. For processors, predictions include multi-socket systems becoming less common, the future of simultaneous multithreading being unclear, practical core count limits being reached in the 2030s, and more processor vendors including ARM-based and RISC-V options. Memory predictions focus on many workloads being memory-bound currently.

Introduction to DTrace (Dynamic Tracing), written by Brendan Gregg and delivered in 2007. While aimed at a Solaris-based audience, this introduction is still largely relevant today (2012). Since then, DTrace has appeared in other operating systems (Mac OS X, FreeBSD, and is being ported to Linux), and, many user-level providers have been developed to aid tracing of other languages.

Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.

MeetBSDCA 2014 Performance Analysis for BSD, by Brendan Gregg. A tour of five relevant topics: observability tools, methodologies, benchmarking, profiling, and tracing. Tools summarized include pmcstat and DTrace.

keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems. There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more. This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."

The document summarizes a talk on container performance analysis. It discusses identifying bottlenecks at the host, container, and kernel level using various Linux performance tools. It then provides an overview of how containers work in Linux using namespaces and control groups (cgroups). Finally, it demonstrates some example commands like docker stats, systemd-cgtop, and bcc/BPF tools that can be used to analyze containers and cgroups from the host system.

Keynote for PerconaLive 2018 by Brendan Gregg. Video: https://youtu.be/sV3XfrfjrPo?t=30m51s . "At over one thousand code commits per week, it's hard to keep up with Linux developments. This keynote will summarize recent Linux performance features, for a wide audience: the KPTI patches for Meltdown, eBPF for performance observability, Kyber for disk I/O scheduling, BBR for TCP congestion control, and more. This is about exposure: knowing what exists, so you can learn and use it later when needed. Get the most out of your systems, whether they are databases or application servers, with the latest Linux kernels and exciting features."

Delivered at the FISL13 conference in Brazil: http://www.youtube.com/watch?v=K9w2cipqfvc This talk introduces the USE Method: a simple strategy for performing a complete check of system performance health, identifying common bottlenecks and errors. This methodology can be used early in a performance investigation to quickly identify the most severe system performance issues, and is a methodology the speaker has used successfully for years in both enterprise and cloud computing environments. Checklists have been developed to show how the USE Method can be applied to Solaris/illumos-based and Linux-based systems. Many hardware and software resource types have been commonly overlooked, including memory and I/O busses, CPU interconnects, and kernel locks. Any of these can become a system bottleneck. The USE Method provides a way to find and identify these. This approach focuses on the questions to ask of the system, before reaching for the tools. Tools that are ultimately used include all the standard performance tools (vmstat, iostat, top), and more advanced tools, including dynamic tracing (DTrace), and hardware performance counters. Other performance methodologies are included for comparison: the Problem Statement Method, Workload Characterization Method, and Drill-Down Analysis Method.

Video: http://joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x ; This talk for SCaLE11x covers system performance analysis methodologies and the Linux tools to support them, so that you can get the most out of your systems and solve performance issues quickly. This includes a wide variety of tools, including basics like top(1), advanced tools like perf, and new tools like the DTrace for Linux prototypes.

Delivered as plenary at USENIX LISA 2013. video here: https://www.youtube.com/watch?v=nZfNehCzGdw and https://www.usenix.org/conference/lisa13/technical-sessions/plenary/gregg . "How did we ever analyze performance before Flame Graphs?" This new visualization invented by Brendan can help you quickly understand application and kernel performance, especially CPU usage, where stacks (call graphs) can be sampled and then visualized as an interactive flame graph. Flame Graphs are now used for a growing variety of targets: for applications and kernels on Linux, SmartOS, Mac OS X, and Windows; for languages including C, C++, node.js, ruby, and Lua; and in WebKit Web Inspector. This talk will explain them and provide use cases and new visualizations for other event types, including I/O, memory usage, and latency.

This document discusses the challenges of building and debugging DIRT (data-intensive real-time) applications in production. It provides examples from the mobile push-to-talk app Voxer, which is described as a canonical DIRT app. Specific issues covered include application restarts inducing latency bubbles, dropped TCP connections causing latency outliers, and identifying sources of slow disk I/O. Tools like DTrace are highlighted as being essential for instrumentation and problem diagnosis in DIRT apps.

HTTP applications concentrate many performance issues: - They are a common way to let internal & external users access and modify data. - They rely on a delivery chain which contains many elements, which are all performance drivers: browser, workstation, network, front-server, application server, file server, database, images, etc... - They raise specific troubleshooting issues: among others, the end user feedback is based on the concept of page, while most network based performance analysis is based on every transaction / object in the page (html, css, image, script, etc.) This one-hour webinar will enable you to: - Understand the challenges of performance troubleshooting for HTTP Applications. - View a series of concrete diagnostic cases with Performance Vision newest version.

Performance analysis aims to capture, analyze, and evaluate key components of performance through systematic observation. Coaches observe to better understand technical, tactical, behavioral, and physical aspects of performance. They then provide feedback to improve future practice. Coaches use various methods like notation, video, biomechanics, tests, and questionnaires to gather both qualitative and quantitative data on performance. Technology applications and software programs help support detailed analysis.

A presentation from SEO Campixx Barcamp 2011 in Berlin. Web Performance Optimization is about making websites faster. Here i discussed different measures and show the impact on competitive advantage and possibly rankings on Google. Undeniably you can say that better performance leads to more sales and better usability in terms of bouncing rates. View image slides here: http://b0i.de/wpopresentation

A presentation that provides an overview of software testing approaches including "schools" of software testing and a variety of testing techniques and practices.

"DTracing the Cloud", Brendan Gregg, illumosday 2012 Cloud computing facilitates rapid deployment and scaling, often pushing high load at applications under continual development. DTrace allows immediate analysis of issues on live production systems even in these demanding environments – no need to restart or run a special debug kernel. For the illumos kernel, DTrace has been enhanced to support cloud computing, providing more observation capabilities to zones as used by Joyent SmartMachine customers. DTrace is also frequently used by the cloud operators to analyze systems and verify performance isolation of tenants. This talk covers DTrace in the illumos-based cloud, showing examples of real-world performance wins.

A brief talk on systems performance for the July 2013 meetup "A Midsummer Night's System", video: http://www.youtube.com/watch?v=P3SGzykDE4Q. This summarizes how systems performance has changed from the 1990's to today. This was the reason for writing a new book on systems performance, to provide a reference that is up to date, covering new tools, technologies, and methodologies.

LinuxCon Europe, 2014. Video: https://www.youtube.com/watch?v=SN7Z0eCn0VY . There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This talk summarizes the three types of performance tools: observability, benchmarking, and tuning, providing a tour of what exists and why they exist. Advanced tools including those based on tracepoints, kprobes, and uprobes are also included: perf_events, ktap, SystemTap, LTTng, and sysdig. You'll gain a good understanding of the performance tools landscape, knowing what to reach for to get the most out of your systems.

Single page applications are a problem for RUM tools because there are no easy ways to tell when a new page component has been requested asynchronously as a result of an intentional user action. Many network requests are back-end service calls initiated periodically by the app – for example, a ping to check if content has been updated, or to check if the current user should still be signed in to their account. Even with requests that are initiated by a user action, not all may fit into the definition of a “page view.” For example, a user typing into a search box that has auto-complete capabilities will often result in network requests, but these requests result in very small amounts of data transfer, happen very frequently, and do not count toward page views. The scene is further complicated by SPA frameworks like Angular, Backbone, and others. In this talk, we’ll learn about some of the tricks used by boomerang to measure the performance of single page applications, going as far as capturing errors and waterfall information across browsers.

The document outlines a methodology for Application Performance Management (APM). It discusses various components of an APM strategy including top-down monitoring, bottom-up monitoring, reporting and analytics, and aligning with ITIL processes. Top-down monitoring focuses on real-time application monitoring using techniques like synthetic transactions. Bottom-up monitoring ties into infrastructure monitoring tools. Reporting and analytics is used to analyze performance data and establish baselines. APM supports various ITIL processes like incident management, problem management and service level management.

Software performance engineering is becoming increasingly important to businesses as they look to improve the non-functional performance of applications and get more out of IT investments. By leveraging performance engineering techniques, IT professionals can be indispensable in building and optimizing scalable systems. This introductory course will teach you the essentials of software performance engineering including : • The performance challenges faced by Enterprise IT today • What is software performance engineering (SPE)? • Best practices for building scalable software systems • The approaches to integrating SPE into IT project lifecycles • Common frameworks for measuring application performance and service levels • The impact of SPE on software developers, testers, capacity planes, and other IT professionals • Case studies from the finance, retail, and insurance industries Instructor: Walter Kuketz, SVP and CTO, Collaborative Consulting This training is sponsored by Correlsense, Collaborative Consulting, and New Horizons

Learn how Site24x7 gives you end-to-end application performance visibility for your Java, .NET and Ruby web transactions with metrics of all components starting from URLs to SQL queries.

The document discusses effective use of performance analysis in coaching rugby. It provides examples of how performance analysis has evolved from basic notation to advanced digital tools for game analysis, technique analysis, and player tracking. It emphasizes using permanent records of performance to stimulate athlete learning in both training and competition environments. The coach's role is to develop a strategic, periodized approach to performance analysis to support long-term athlete development.

Akamai Edge 14 - Discussion on RUM, Synthetic and setting realistic and meaningful performance goals.

The document discusses Nisa Retail's use of dynaTrace to improve service and cut costs. It provides an overview of Nisa Retail and Intechnica, and how dynaTrace was implemented at Nisa Retail to deliver business value. dynaTrace provided end-to-end application monitoring across all tiers, full transaction tracing, and proactive service level engineering to help optimize performance. This improved the user experience and helped Nisa Retail do more with less staff and budget.

Dynatrace is an APM solution that provides deep visibility into application performance across complex, distributed environments. It uses PurePath technology to capture timing and code-level context for all transactions end-to-end. This allows Dynatrace to identify performance issues and their root causes faster than other tools. Dynatrace can monitor Apache Tomcat servers and provide metrics on JVM performance, database queries, requests, and more. It helps diagnose common issues like inefficient database access, microservice problems, and coding issues.

Video: https://www.youtube.com/watch?v=uibLwoVKjec . Talk by Brendan Gregg for Sysdig CCWFS 2016. Abstract: "You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!"

iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information. In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc. Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.

iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information. In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc. Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.

Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."

The document summarizes a talk on container performance analysis. It discusses identifying bottlenecks at the host, container, and kernel level using various Linux performance tools. It also provides an overview of how containers work in Linux using namespaces and control groups (cgroups). Specifically, it demonstrates analyzing resource usage and limitations for containers using tools like docker stats, systemd-cgtop, and investigating namespaces.

An overview of CERN container service, based on OpenStack Magnum, focusing on the new Docker Swatm Mode driver.

This document discusses instrumenting and running Node.js applications in production environments. It describes how Node.js is well-suited for building "DIRTy" real-time web applications due to its asynchronous and event-driven architecture. The document advocates for using dynamic instrumentation tools like DTrace to measure latency in Node.js and visualize latency data through techniques like 4D heatmaps to debug performance issues.

Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.​

This document provides an overview of systemd and how it differs from traditional init systems. It discusses systemd units and how to manage services using systemctl. It covers customizing units using drop-ins, managing resources with cgroups, converting init scripts, and using the systemd journal. The presentation aims to demystify systemd and provide administrators with practical guidance on using its main features.

This document summarizes a presentation about tuning parallel code on Solaris. It discusses: 1) Using tools like DTrace, prstat, and vmstat to analyze performance issues like thread scheduling and I/O problems in parallel applications on Solaris. 2) Two examples of using DTrace to analyze thread scheduling and troubleshoot I/O performance problems in a virtualized Windows server. 3) How the examples demonstrated using DTrace to identify unbalanced thread scheduling and discover that a domain controller was disabling disk write caching, slowing performance.

System Device Tree is an extension to Device Tree to describe all the hardware on an SoC, including heterogeneous CPU clusters and secure resources not typically visible to an Operating System like Linux. This full view allows the System Device Tree to be the "One true source" of the entire hardware description and helps to prevent the common (and hard-to-debug) problem of conflicting resources and system consistency. Lopper is an Open Source framework to parse and manipulate System Device Tree. With Lopper, it is possible to generate multiple traditional Device Trees from a single larger System Device Tree. This presentation will provide an overview of System Device Tree and will discuss the latest updates of the specification and tooling. The talk will illustrate multiple use-cases for System Device Tree with concrete examples, such as Linux running on the more powerful CPU cluster and Zephyr running on a smaller Cortex-R cluster. It will also show how to use Lopper to generate multiple traditional Device Trees targeting different OSes, not just Linux but also Zephyr/other RTOSes. Finally, an end-to-end demo based on Yocto to build a complete heterogeneous system with multiple OSes and RTOSes running on different clusters on a single reference board will be shown.

Amazon EC2 provides a broad selection of instance types to accommodate a diverse mix of workloads. In this session, we provide an overview of the Amazon EC2 instance platform, key platform features, and the concept of instance generations. We dive into the current generation design choices of the different instance families, including the General Purpose, Compute Optimized, Storage Optimized, Memory Optimized, and Accelerated Computing (GPU and FPGA) instance families. We also detail best practices and share performance tips for getting the most out of your Amazon EC2 instances.

This document discusses high availability for PostgreSQL in a containerized environment. It outlines typical enterprise requirements for high availability including recovery time objectives and recovery point objectives. Shared storage-based high availability is described as well as the advantages and disadvantages of PostgreSQL replication. The use of Linux containers and orchestration tools like Kubernetes and Consul for managing containerized PostgreSQL clusters is also covered. The document advocates for using PostgreSQL replication along with services and self-healing tools to provide highly available and scalable PostgreSQL deployments in modern container environments.

1. The document discusses using OpenStack for a 4G core network, including performance issues and solutions when virtualizing the EPC network functions using OpenStack. 2. Key performance issues identified include high CPU usage, competing for CPU resources, latency, throughput, and packet loss. Solutions proposed are CPU pinning, NUMA awareness, hugepages, DPDK, SR-IOV, and offloading processing to smart NICs. 3. Going forward, the next steps discussed are using OVS-DPDK for offloading, SDN, containers, and cloud architectures for 5G.

There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe several techniques available in R to speed up workloads like these, by running multiple iterations simultaneously, in parallel. Many of these techniques require the use of a cluster of machines running R, and I'll provide examples of using cloud-based services to provision clusters for parallel computations. In particular, I will describe how you can use the SparklyR package to distribute data manipulations using the dplyr syntax, on a cluster of servers provisioned in the Azure cloud. Presented by David Smith at Data Day Texas in Austin, January 27 2018.

Vinetalk is a software abstraction layer that allows cluster managers like Mesos and Kubernetes to offer fractions of GPU resources, enabling more efficient sharing of accelerators. Existing cluster managers cannot share accelerators because device drivers do not support it. Vinetalk implements an abstraction layer that decouples executors from vendor-specific drivers, representing accelerators as virtual access queues. This allows multiple tasks to concurrently use the same physical accelerator. Vinetalk has been shown to reduce queuing times for tasks sharing a GPU compared to Mesos alone. It also easier for developers to use, hiding proprietary device APIs, and has low overhead of 1-5% due to memory transfers.

This document provides an introduction to CUDA programming. It discusses the programmer's view of the GPU as a co-processor with its own memory, and how GPUs are well-suited for data-parallel applications with many independent computations. It describes how CUDA uses a grid of blocks of threads to run kernels in parallel. Memory is organized into global, constant, shared, and local memory. Kernels launch a grid of blocks, and threads within blocks can cooperate through shared memory and synchronization.

Learn the basics of monitoring Scylla, including monitoring infrastructure and understanding Scylla metrics.

The increasing complexity of learning algorithms and deep neural networks, combined with size of data and parameters, has made it challenging to exploit existing large-scale data processing pipelines for training and inference. Approaches are outlined for preprocessing, training, inference, and deployment across datasets that leverage Spark, its extended ecosystem of libraries, and deep learning frameworks.

This document provides an overview of Oracle performance tuning fundamentals. It discusses key concepts like wait events, statistics, CPU utilization, and the importance of understanding the operating system, database, and business needs. It also introduces tools for monitoring performance like AWR, ASH, and dynamic views. The goal is to establish a foundational understanding of Oracle performance concepts and monitoring techniques.

This document discusses tuning Oracle GoldenGate for optimal performance. It begins with an overview of GoldenGate architecture and use cases, then discusses the importance of baseline monitoring. Key metrics to monitor are identified as lag times, checkpoint information, CPU usage, memory usage, and disk I/O. The document provides examples of commands to gather baseline data on these metrics. It then discusses configuring GoldenGate for parallel processing using multiple process groups to optimize performance. Overall it provides guidance on setting baselines and configuring GoldenGate to minimize lag times and resource utilization.

The document discusses challenges with processor benchmarking and provides recommendations. It summarizes a case study where a popular CPU benchmark claimed a new processor was 2.6x faster than Intel, but detailed analysis found the benchmark was testing division speed, which accounted for only 0.1% of cycles on Netflix servers. The document advocates for low-level, active benchmarking and profiling over statistical analysis. It also provides a checklist for evaluating benchmarks and cautions that increased processor complexity and cloud environments make accurate benchmarking more difficult.

This document provides an overview of using eBPF (extended Berkeley Packet Filter) to quickly get performance wins as a sysadmin. It recommends installing BCC and bpftrace tools to easily find issues like periodic processes, misconfigurations, unexpected TCP sessions, or slow file system I/O. A case study examines using biosnoop to identify which processes were causing disk latency issues. The document suggests thinking like a sysadmin first by running tools, then like a programmer if a problem requires new tools. It also outlines recommended frontends depending on use cases and provides references to learn more about BPF.

Talk for Facebook Systems@Scale 2021 by Brendan Gregg: "BPF (eBPF) tracing is the superpower that can analyze everything, helping you find performance wins, troubleshoot software, and more. But with many different front-ends and languages, and years of evolution, finding the right starting point can be hard. This talk will make it easy, showing how to install and run selected BPF tools in the bcc and bpftrace open source projects for some quick wins. Think like a sysadmin, not like a programmer."

Talk by Brendan Gregg for USENIX LISA 2021. https://www.youtube.com/watch?v=5nN1wjA_S30 . "The future of computer performance involves clouds with hardware hypervisors and custom processors, servers running a new type of BPF software to allow high-speed applications and kernel customizations, observability of everything in production, new Linux kernel technologies, and more. This talk covers interesting developments in systems and computing performance, their challenges, and where things are headed."

USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.

Keynote by Brendan Gregg for the eBPF summit, 2020. How to get started finding performance wins using the BPF (eBPF) technology. This short talk covers the quickest and easiest way to find performance wins using BPF observability tools on Linux.