This document discusses the crash reporting mechanism in Tizen. It describes the crash client, which handles crash signals and generates crash reports. It covers Samsung's crash-work-sdk and Intel's corewatcher crash clients. It also discusses the crash server that receives reports and the CrashDB web interface. Finally, it mentions crash reason location algorithms.
This document discusses Cisco IOS shellcoding and reverse engineering. It covers topics like Cisco IOS shellcodes that are image-independent by disassembling or interrupting hijacking. It also discusses Tcl shellcodes, Cisco IOS reverse engineering challenges including lack of modularity and APIs. The document details subsystems, registries, processes, command parser tree, debugging Cisco IOS, and magic numbers used in Cisco IOS.
This document provides an overview of kernel debugging on Solaris systems using the modular debugger Mdb and dynamic tracing framework DTrace. It discusses debugging live kernels with Mdb, analyzing system crash dumps with Mdb, and using DTrace to monitor the kernel at runtime by enabling probes published by different providers. The document outlines the key tools, techniques, and challenges involved in kernel debugging and crash analysis on Solaris.
Compared to x86, ARM shellcode has made little progress. The x86 hardware is largely homogenous. ARM, however, has several versions and variants across devices today. There are several constraints and subtleties involved in writing production quality ARM shellcode which works on modern ARM hardware, not just on QEMU emulators.
In this talk, we shall explore issues such as overcoming cache coherency, reliable polymorphic shellcode, ARM egghunting and last but not the least, polyglot ARM shellcode. A bonus side effect of this talk will be creating headaches for those who like to defend agaisnt attacks using age old signature based techniques.
The document describes how to port and modify drivers for UART, Ethernet, LCD, and keypad on a Mango100 board running Android. It provides instructions on configuring the kernel, modifying driver source code files, and checking that the drivers are functioning properly. Key steps include enabling drivers in the kernel .config file, adding device registration code, and modifying functions to set GPIO pins and timing parameters for devices like the LCD. It also explains how to view input events in logcat and trace the flow of key events through the Android framework.
Creating "Secure" PHP applications, Part 2, Server Hardening
The document provides guidance on server hardening techniques. It discusses using netstat to view listening services on a server and using update-rc.d or chkconfig to disable unnecessary services from starting at boot. It also recommends enabling access control lists (ACLs) in file system mounts, using SELinux or AppArmor to enforce mandatory access controls, and setting reasonable PHP memory limits to prevent potential denial of service attacks. The document stresses the importance of only allowing approved applications to execute and knowing the resource limits of the server to avoid potential outages.
Slides from my presentation on ARM Shellcode at #44CON 2018, London.
In this talk, we explore ARM egghunting and "Quantum Leap" code - polyglot ARM shellcode. A bonus side effect of this talk will be creating headaches for those who like to defend agaisnt attacks using age old signature based techniques.
The document discusses kernel debugging techniques. It covers analyzing kernel error logs, debugging memory errors, detecting kernel deadlocks, and troubleshooting login issues. The key debugging steps involve prerequisites like having the kernel source code, reading the error logs and call traces to understand the bug, and tracing from assembly to C code to transactions. Memory debugging tools like AddressSanitizer are also covered.
The document discusses SPARC, an open instruction set architecture developed by Sun Microsystems. It provides an overview of SPARC features such as the 32-bit and 64-bit versions, 128 general purpose registers, and floating-point unit capabilities. It also describes the register window mechanism and how it is used in a "Hello World" program example.
TIP1 - Overview of C/C++ Debugging/Tracing/Profiling Tools
This document provides an overview and comparison of various tools for debugging, profiling, and tracing C/C++ programs on Linux. It discusses concepts and implementations of debugging tools like gdb and Valgrind, profiling tools like gprof and perf, and tracing tools like strace, SystemTap, and LTTng. For each tool, it provides examples of common use cases and highlights strengths and limitations. The document aims to help developers select the appropriate tool based on their specific needs such as debugging memory errors, profiling performance bottlenecks, or tracing system calls.
The document discusses honeypots, which are computer resources dedicated to being probed, attacked, or compromised. Honeypots can be used to detect internal attacks, identify scans and automated attacks, identify trends, keep attackers away from important systems, and collect signatures of attacks and malicious code. They work by emulating known vulnerabilities to collect information about attacks. Honeypots include low and high interaction varieties. Popular honeypot software includes Honeyd, which simulates virtual networks, and Nepenthes, which emulates vulnerabilities to capture binaries and commands executed by worms. Logs from honeypots can be analyzed to identify attack sources and collect malware samples.
Since the emerging of the OpenStack cloud computing platform in the Ubuntu community, increasing number of public/private cloud service providers choose to deploy it all over the world. Recently, Spectre and Meltdown have caused a panic in the world and the Spectre V2 is the only one which can attack the host system from the guest VM. It's vital to know the detailed process of the attack. Gavin Guo will give a detail explanation and an example of how to attack the host system. Besides, v1/v3/v4 are also introduced in the slide.
This document discusses eBPF (extended Berkeley Packet Filter), which allows tracing from the Linux kernel to userspace using BPF programs. It provides an overview of eBPF including extended registers, verification, maps, and probes. Examples are given of using eBPF for tracing functions like kfree_skb() and the C library function malloc. The Berkeley Compiler Collection (BCC) makes it easy to write eBPF programs in C and Python.
bcc/BPF tools - Strategy, current tools, future challenges
Brendan Gregg discusses the current state and future potential of BPF and BCC tools for observability in Linux. He outlines 18 areas where BPF support has progressed and 16 areas still needing work. Gregg also discusses challenges like dynamic tracing stability, overhead, ease of coding, and developing visualizations. He proposes finishing ports of his old DTrace tools and links to resources on BPF, BCC, and flame graphs.
UrQA is a mobile QA system that collects bug reports from Android apps to provide more detailed information than competitors like BugSense and ACRA. It displays an "event path" for each bug to show the steps to reproduce crashes, while competitors only show basic numbers. UrQA also uses a client-server architecture where the client handles native crashes and sends crash dumps to the server for analysis and storage.
PCD – Process Control Daemon is a light-weight system level process manager for Embedded-Linux based projects (consumer electronics, network devices, etc.).
PCD starts, stops and monitors all the user space processes in the system, in a synchronized manner, using a textual configuration file.
PCD recovers the system in case of errors and provides useful and detailed debug information.
Slides of my talk on Devel::NYTProf and optimizing perl code at YAPC::NA in June 2014. It covers use of NYTProf and outlines a multi-phase approach to optimizing your perl code.
A video of the talk and questions is available at https://www.youtube.com/watch?v=T7EK6RZAnEA&list=UU7y4qaRSb5w2O8cCHOsKZDw
This document discusses Android memory management. It explains that Android does not have swap space and relies on garbage collection. It describes the different memory regions in Android including the Java heap, native heap, and ashmem. It provides tips on increasing heap size, viewing heap updates, and how Android manages memory through processes. It also covers topics like memory leaks, references, and tools for analyzing memory usage.
Controlling Memory Footprint at All Layers: Linux Kernel, Applications, Libra...
Reducing memory usage is well covered in the history of this conference, yet new tricks still do exist. When optimizing memory footprint for an home gateway device, the author found some unexpected places where small changes can save valuable amount of DRAM or Flash space. This talk will visit different areas including - Kernel: fragmentation threshold, page frame reclamation task and atomic memory. Application level: Memory inefficient shared libraries due to ABI compliance and dynamic loading. Toolchain: Tuning malloc allocator parameters and compiler options. System level: General kernel might be more memory efficient than MMU-less uClinux, and preventing lock up when the system is on the brink of running out of memory.
Slides for my Perl Memory Use talk at YAPC::Asia in Tokyo, September 2012.
(This uploaded version includes quite a few slides from the OSCON version that I skipped at YAPC::Asia in order to have more time for a demo.)
This document discusses profiling the AllPaths-LG genome assembler to optimize its performance. It analyzed the CPU and memory usage of each program step on various systems. The profiling identified seven routines that use the most CPU time and I/O. Modules like FindErrors, AlignReads, and CommonPather were prioritized for optimization to reduce assembly time. Future work will involve more detailed profiling and exploring code optimizations for the most resource-intensive modules.
- The document discusses various Linux system log files such as /var/log/messages, /var/log/secure, and /var/log/cron and provides examples of log entries.
- It also covers log rotation tools like logrotate and logwatch that are used to manage log files.
- Networking topics like IP addressing, subnet masking, routing, ARP, and tcpdump for packet sniffing are explained along with examples.
No instrumentation Golang Logging with eBPF (GoSF talk 11/11/20)Pixie Labs
The document discusses using eBPF for instrumentation and logging in Golang applications without source code modifications. It provides an example of using eBPF to log function arguments by attaching a BPF program to the computeE function via uprobes. This allows viewing function parameters in production without recompiling or using a debugger. eBPF provides low overhead dynamic tracing of all application code compared to other options like debuggers or static tracing tools.
The document provides information on systemd service management commands. It shows examples of using systemctl to start, stop, restart, and check the status of the httpd service. It also displays the output of systemctl status httpd which shows details about the loaded unit, active state, process IDs, and log entries for the Apache HTTP Server service.
This document discusses Cisco IOS shellcoding and reverse engineering. It covers topics like Cisco IOS shellcodes that are image-independent by disassembling or interrupting hijacking. It also discusses Tcl shellcodes, Cisco IOS reverse engineering challenges including lack of modularity and APIs. The document details subsystems, registries, processes, command parser tree, debugging Cisco IOS, and magic numbers used in Cisco IOS.
This document provides an overview of kernel debugging on Solaris systems using the modular debugger Mdb and dynamic tracing framework DTrace. It discusses debugging live kernels with Mdb, analyzing system crash dumps with Mdb, and using DTrace to monitor the kernel at runtime by enabling probes published by different providers. The document outlines the key tools, techniques, and challenges involved in kernel debugging and crash analysis on Solaris.
HackLU 2018 Make ARM Shellcode Great AgainSaumil Shah
Compared to x86, ARM shellcode has made little progress. The x86 hardware is largely homogenous. ARM, however, has several versions and variants across devices today. There are several constraints and subtleties involved in writing production quality ARM shellcode which works on modern ARM hardware, not just on QEMU emulators.
In this talk, we shall explore issues such as overcoming cache coherency, reliable polymorphic shellcode, ARM egghunting and last but not the least, polyglot ARM shellcode. A bonus side effect of this talk will be creating headaches for those who like to defend agaisnt attacks using age old signature based techniques.
The document describes how to port and modify drivers for UART, Ethernet, LCD, and keypad on a Mango100 board running Android. It provides instructions on configuring the kernel, modifying driver source code files, and checking that the drivers are functioning properly. Key steps include enabling drivers in the kernel .config file, adding device registration code, and modifying functions to set GPIO pins and timing parameters for devices like the LCD. It also explains how to view input events in logcat and trace the flow of key events through the Android framework.
Creating "Secure" PHP applications, Part 2, Server Hardeningarchwisp
The document provides guidance on server hardening techniques. It discusses using netstat to view listening services on a server and using update-rc.d or chkconfig to disable unnecessary services from starting at boot. It also recommends enabling access control lists (ACLs) in file system mounts, using SELinux or AppArmor to enforce mandatory access controls, and setting reasonable PHP memory limits to prevent potential denial of service attacks. The document stresses the importance of only allowing approved applications to execute and knowing the resource limits of the server to avoid potential outages.
Slides from my presentation on ARM Shellcode at #44CON 2018, London.
In this talk, we explore ARM egghunting and "Quantum Leap" code - polyglot ARM shellcode. A bonus side effect of this talk will be creating headaches for those who like to defend agaisnt attacks using age old signature based techniques.
The document discusses kernel debugging techniques. It covers analyzing kernel error logs, debugging memory errors, detecting kernel deadlocks, and troubleshooting login issues. The key debugging steps involve prerequisites like having the kernel source code, reading the error logs and call traces to understand the bug, and tracing from assembly to C code to transactions. Memory debugging tools like AddressSanitizer are also covered.
The document discusses SPARC, an open instruction set architecture developed by Sun Microsystems. It provides an overview of SPARC features such as the 32-bit and 64-bit versions, 128 general purpose registers, and floating-point unit capabilities. It also describes the register window mechanism and how it is used in a "Hello World" program example.
TIP1 - Overview of C/C++ Debugging/Tracing/Profiling ToolsXiaozhe Wang
This document provides an overview and comparison of various tools for debugging, profiling, and tracing C/C++ programs on Linux. It discusses concepts and implementations of debugging tools like gdb and Valgrind, profiling tools like gprof and perf, and tracing tools like strace, SystemTap, and LTTng. For each tool, it provides examples of common use cases and highlights strengths and limitations. The document aims to help developers select the appropriate tool based on their specific needs such as debugging memory errors, profiling performance bottlenecks, or tracing system calls.
The document discusses honeypots, which are computer resources dedicated to being probed, attacked, or compromised. Honeypots can be used to detect internal attacks, identify scans and automated attacks, identify trends, keep attackers away from important systems, and collect signatures of attacks and malicious code. They work by emulating known vulnerabilities to collect information about attacks. Honeypots include low and high interaction varieties. Popular honeypot software includes Honeyd, which simulates virtual networks, and Nepenthes, which emulates vulnerabilities to capture binaries and commands executed by worms. Logs from honeypots can be analyzed to identify attack sources and collect malware samples.
Since the emerging of the OpenStack cloud computing platform in the Ubuntu community, increasing number of public/private cloud service providers choose to deploy it all over the world. Recently, Spectre and Meltdown have caused a panic in the world and the Spectre V2 is the only one which can attack the host system from the guest VM. It's vital to know the detailed process of the attack. Gavin Guo will give a detail explanation and an example of how to attack the host system. Besides, v1/v3/v4 are also introduced in the slide.
This document discusses eBPF (extended Berkeley Packet Filter), which allows tracing from the Linux kernel to userspace using BPF programs. It provides an overview of eBPF including extended registers, verification, maps, and probes. Examples are given of using eBPF for tracing functions like kfree_skb() and the C library function malloc. The Berkeley Compiler Collection (BCC) makes it easy to write eBPF programs in C and Python.
bcc/BPF tools - Strategy, current tools, future challengesIO Visor Project
Brendan Gregg discusses the current state and future potential of BPF and BCC tools for observability in Linux. He outlines 18 areas where BPF support has progressed and 16 areas still needing work. Gregg also discusses challenges like dynamic tracing stability, overhead, ease of coding, and developing visualizations. He proposes finishing ports of his old DTrace tools and links to resources on BPF, BCC, and flame graphs.
UrQA is a mobile QA system that collects bug reports from Android apps to provide more detailed information than competitors like BugSense and ACRA. It displays an "event path" for each bug to show the steps to reproduce crashes, while competitors only show basic numbers. UrQA also uses a client-server architecture where the client handles native crashes and sends crash dumps to the server for analysis and storage.
PCD – Process Control Daemon is a light-weight system level process manager for Embedded-Linux based projects (consumer electronics, network devices, etc.).
PCD starts, stops and monitors all the user space processes in the system, in a synchronized manner, using a textual configuration file.
PCD recovers the system in case of errors and provides useful and detailed debug information.
Slides of my talk on Devel::NYTProf and optimizing perl code at YAPC::NA in June 2014. It covers use of NYTProf and outlines a multi-phase approach to optimizing your perl code.
A video of the talk and questions is available at https://www.youtube.com/watch?v=T7EK6RZAnEA&list=UU7y4qaRSb5w2O8cCHOsKZDw
This document discusses Android memory management. It explains that Android does not have swap space and relies on garbage collection. It describes the different memory regions in Android including the Java heap, native heap, and ashmem. It provides tips on increasing heap size, viewing heap updates, and how Android manages memory through processes. It also covers topics like memory leaks, references, and tools for analyzing memory usage.
Controlling Memory Footprint at All Layers: Linux Kernel, Applications, Libra...peknap
Reducing memory usage is well covered in the history of this conference, yet new tricks still do exist. When optimizing memory footprint for an home gateway device, the author found some unexpected places where small changes can save valuable amount of DRAM or Flash space. This talk will visit different areas including - Kernel: fragmentation threshold, page frame reclamation task and atomic memory. Application level: Memory inefficient shared libraries due to ABI compliance and dynamic loading. Toolchain: Tuning malloc allocator parameters and compiler options. System level: General kernel might be more memory efficient than MMU-less uClinux, and preventing lock up when the system is on the brink of running out of memory.
Slides for my Perl Memory Use talk at YAPC::Asia in Tokyo, September 2012.
(This uploaded version includes quite a few slides from the OSCON version that I skipped at YAPC::Asia in order to have more time for a demo.)
This document discusses profiling the AllPaths-LG genome assembler to optimize its performance. It analyzed the CPU and memory usage of each program step on various systems. The profiling identified seven routines that use the most CPU time and I/O. Modules like FindErrors, AlignReads, and CommonPather were prioritized for optimization to reduce assembly time. Future work will involve more detailed profiling and exploring code optimizations for the most resource-intensive modules.
The document discusses Process Control Daemon (PCD), an open source process manager for embedded Linux platforms. PCD aims to improve over traditional shell script-based startup by allowing deterministic, parallel startup and recovery actions for processes. It provides a centralized way to define and manage processes and their dependencies. Key features include event-driven startup, crash handling and logging, and a process API. PCD has modest resource needs and supports various architectures. It has benefited products by improving startup time, robustness, and debug capabilities.
Workshop - Linux Memory Analysis with VolatilityAndrew Case
Slides from my 3 hour workshop at Blackhat Vegas 2011. Covers using Volatility to perform Linux memory analysis investigations as well Linux kernel internals.
Александр Терещук - Memory Analyzer Tool and memory optimization tips in AndroidUA Mobile
The document discusses memory analysis and optimization tips for Android applications. It introduces the Eclipse Memory Analyzer tool, which can be used to analyze memory usage and find memory leaks. Some key points covered include the low RAM sizes on Android devices, how the garbage collector works, how to analyze an object's shallow size and retained size, finding memory dominators, and common memory leak pitfalls like non-static handlers and large bitmaps stored in static fields. The document encourages analyzing heap dumps in the Memory Analyzer tool to optimize memory usage and find leaks.
Slides for my talk at the London Perl Workshop in Nov 2013, featuring the Devel::SizeMe perl module.
See also the screencast at https://archive.org/details/Perl-Memory-Profiling-LPW2013
This document discusses memory management in Android, including:
1) An overview of the Java memory model and garbage collection process in Android, including the heap structure and garbage collector.
2) The java.lang.ref package which contains classes like WeakReference and SoftReference that are used in memory management.
3) Tips for optimizing memory usage in Android such as limiting heap size, using large heap configuration, and analyzing garbage collection logs.
This document provides an overview of Linux memory management concepts including:
- RAM usage and primary vs secondary memory
- Memory mapping, process address spaces, and segmentation
- Pages, frames, page tables, and virtual memory
- Memory nodes, zones, and NUMA concepts
- Kernel memory allocation, page faults, and troubleshooting memory issues
Linux memory consumption - Why memory utilities show a little amount of free RAM? How does Linux kernel utilizes free RAM? What is the real amount of free RAM in the system?
The Linux kernel tracks each process's memory usage through data structures stored in the process's task_struct. The mm_struct stored there contains pointers to vm_area_struct objects representing each memory mapping. When a process calls malloc(), the kernel allocates physical pages and updates the process's mm_struct and vm_area_structs to map the new memory region into its virtual address space. Similarly when a process forks, the child process inherits copies of the parent's mm_struct and vm_area_structs, giving it the same memory mappings while keeping the two processes' memory private.
This document provides information about Pythian, a company that provides database management and consulting services. It begins by introducing the presenter, Christo Kutrovsky, and his background. It then provides details about Pythian, including that it was founded in 1997, has over 200 employees, 200 customers worldwide, and 5 offices globally. It notes Pythian's partnerships and awards. The document emphasizes Pythian's expertise in Oracle, SQL Server, and other technologies. It positions Pythian as a recognized leader in database management.
Slides from Android Builder's Summit 2014 in San Jose, CA
The 4.4 KitKat release includes the results of “Project Svelte”: a set of tweaks to the operating system to make it run more easily on devices with around 512 MiB RAM. This is likely to be especially important for people working with “Embedded Android”, that is, implementing Android on devices that are not smart phones or tablets.
This document discusses various techniques for optimizing Android for low-RAM devices, including:
1. Tuning apps to release memory more aggressively, the Dalvik VM to use less memory, and the ActivityManager to better manage processes.
2. Configuring Dalvik VM properties like heap sizes and disabling JIT compilation to reduce memory usage.
3. Enabling kernel features like KSM for memory sharing and adjusting lowmemkiller parameters to more aggressively free memory.
4. Using tools like dumpsys, procrank, and meminfo to monitor memory usage and identify optimization opportunities.
Как Linux работает с памятью — Вячеслав БирюковYandex
Поговорим о том, как Linux считает память и какие есть виды памяти. Проведём обзор средств и утилит. Рассмотрим, зачем нужен page cache и как он помогает системе, а также способы ограничения памяти для приложений.
The document provides best practices for optimizing Android app performance related to memory management. It discusses how Android manages memory using paging and memory mapping. It also outlines several things app developers should do to manage memory efficiently in their apps, such as releasing memory when the UI is hidden, using services sparingly, and releasing resources as overall device memory becomes tight. The document provides examples and explanations for each recommendation to help developers understand how to optimize their app's memory usage.
This document discusses the Linux tracing tool systemtap. It provides an overview of systemtap and what it can be used for, including tracing system calls, kernel functions, and application functions. It also discusses how systemtap works, how it uses debugging symbols, and how RPMs handle separate debug information files. Several examples are given of using systemtap probes to trace requests for Nginx, cURL, Redis, MySQL, and TCP retransmissions. The document concludes by mentioning using DTrace for other languages beyond C, such as MySQL, Python, and Java.
This document discusses the Linux tracing tool systemtap. It provides an overview of systemtap and what it can be used for, including tracing system calls, kernel functions, and application functions. It also discusses how systemtap works, how it uses debugging symbols, and how RPMs handle separate debug information files. Several examples are given of using systemtap probes to trace requests for Nginx, cURL, Redis, MySQL, and TCP retransmissions. The document suggests systemtap can be used beyond C for tracing languages like MySQL, Python and Java.
The document discusses reverse engineering the firmware of Swisscom's Centro Grande modems. It identifies several vulnerabilities found, including a command overflow issue that allows complete control of the device by exceeding the input buffer, and multiple buffer overflow issues that can be exploited to execute code remotely by crafting specially formatted XML files. Details are provided on the exploitation techniques and timeline of coordination with Swisscom to address the vulnerabilities.
OSSNA 2017 Performance Analysis Superpowers with Linux BPFBrendan Gregg
Talk by Brendan Gregg for OSSNA 2017. "Advanced performance observability and debugging have arrived built into the Linux 4.x series, thanks to enhancements to Berkeley Packet Filter (BPF, or eBPF) and the repurposing of its sandboxed virtual machine to provide programmatic capabilities to system tracing. Netflix has been investigating its use for new observability tools, monitoring, security uses, and more. This talk will be a dive deep on these new tracing, observability, and debugging capabilities, which sooner or later will be available to everyone who uses Linux. Whether you’re doing analysis over an ssh session, or via a monitoring GUI, BPF can be used to provide an efficient, custom, and deep level of detail into system and application performance.
This talk will also demonstrate the new open source tools that have been developed, which make use of kernel- and user-level dynamic tracing (kprobes and uprobes), and kernel- and user-level static tracing (tracepoints). These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and a whole lot more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations."
PGCon 2014 - What Do You Mean my Database Server Core Dumped? - How to Inspec...Faisal Akber
Presented at PGCon 2014 in Ottawa.
Program crashes are a fact of life and occasionally unavoidable. If there are core dumps that get generated then understanding what happened becomes easier.
This document provides information on various debugging and profiling tools that can be used for Ruby including:
- lsof to list open files for a process
- strace to trace system calls and signals
- tcpdump to dump network traffic
- google perftools profiler for CPU profiling
- pprof to analyze profiling data
It also discusses how some of these tools have helped identify specific performance issues with Ruby like excessive calls to sigprocmask and memcpy calls slowing down EventMachine with threads.
The document provides instructions for setting up a TI-RTOS project for the CC1352R wireless microcontroller. It describes creating a CCS project targeting the CC1352R, configuring compiler and linker settings, generating a system configuration file, and adding TI-RTOS and driver library files. The goal is to build a basic "hello world" project to demonstrate real-time operating system functionality on the CC1352R wireless microcontroller.
Linux 4.x Tracing: Performance Analysis with bcc/BPFBrendan Gregg
Talk about bcc/eBPF for SCALE15x (2017) by Brendan Gregg. "BPF (Berkeley Packet Filter) has been enhanced in the Linux 4.x series and now powers a large collection of performance analysis and observability tools ready for you to use, included in the bcc (BPF Complier Collection) open source project. BPF nowadays can do system tracing, software defined networks, and kernel fast path: much more than just filtering packets! This talk will focus on the bcc/BPF tools for performance analysis, which make use of other built in Linux capabilities: dynamic tracing (kprobes and uprobes) and static tracing (tracepoints and USDT). There are now bcc tools for measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more. These lead to performance wins large and small, especially when instrumenting areas that previously had zero visibility. Tracing superpowers have finally arrived, built in to Linux."
The document describes how to debug a kernel crash by recording the full kernel panic text using techniques like configuring a serial console, using the netconsole kernel feature, or manually dumping memory on a virtual machine. It also explains how to use the crash analysis tool to examine the crash dump, including getting a backtrace, disassembling instructions, and viewing the kernel log.
The document summarizes Maycon Vitali's presentation on hacking embedded devices. It includes an agenda covering extracting firmware from devices using tools like BusPirate and flashrom, decompressing firmware to view file systems and binaries, emulating binaries using QEMU, reverse engineering code to find vulnerabilities, and details four vulnerabilities discovered in Ubiquiti networking devices designated as CVEs. The presentation aims to demonstrate common weaknesses in embedded device security and how tools can be used to analyze and hack these ubiquitous connected systems.
This document provides an overview of the Linux kernel, including its history, structure, build process, installation, updating, and customization. It discusses getting the kernel source code, configuring and building the kernel, installing modules and the kernel, applying updates via patches, and determining the correct driver for PCI devices by matching the vendor and device IDs. The key steps are to find the PCI IDs, search for the IDs in kernel headers to identify the driver, search the kernel makefiles and configuration to enable that driver for compilation.
HKG18-TR14 - Postmortem Debugging with CoresightLinaro
Session ID: HKG18-TR14
Session Name: HKG18-TR14 - Postmortem Debugging with Coresight
Speaker: Leo Yan
Track: Training
★ Session Summary ★
For most cases we can easily debug with kernel's oops dumping info, but sometimes we need to know more information for program execution flow before the issue happens. So we can rely on two tracing methods to reproduce the program execution flow, one method is using software tracing which is kernel's pstore method; another method is to rely on Coresight hardware tracing, this method also can avoid extra workload introduced by tracing itself. Coresight has provided two mechanisms for Postmortem debugging, one method is Coresight CPU debug module so we can extract CPU program counter info, this is quite straightforward to debug CPU lockup issue; Another is Coresight panic kdump, we connect kernel kdump mechanism to extract Coresight tracing data so we can reproduce the last execution flow before panic (even hang issue with some tweaking in kernel). This session wants to go through these topics and demonstrate the debugging tools on 96boards Hikey in 25 minutes session.
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/hkg18/hkg18-tr14/
Presentation: http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-tr14.pdf
Video: http://connect.linaro.org.s3.amazonaws.com/hkg18/videos/hkg18-tr14.mp4
---------------------------------------------------
★ Event Details ★
Linaro Connect Hong Kong 2018 (HKG18)
19-23 March 2018
Regal Airport Hotel Hong Kong
---------------------------------------------------
Keyword: Training
'http://www.linaro.org'
'http://connect.linaro.org'
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringNETWAYS
Nowadays system administrators have great choices when it comes down to Linux performance profiling and monitoring. The challenge is to pick the appropriate tools and interpret their results correctly.
This talk is a chance to take a tour through various performance profiling and benchmarking tools, focusing on their benefit for every sysadmin.
More than 25 different tools are presented. Ranging from well known tools like strace, iostat, tcpdump or vmstat to new features like Linux tracepoints or perf_events. You will also learn which tools can be monitored by Icinga and which monitoring plugins are already available for that.
At the end the goal is to gather reference points to look at, whenever you are faced with performance problems.
Take the chance to close your knowledge gaps and learn how to get the most out of your system.
Kernel Recipes 2017: Performance Analysis with BPFBrendan Gregg
Talk by Brendan Gregg at Kernel Recipes 2017 (Paris): "The in-kernel Berkeley Packet Filter (BPF) has been enhanced in recent kernels to do much more than just filtering packets. It can now run user-defined programs on events, such as on tracepoints, kprobes, uprobes, and perf_events, allowing advanced performance analysis tools to be created. These can be used in production as the BPF virtual machine is sandboxed and will reject unsafe code, and are already in use at Netflix.
Beginning with the bpf() syscall in 3.18, enhancements have been added in many kernel versions since, with major features for BPF analysis landing in Linux 4.1, 4.4, 4.7, and 4.9. Specific capabilities these provide include custom in-kernel summaries of metrics, custom latency measurements, and frequency counting kernel and user stack traces on events. One interesting case involves saving stack traces on wake up events, and associating them with the blocked stack trace: so that we can see the blocking stack trace and the waker together, merged in kernel by a BPF program (that particular example is in the kernel as samples/bpf/offwaketime).
This talk will discuss the new BPF capabilities for performance analysis and debugging, and demonstrate the new open source tools that have been developed to use it, many of which are in the Linux Foundation iovisor bcc (BPF Compiler Collection) project. These include tools to analyze the CPU scheduler, TCP performance, file system performance, block I/O, and more."
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...Anne Nicolas
The in-kernel Berkeley Packet Filter (BPF) has been enhanced in recent kernels to do much more than just filtering packets. It can now run user-defined programs on events, such as on tracepoints, kprobes, uprobes, and perf_events, allowing advanced performance analysis tools to be created. These can be used in production as the BPF virtual machine is sandboxed and will reject unsafe code, and are already in use at Netflix.
Beginning with the bpf() syscall in 3.18, enhancements have been added in many kernel versions since, with major features for BPF analysis landing in Linux 4.1, 4.4, 4.7, and 4.9. Specific capabilities these provide include custom in-kernel summaries of metrics, custom latency measurements, and frequency counting kernel and user stack traces on events. One interesting case involves saving stack traces on wake up events, and associating them with the blocked stack trace: so that we can see the blocking stack trace and the waker together, merged in kernel by a BPF program (that particular example is in the kernel as samples/bpf/offwaketime).
This talk will discuss the new BPF capabilities for performance analysis and debugging, and demonstrate the new open source tools that have been developed to use it, many of which are in the Linux Foundation iovisor bcc (BPF Compiler Collection) project. These include tools to analyze the CPU scheduler, TCP performance, file system performance, block I/O, and more.
Brendan Gregg, Netflix
Performance tweaks and tools for Linux (Joe Damato)Ontico
The document discusses various Linux performance analysis tools including lsof to list open files, strace to trace system calls, tcpdump to dump network traffic, perftools from Google for profiling CPU usage, and a Ruby library called perftools.rb for profiling Ruby code. Examples are provided for using these tools to analyze memory usage, slow queries, Ruby interpreter signals, thread scheduling overhead, and identifying hot spots in Ruby web applications.
USENIX ATC 2017 Performance Superpowers with Enhanced BPFBrendan Gregg
Talk for USENIX ATC 2017 by Brendan Gregg
"The Berkeley Packet Filter (BPF) in Linux has been enhanced in very recent versions to do much more than just filter packets, and has become a hot area of operating systems innovation, with much more yet to be discovered. BPF is a sandboxed virtual machine that runs user-level defined programs in kernel context, and is part of many kernels. The Linux enhancements allow it to run custom programs on other events, including kernel- and user-level dynamic tracing (kprobes and uprobes), static tracing (tracepoints), and hardware events. This is finding uses for the generation of new performance analysis tools, network acceleration technologies, and security intrusion detection systems.
This talk will explain the BPF enhancements, then discuss the new performance observability tools that are in use and being created, especially from the BPF compiler collection (bcc) open source project. These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and much more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations.
Because these BPF enhancements are only in very recent Linux (such as Linux 4.9), most companies are not yet running new enough kernels to be exploring BPF yet. This will change in the next year or two, as companies including Netflix upgrade their kernels. This talk will give you a head start on this growing technology, and also discuss areas of future work and unsolved problems."
The document discusses analyzing Linux kernel crash dumps. It covers various ways to gather crash data like serial console, netconsole, kmsg dumpers, Kdump, and Pstore. It then discusses analyzing the crashed kernel using tools like ksymoops, crash utility, and examining the backtrace, kernel logs, processes, and file descriptors. The document provides examples of gathering data from Pstore and using commands like bt, log, and ps with the crash utility to extract information from a crash dump.
6. 6
Crash Client - Samsung’s crash-work-sdk
• Crash process flow
Stage 1: in sys-assert.c (libsys-assert.so)
-> int sig_to_handle[] = { SIGILL, SIGABRT, SIGBUS, SIGFPE, SIGSEGV, };
-> sighandler notify system_server via /opt/share/crash/curbs.log pipeline.
Stage 2: in ss_bs.c (system-server)
-> ecore_file_monitor_add(CRASH_NOTI_PATH,(void *) __crash_file_cb, NULL);
-> __crash_file_cb:
-> launch_crash_worker()
......
launch /usr/bin/crash-worker to generate cs file
launch /usr/apps/org.tizen.crash-popup/bin/crash-popup to popup crash (only 1st)
7. 7
Crash Client - Samsung’s crash-work-sdk
• Crash process flow
Breakpoint 1, launch_app_with_nice (file=0xb46017b0 "/usr/bin/crash-worker", argv=0xbfe52624, pid=0x0, _nice=0)
at /usr/src/debug/system-server-0.1.65/ss_launch.c:140
140 {
(gdb) bt
#0 launch_app_with_nice (file=0xb46017b0 "/usr/bin/crash-worker",
argv=0xbfe52624, pid=0x0, _nice=0)
at /usr/src/debug/system-server-0.1.65/ss_launch.c:140
#1 0x0804d3c9 in launch_app_cmd_with_nice (
cmdline=0xb4601758 "/usr/bin/crash-worker S top 391655492 913 top",
_nice=0) at /usr/src/debug/system-server-0.1.65/ss_launch.c:196
#2 0x0804d744 in ss_launch_evenif_exist (
execpath=0x8060e71 "/usr/bin/crash-worker",
arg=0xbfe53d16 "S top 391655492 913 top")
at /usr/src/debug/system-server-0.1.65/ss_launch.c:289
#3 0x08058b73 in launch_crash_worker (
filename=0xbfe5823c "/opt/share/crash/curbs.log", popup_on=1)
at /usr/src/debug/system-server-0.1.65/ss_bs.c:327
#4 0x08058d47 in __crash_file_cb (data=0x0, em=0x8dabb10,
event=ECORE_FILE_EVENT_MODIFIED,
path=0xbfe5823c "/opt/share/crash/curbs.log")
at /usr/src/debug/system-server-0.1.65/ss_bs.c:374
#5 0xb782f345 in _ecore_file_monitor_inotify_handler ()
from /usr/lib/libecore_file.so.1
#6 0xb7840e5c in _ecore_main_loop_iterate_internal ()
from /usr/lib/libecore.so.1
#7 0xb784141f in ecore_main_loop_begin () from /usr/lib/libecore.so.1
#8 0x0804bd95 in system_main (argc=1, argv=0xbfe59404)
---Type <return> to continue, or q <return> to quit---
at /usr/src/debug/system-server-0.1.65/ss_main.c:102
#9 0x0804bdf1 in elm_main (argc=1, argv=0xbfe59404)
at /usr/src/debug/system-server-0.1.65/ss_main.c:112
#10 0x0804be4e in main (argc=1, argv=0xbfe59404)
at /usr/src/debug/system-server-0.1.65/ss_main.c:119
12. 12
Crash Client - Intel’s corewatcher
• Mechanism.
• How to upload crashes to server.
• Crash report file: /var/lib/corewatcher/processed/*.txt.
13. 13
Crash Client - Intel’s corewatcher
• Mechanism.
• Corewatcher as daemon
• Listen to /var/lib/corewatcher/
• When crash comes, invoke gdb to analysis
• Upload crashes to CrashDB server
• Environment about corewatcher
• /proc/sys/kernel/core_pattern=/var/lib/corewatcher/core_%e_%t
• core_uses_pid=1
bt full
info shared
14. 14
Crash Client - Intel’s corewatcher
• CrashDB server: https://tz.otcshare.org/crashdb/
• How to upload crash to server
• WWLAN(3G/2G)
• WiFi/SED
• crash_submit: http://otcqa.sh.intel.com/wiki/Crash_Submit
Even though tz.otcshare.org has security restriction(403 forbidden outside of
Intel), crash submit is allowed.
15. 15
Crash Client - Intel’s corewatcher
• Crash report path: /var/lib/corewatcher/processed/*.txt
• Crash report content(without debug info):
cmdline: /usr/bin/mate-calc
version: 2.1.0
backtrace: |
#0 0x00007fd494c2db41 in g_logv () from /usr/lib64/libglib-2.0.so.0
#0 0x00007fd494c2db41 in g_logv () from /usr/lib64/libglib-2.0.so.0
#1 0x00007fd494c2dcfd in g_log () from /usr/lib64/libglib-2.0.so.0
#2 0x00007fd4959a10ee in g_settings_set_property () from /usr/lib64/libgio-2.0.so.0
#3 0x00007fd4956ae098 in g_object_constructor () from /usr/lib64/libgobject-2.0.so.0
#4 0x00007fd4956af562 in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
26. 26
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:923
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6
27. 27
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:923
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6
28. 28
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:923
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6
Guilty Function
29. 29
Crash Server – Guilty Function Location
(gdb) f 3
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
367 SEC_SVR_DBG("Server: Cookie created for client PID %d LABEL >%s<",
(gdb) p created_cookie->smack_label
$2 = 0x1777 <Address 0x1777 out of bounds> (gdb) p *created_cookie $4 = {cookie =
"270217{257354063221 e筗Y370230~021024004244", path_len = 49, permission_len = 90, pid =
1562, path = 0x85e0ba8 "/usr/apps/org.tizen.video-player/bin/video-player", permissions = 0x85b6168,
smack_label = 0x1777 <Address 0x1777 out of bounds>, prev = 0x8589190, next = 0x0}
30. 30
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:923
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6
White list
• /lib/libc.so.6
• /usr/lib/libdlog.so.0
31. 31
Crash Server – Guilty Function Location
https://bugs.tizen.org/jira/browse/TIVI-649
'Security-server has closed unexpectedly' popped up when playing videos or launching clock
(gdb) bt
#0 0xb4e9c999 in vfprintf () from /lib/libc.so.6
#1 0xb4f3e7b4 in __vsnprintf_chk () from /lib/libc.so.6
#2 0xb5560c00 in __dlog_print () from /usr/lib/libdlog.so.0
#3 0x081019fd in process_cookie_request (sockfd=27) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:367
#4 0x08103b7e in security_server_thread (param=0xb451519c) at /usr/src/debug/security-server-
0.0.61/src/security-srv/server/security-server-main.c:923
#5 0xb554be19 in start_thread () from /lib/libpthread.so.0 #6 0xb4f2affe in clone () from
/lib/libc.so.6
White list
• /lib/libc.so.6
• /usr/lib/libdlog.so.0
Guilty Function