SlideShare a Scribd company logo
Linaro Test and
Validation Summit
Linaro Engineering Teams
LCE13 - Dublin, July 2013
How Do We Better Test our Engineering
2nd Half
PART 1: Linaro Platform
Overview
LAVA
Citius,Altius,Fortius
"Faster,Higher,Stronger"
Builds&CI
BuildYourCodeWhenYou
areNotLooking
QAServices
Coverallbases
PART 2: Linaro Engineering
KernelDevelopers
&Maintainers
LandingTeams LinaroGroups
LEG/LNG
(DotheyuseCI? Manual orAutomatedTesting?)*Howdotheyvalidate/verifytheiroutput?
*What/howdotheydevelop?
&easiertouse
Linaro Test and
Validation Summit
Mike Holmes
LCE13 - Dublin, July 2013
LNG Engineering
● Create:
Two Linux kernels (with and without RT) and
a yocto filesystem.
● Benchmark:
RT + KVM + HugePage + Dataplane APIs
Required to test kernel and userspace
performance, some tests may be run in both
spaces.
● Platforms:
Arndale, AM335x Starter Kit ? (LSI & TI
boards in future ?) QEMU - versatile Express
?
LNG outputs

Recommended for you

LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)

Resource: LCE13 Name: Test and Validation Summit: Evolution of Testing in Linaro (I) Date: 09-07-2013 Speaker: Video: http://youtu.be/59i6Zblr6cg

2013lce13
Docker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine EvolutionDocker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine Evolution

A meetup talk on the evolution of the Docker engine from 2014-2019, including the refactoring and spin out of OCI runc and CNCF containerd codebases. This talk was given at the Docker London meetup group on Thursday, 31st January, 2019.

dockercontainerscloud native
Performance Profiling Tools and Tricks
Performance Profiling Tools and TricksPerformance Profiling Tools and Tricks
Performance Profiling Tools and Tricks

The document discusses profiling tools and techniques for optimizing performance. It begins with an introduction to profiling and defines it as gathering data on a system's performance metrics like CPU and memory usage to find where time is being spent. It then covers types of profiling like event-based and statistical profiling. The rest of the document provides tips on profiling different languages and frameworks like PHP, Ruby, and Node.js. It also discusses tools like XHProf, XDebug, flamegraphs, and EXPLAIN queries to help optimize databases.

phase2performance testingall things open
● Our code is validated using CI, and
performance trends monitored.
● Our output is verified on one general
purpose ARM platform and against two SoC
vendor platforms via a configurable switch to
allow for dedicated links between nodes
under test.
● Using open source software, we use one
realistic network application, a general
purpose benchmark and five feature specific
test suites.
LNG outputs are verified by
Automated testing is done using
● custom scripts run via jenkins & lava executing
○ (RT) LTP (Real Time Test Tree)
○ (RT) Cyclictest
○ (RT) Hackbench
○ (KVM) virt-test
○ (Hugepage) sysbench OLTP
○ (KVM, Hugepage, RT) openvswitch (Kernel and
userspace)
○ (KVM, Hugepage, RT) netperf
○ Traffic test cases via pcap files and tcpreplay
LNG uses these tools
● We test against three branches
○ linux-lng-tip (development)
○ linux-lng-lsk (bug fixes to stable)
○ linux-lng-lsk-RT (bug fixes to stable RT variant)
● LNG specific CFG fragments
○ KVM (or will this be in a lsk kernel per default?)
○ PREEMPT_RT
○ NO_HZ_FULL (or will this be in a lsk kernel per
default?)
○ HUGEPAGE (is that a CFG option?)
LNG Kernel branches / configuration
● Some of the SoC vendors hardware has up to 16 x
10Gb links, generating this much traffic is non trivial.
● Tests equipment such as IXIA traffic generators are
expensive.
● Test equipment needs to be remotely switched between
the different hardware under test in an automated way
● Scheduling test runs that take days and requires
specific equipment to be dedicated to the task.
LNG unique challenges

Recommended for you

Extended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use casesExtended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use cases

A talk given at FOSDEM 2020 in the containers devroom on the current status of the CNCF containerd project as well as a dive into the ways users are extending and embedding containerd in other platforms and projects.

containerdocikubernetes
BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience

An overview of collaborative effort done by Builds and Baselines, LMG, 96boards and HiKey landing team in getting HiKey integrated into AOSP. Covers work on the AOSP common.git branches, cross kernel/bootloader feature work that provides more form-factor like integration not commonly found on devboards, lessons learned, etc.

linaroconnectopen sourcelinaro
BKK16-307 LHG OE Initiative
BKK16-307 LHG OE InitiativeBKK16-307 LHG OE Initiative
BKK16-307 LHG OE Initiative

The document discusses the Linaro Hardware Group's Open Embedded layer strategy and current status of implementation. The strategy proposes meta layers for integrating established recipes, hosting Linaro recipes, and building reference images. Current status reports successful builds of Weston, multimedia, and Chromium images on Dragonboard 410c and BeagleBone Black. Meta-backports are used to backport components like Wayland from the latest OpenEmbedded release to the current stable release for reference builds.

open sourcelinaroconnectlinaro
● Multiple nodes may be needed to test traffic
interoperability.
● It is not feasible to replicate the test environment at
every developer's desk.
● the applied RT patch even when disabled, alters the
execution paths
● Some test run for 24 hours or more
LNG unique challenges
Questions
○ LAVA is(isn't) working for us
■ Interactive shells in the LAVA environment would
speed debugging given that testing can only be
performed with the test equipment in the lab
■ Multinode testing, with the reservation and
configuration of network switches is required.
■ Long term trends in performance data need to
analysed and compared for regression analysis,
triggering alerts for deviations.
○ Further thoughts on Friday
○ https://lce-13.zerista.
com/event/member/79674
LNG Q&A
Linaro Test and
Validation Summit
Scott Bambrough
LCE13 - Dublin, July 2013
Landing Team Engineering
● Bootloaders
● Linux kernels based on mainline or current
RC's
● Linux kernels based on LSK (expected)
● Ubuntu member builds
● Android member builds
● ALIP member build
Some outputs are public, others confidential.
LT Outputs

Recommended for you

BKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVABKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVA

The session will cover the steps needed to prepare and use new workload for Workload Automation tool. The talk is based on the use case of preparing the ‘applaunch + I/O’ test case for benchmarking I/O schedulers.

open sourcelinarolinaroconnect
Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO

This talk was presented at FOSS ASIA 2018 summit at Singapore on 25th Mar, 2018. https://2018.fossasia.org/event/schedule.html#

rdocentosopenstack
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approach

If you want to provide Kubernetes on your bare-metal infrastructure you need to configure and manage multiple components, like PKI, DHCP, iPXE and some more. Deploying your infrastructure as immutable without SSH access will confront you with some additional challenges, because traditional configuration management tools like Puppet or Ansible require SSH access to each node. Within the scope of this workshop we will present you a practicable way to manage your bare-metal Kubernetes clusters by the use of a CI/CD-driven approach, including Gitlab CI, Terraform, Confd, Vault and some others. As an extra we will keep a fully functional local development setup ready, so you can gain real hands-on experience. The whole configuration is written in code (Infrastructure as Code – IaC) and will automatically update all components the moment you update your configuration. Der komplette Workshop ist hier zu finden: https://github.com/inovex/fluffy-unicorn Speaker: Johannes M. Scheuermann Event: ContainerDays 2018 Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Mehr Tech-Artikel: https://www.inovex.de/blog/

infrastructurekuberneteskubernetes platform
● Kernel code is validated using CI in the
Linaro LAVA Lab, on various member
hardware devices and ARM fast models.
● Our kernel code is also validated in member
LAVA labs on both current and next gen
hardware.
● Our builds at present are a sanity tested by
the LT's but most testing is done by
piggybacking on QA or automated testing
set up by the platform team.
Verification of LT outputs
● Currently run only basic compile boot test + default CI
tests (LTP, powermgmt)
● This needs to change, we want/need to do more
● We need more SoC level tests, having LT's aware of
how to produce tests to run in LAVA will become more
important
LT and kernel tests
1. Much better LAVA documentation
2. Document the tests themselves
3. Infrastructure for testing
4. Infrastructure for better analysis of results
LT & Member Services Needs
● Deployment Guide
○ what are the hardware requirements for a LAB
○ what are the infrastructure requirements for a LAB
○ hardware setup, software installation instructions
● Administrator's Guide
○ basically how Dave Piggot does his job
○ after initial setup, day to day ops and maintenance
Better Documentation

Recommended for you

Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...

The Coccinelle C-program matching and transformation tool was first released in 2008 to facilitate specification and automation in the evolution of Linux kernel code. The novel contribution of Coccinelle is to allow software developers to write code manipulation rules in terms of the code structure itself, via a generalization of the patch syntax. Over the years, Coccinelle has been extensively used in Linux kernel development, resulting in over 6000 commits to the Linux kernel, and has found its place as part of the Linux kernel development process. This talk will review the history of Coccinelle and its impact on the Linux kernel. It will also briefly present two newer tools, prequel and spinfer, that have built on the Coccinelle infrastructure.

linux kerneldevelopmentconference
Let's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for KubernetesLet's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for Kubernetes

A talk given at KubeCon/CloudNativeCon EU in Barcelona, Spain on May 23, 2019. In this talk Phil presented the explosion of OCI-compliant CRI-enabled runtimes that can be used underneath Kubernetes, and demonstrated several of them live.

containerscontainerdcri
Enabling Security via Container Runtimes
Enabling Security via Container RuntimesEnabling Security via Container Runtimes
Enabling Security via Container Runtimes

A talk given at the Google-hosted Container Security Summit on Wednesday, February 12th, 2020 in Seattle, Washington. This talk covered the impact of work done at the lower-level runtimes layer and up through layers like cri-o, containerd, and Docker to bring specific security features to overall platforms like Kubernetes.

runcocikubernetes
● Test Developer's Guide
○ how to integrate tests to be run in lava-test-shell
(lava glue)
○ recommendations on how best to write tests for lava-
test-shell
● User's Guide for lava-test-shell
○ for developers to use lava-test-shell
○ section devoted to using lava-test-shell in workflow
of kernel developer?
Better Documentation
● Impossible to answer the question: What tests are
available in LAVA?
● http://lava-test.readthedocs.org/en/latest/index.html
○ not sufficient, not up to date
○ problem isn't LAVA team, Linaro needs an
acceptance policy on what a test has available
before being used in LAVA
● would like to see meta-data in test documentation that
can be used in test reports
○ in a format that can be used in report generation
Document the tests
● Buddy systems
○ TI LT developed tests that require access to
reference material for comparison
■ video frame captures
■ audio filed
○ TI LT audio/video tests required external box to
capture hdmi/audio output
○ Need to do more of this type of automated testing to
verify that lower level functions work correctly at
BSP level
○ GStreamer insanity test suite requires access to
multimedia content
Infrastructure for Testing
● Web dashboard won't cut it
● need to separate analysis from display
○ rather do an analysis, then decide how to display
● why infrastructure?
○ think there should be a level of reuse for
components used to do analysis
○ think these should be separate from LAVA
○ think of this a more of a data mining operation
Infrastructure for Analysis

Recommended for you

BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2

Introduction to CI v2 for AOSP builds. There has been changes on how we have been doing AOSP development in Linaro. It has changed from daily builds to commit based builds. The session will give overview on how we can do all the builds on ci.linaro.org that we did on android-build.linaro.org.

open sourcelinarolinaroconnect
CRI-containerd
CRI-containerdCRI-containerd
CRI-containerd

This document discusses CRI-Containerd, which is a CRI implementation that allows Kubernetes to use Containerd as its container runtime. It provides an overview of CRI and how it enables Kubernetes to work with different runtimes. It then discusses how Containerd's scope aligns well with CRI's requirements. It introduces CRI-Containerd and its architecture, and notes that it is now feature complete and passing all CRI and Kubernetes tests. Upcoming work includes additional testing, integration with kube-up.sh, and a 1.0.0-beta.0 release by the end of 2017. It concludes with a demo of CRI-Containerd in action.

containerdkubernetes
Kubernetes on CRI-O
Kubernetes on CRI-OKubernetes on CRI-O
Kubernetes on CRI-O

At Bangalore Kubernetes meetup April 2017. This is about running Kubernetes using alternative container runtime cri-o and runc. Event report for the meetup: suraj.io/post/blr-k8s-meetup-april-2017/

containerskubernetescri-o
example:
● generate test report as PDF
○ perform tests, generate a report
○ include metadata regarding tests
■ metadata from test documentation?
example:
● test report comparing:
○ current member BSP kernel
○ current LT kernel based on mainline
● evidence of quality/stability of LT/mainline kernel
● could be used to convince product teams
Infrastructure for Analysis
example:
● regression analysis of kernel changes
○ perform tests one day, make changes, test next
○ did any test results change?
■ yes, send report of changes via email
example:
● generate test report as PDF
○ perform tests, generate a report
○ include metadata regarding tests
■ metadata from test documentation?
Infrastructure for Analysis
example:
● test report comparing:
○ current member BSP kernel
○ current LT kernel based on mainline
● evidence of quality/stability of LT/mainline kernel
● could be used to convince product teams
Infrastructure for Analysis
Linaro Test and
Validation Summit
Kevin Hilman
LCE13 - Dublin, July 2013
Kernel Developer/Maintainer

Recommended for you

The automated tests inside Openshift
The automated tests inside OpenshiftThe automated tests inside Openshift
The automated tests inside Openshift

The document discusses automated testing inside Openshift including the contour of automated testing, dynamic configurations of Openshift, and integration with TestLink. It outlines pros and cons of automated testing using lightweight containers across a variety of operating systems. It describes how to dynamically configure tests by variables like project, branch, product name and version, and OS. It also covers how to create, execute, verify and delete configurations as well as generate reports and artifacts from test runs. Finally, it discusses integrating test results with TestLink for reporting.

robot frameworkdrwebopenshift
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...

rkt and Kubernetes provide container runtimes and orchestration tools to seamlessly update operating systems without affecting application dependencies or uptime. rkt is a modern, secure container runtime that implements open standards and has a simple, modular architecture. It can be used as the container runtime for Kubernetes (rktnetes) or to run Kubernetes components directly. Both tools use the Container Networking Interface (CNI) plugin-based model for networking, allowing IP addresses to be assigned at the pod level. Integration between rkt and Kubernetes continues to improve to support features like TPM attestation and more seamless kubelet upgrades.

netwaysconferencedata center
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)

This document summarizes discussions from the Linaro Test and Validation Summit in Dublin regarding improving engineering testing at Linaro. It describes Linaro's continuous integration (CI) process and tools like LAVA, and plans to enhance testing capabilities. Key points include: 1. Fathi Boudra provided an overview of Linaro's CI loop using tools like Jenkins and LAVA, and future plans to improve the CI process and LAVA. 2. Milosz Wasilewski discussed QA services tasks like manual testing and dashboard monitoring, and ideas to streamline these processes. 3. Antonio Terceiro outlined improvements to LAVA like added monitoring, packaging, documentation and a test helper tool. He

2013lce13
Most kernel development is done with little or
no automation
● build: local, custom build scripts
● boot: manual boot testing on local hardware
● debug: custom unit-test scripts, manual verification of
results
● publish: to public mailing lists
● merged: into maintainer trees, linux-next
● test: manual test of maintainer trees, linux-next
○ but many (most?) developers don't do this
Current workflow: development
● Code review on mailing list
● build/boot testing by maintainers
● build testing in linux-next (manual)
○ several developers do manual build tests of their pet
platforms in linux-next and report failures
● Intel's 0-day tester (automated, but closed)
○ regular, automatic build tests
○ multi-arch build tests
○ boot tests (x86)
○ automatic git bisect for failures
○ very fast results
○ detailed email reports
○ extremely useful
Current workflow: validation
This model is "good enough" for most
developers and maintainers, so...
Why should we use Jenkins/LAVA?
Linaro test/validation will have to be
● at least as easy to use (locally and remotely)
● output/results more useful
● faster
○ build time
○ diagnostic time
Current workflow: "good enough"
● Local testing: aid in build, boot, test cycle
○ local LAVA install, using local boards
○ reduce duplication of custom scripts/setup
○ encourage writing LAVA-ready tests
○ easy to switch between local, and remote LAVA lab
● Remote CI: broader coverage
○ "I'm about ready to push this, I wonder if broke any
other platforms..."
○ automatic, fast (ish) response
Potential Usage models

Recommended for you

SFO15-110: Toolchain Collaboration
SFO15-110: Toolchain CollaborationSFO15-110: Toolchain Collaboration
SFO15-110: Toolchain Collaboration

SFO15-110: Toolchain Collaboration Speaker: Ryan Arnold Date: September 21, 2015 ★ Session Description ★ Linaro and its members discuss the work they are doing in the GNU & LLVM Toolchains for ARM processors. Work done in the previous six months will be discussed, and also discussions about the priorities each member is looking at for the next six months. ★ Resources ★ Video: https://www.youtube.com/watch?v=3BYl-1wGZg4 Presentation: http://www.slideshare.net/linaroorg/sfo15110-toolchain-collaboration Etherpad: pad.linaro.org/p/sfo15-110 Pathable: https://sfo15.pathable.com/meetings/302660 ★ Event Details ★ Linaro Connect San Francisco 2015 - #SFO15 September 21-25, 2015 Hyatt Regency Hotel http://www.linaro.org http://connect.linaro.org

linaroconnectsfo15
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMGLAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG

LMG's finished and upcoming projects include: - Memory allocator and file system analyses to reduce memory usage on low-RAM devices. - Monthly LCR releases and migrating their builds to ci.linaro.org. - Updating toolchains and enabling new hardware like the HiKey board in AOSP. - Increasing participation in upstream projects like merging an SystemUI patch. - Integrating features in AOSP like Energy Aware Scheduling, OP-TEE, and an Overlay Manager. - Continuing work on the HiKey board in AOSP including new features, fixes, and upstreaming components.

linaroconnectlas16
LCU14 303- Toolchain Collaboration
LCU14 303- Toolchain CollaborationLCU14 303- Toolchain Collaboration
LCU14 303- Toolchain Collaboration

★ Session Summary ★ This session will be working through the planned open source contributions from Linaro, ARM, and other members who want to share their open source contribution plans for the next year. Projects to be included are: gcc, llvm, glibc, gold, gdb, binutils. --------------------------------------------------- ★ Resources ★ Zerista: http://lcu14.zerista.com/event/member/137749 Google Event: https://plus.google.com/u/0/events/c3knobs1t2fd2vi9f9mhejehts0 Video: https://www.youtube.com/watch?v=b-mtKxOm0m8&list=UUIVqQKxCyQLJS6xvSmfndLA Etherpad: http://pad.linaro.org/p/lcu14-303 --------------------------------------------------- ★ Event Details ★ Linaro Connect USA - #LCU14 September 15-19th, 2014 Hyatt Regency San Francisco Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org

lcu14
● Has to be easy to install
○ packaged (deb, rpm)
○ or git repo for development (bzr is ......)
● Has to fit into existing developer work flow
○ LAVA does not exclusively own hardware
○ developers have non-Linaro platforms
○ command-line driven
○ must co-exist with existing interactive use of boards
■ existing Apache setup
■ existing TFTP setup
■ existing, customized bootloaders
■ ...
Local testing: LAVA
● Broad testing
● multi-arch (not just ARM)
● ARM: all defconfigs (not just Linaro boards)
○ also: allnoconfig, allmodconfig, randconfig, ...
● Continuous builds
○ Linus' tree, linux-next, arm-soc/for-next, ...
○ developers can submit their own branches
● On-demand builds
○ register a tree/branch
○ push triggers a build
● fast, automatic reporting of failures
○ without manual monitoring/clicking through jenkins
Remote CI
Tracking build breakage in upstream trees
● when did build start breaking
● what are the exact build error messages
(without Jenkins click fest)
● which commit (probably) broke the build
○ automated bisect
Useful output: build testing
Where is the line between Jenkins and LAVA?
● Jenkins == build, LAVA == test?
● when a LAVA test fails how do I know...
○ was this a new/updated test?
○ was this a new/updated kernel?
○ if so, can I get to the Jenkins build?
In less than 10 clicks?
Issues: Big picture

Recommended for you

BKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcherBKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcher

1. The presentation discussed migrating LAVA tests to a new dispatcher design with an explicit pipeline structure. 2. It outlined what the new LAVA dispatcher is capable of, encouraged redesigning existing tests to work with the new framework, and detailed timelines for migration and removal of the old dispatcher. 3. Advice was given on working with the new dispatcher including being explicit about deployment, boot, and test details in job definitions.

open sourcelinarolinaroconnect
BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?

Linaro has multiple labs and board farms with varying purposes. This session will start with an overview of each of these, their locations, their focus, etc. It will then provide examples and direction on how a Member can add their hardware to a board farm. It will also provide an overview of how a Member or employee can navigate/leverage/check out a board for experimentation and usage (this varies based upon which lab/board farm is being considered, so all will be reviewed) in each of the farm locations. Finally, the session will provide pointers to any respective documentation, user guides, etc. for each of the locations.

linaroconnectopen sourcelinaro
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift

This document discusses implementing DevOps with Python using Ansible. It provides an agenda for the presentation including discussing DevOps hotspots, infrastructure as code with Ansible, continuous integration/continuous delivery (CI/CD) using TravisCI and CircleCI, and an open discussion on monitoring and automated tests. It then covers problems commonly faced, how DevOps solves these problems, and the expected benefits of adopting a DevOps culture including standardized environments, infrastructure as code, automated delivery, monitoring, and improved collaboration. It provides an overview of Ansible concepts like inventories, ad-hoc commands, modules, playbooks, roles, and templates. It also demonstrates writing a custom Python module for Ansible and using it in a playbook. Finally, it

ansibleautomationcd
● "Master image" is not useful
○ LAVA assumes are powered on and running master
image (or will reboot into master image)
○ assumptions about SD card existence, partitioning...
○ assumptions about shell prompts
linaro-test [rc=0] #
○ etc. etc.
● Goal: LAVA directly controls bootloader
○ netboot: get kernel + DTB + initrd via TFTP
○ extension via board-specific bootloader scripting
Tyler's new "bootloader" device support in LAVA
appears to have mostly solved this !!
Issues: LAVA design
● Terminology learning curve
○ dispatcher, scheduler, dashboard
○ device, device-type
○ What is a bundle?
○ WTF is a bundle stream?
○ Documentation... not helpful (enough said)
● Navigation
○ click intensive
○ how to get from a log to the test results? or...
○ from a test back to the boot log?
○ what about build log (Jenkins?)
○ can I navigate from Jenkins log to the LAVA test?
Issues: LAVA usability
Kernel + modules: omap2plus_defconfig
● 1 minute
○ hackbox.linaro.org (-j48: 12 x 3.5GHz Xeon, 24G)
● 1.5 minutes
○ khilman local (-j24: 6 x 3.3GHz i7, 16G RAM)
● 8 minutes
○ Macbook Air (-j8: 2 x 1.8GHz i7, 4G)
● 14 minutes
○ Thinkpad T61 (-j4: 2 x Core2Duo, 4G RAM)
● 16 minutes
○ Linaro Jenkins (-j8: EC2 node, built in tmpfs)
● 17 minutes
○ ARM chromebook (-j4: 2 x 1.7 GHz A15, 2G RAM)
Issues: Jenkins performance
Linaro Test and
Validation Summit
Grant Likely
LCE13 - Dublin, July 2013
LEG Engineering

Recommended for you

Building a Small Datacenter
Building a Small DatacenterBuilding a Small Datacenter
Building a Small Datacenter

1. The document discusses building a small data center network for Dyn, focusing on lessons learned from two design iterations. 2. The first design used MPLS VPNs but had issues with routing and IPv6 support. The second design used virtual routing and forwarding with multiple routing tables to separate traffic and improve service mobility. 3. Key lessons included validating designs before deploying, automating network operations, and moving security policies to instances rather than the network to improve agility and isolate impacts.

datacenter cooling power
Cognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & TricksCognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & Tricks

Learn how to improve the performance of your Cognos environment. We cover hardware and server specifics, architecture setup, dispatcher tuning, report specific tuning including the Interactive Performance Assistant and more. See the recording and download this deck: https://senturus.com/resources/cognos-analytics-performance-tuning/ Senturus offers a full spectrum of services for business analytics. Our Knowledge Center has hundreds of free live and recorded webinars, blog posts, demos and unbiased product reviews available on our website at: https://senturus.com/resources/

interactive performance assistcognoscognos tuning
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC

Building a Small DC, by Karl Brumund. A presentation given at APRICOT 2016’s Network Infrastructure session on 24 February 2016.

apricot2016apnicnetwork infrastructure
LCE13: Test and Validation Mini-Summit: Review Current Linaro Engineering Processes

More Related Content

What's hot

BKK16-306 ART ii
BKK16-306 ART iiBKK16-306 ART ii
BKK16-306 ART ii
Linaro
 
Comparison of existing cni plugins for kubernetes
Comparison of existing cni plugins for kubernetesComparison of existing cni plugins for kubernetes
Comparison of existing cni plugins for kubernetes
Adam Hamsik
 
OpenDataPlane Testing in Travis
OpenDataPlane Testing in TravisOpenDataPlane Testing in Travis
OpenDataPlane Testing in Travis
Dmitry Baryshkov
 
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
Linaro
 
Docker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine EvolutionDocker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine Evolution
Phil Estes
 
Performance Profiling Tools and Tricks
Performance Profiling Tools and TricksPerformance Profiling Tools and Tricks
Performance Profiling Tools and Tricks
Phase2
 
Extended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use casesExtended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use cases
Phil Estes
 
BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience
Linaro
 
BKK16-307 LHG OE Initiative
BKK16-307 LHG OE InitiativeBKK16-307 LHG OE Initiative
BKK16-307 LHG OE Initiative
Linaro
 
BKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVABKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVA
Linaro
 
Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO
Chandan Kumar
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approach
inovex GmbH
 
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Anne Nicolas
 
Let's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for KubernetesLet's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for Kubernetes
Phil Estes
 
Enabling Security via Container Runtimes
Enabling Security via Container RuntimesEnabling Security via Container Runtimes
Enabling Security via Container Runtimes
Phil Estes
 
BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2
Linaro
 
CRI-containerd
CRI-containerdCRI-containerd
CRI-containerd
Moby Project
 
Kubernetes on CRI-O
Kubernetes on CRI-OKubernetes on CRI-O
Kubernetes on CRI-O
Suraj Deshmukh
 
The automated tests inside Openshift
The automated tests inside OpenshiftThe automated tests inside Openshift
The automated tests inside Openshift
Oleg Popov
 
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
NETWAYS
 

What's hot (20)

BKK16-306 ART ii
BKK16-306 ART iiBKK16-306 ART ii
BKK16-306 ART ii
 
Comparison of existing cni plugins for kubernetes
Comparison of existing cni plugins for kubernetesComparison of existing cni plugins for kubernetes
Comparison of existing cni plugins for kubernetes
 
OpenDataPlane Testing in Travis
OpenDataPlane Testing in TravisOpenDataPlane Testing in Travis
OpenDataPlane Testing in Travis
 
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (I)
 
Docker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine EvolutionDocker London Meetup: Docker Engine Evolution
Docker London Meetup: Docker Engine Evolution
 
Performance Profiling Tools and Tricks
Performance Profiling Tools and TricksPerformance Profiling Tools and Tricks
Performance Profiling Tools and Tricks
 
Extended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use casesExtended and embedding: containerd update & project use cases
Extended and embedding: containerd update & project use cases
 
BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience BKK16-310 The HiKey AOSP collaborative experience
BKK16-310 The HiKey AOSP collaborative experience
 
BKK16-307 LHG OE Initiative
BKK16-307 LHG OE InitiativeBKK16-307 LHG OE Initiative
BKK16-307 LHG OE Initiative
 
BKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVABKK16-102 Creating new workload for Workload Automation & using WA with LAVA
BKK16-102 Creating new workload for Workload Automation & using WA with LAVA
 
Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO Delivering a bleeding edge community-led openstack distribution: RDO
Delivering a bleeding edge community-led openstack distribution: RDO
 
Manage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approachManage your bare-metal infrastructure with a CI/CD-driven approach
Manage your bare-metal infrastructure with a CI/CD-driven approach
 
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
Kernel Recipes 2018 - 10 years of automated evolution in the Linux kernel - J...
 
Let's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for KubernetesLet's Try Every CRI Runtime Available for Kubernetes
Let's Try Every CRI Runtime Available for Kubernetes
 
Enabling Security via Container Runtimes
Enabling Security via Container RuntimesEnabling Security via Container Runtimes
Enabling Security via Container Runtimes
 
BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2BKK16-507 AOSP builds of Linaro with CI v2
BKK16-507 AOSP builds of Linaro with CI v2
 
CRI-containerd
CRI-containerdCRI-containerd
CRI-containerd
 
Kubernetes on CRI-O
Kubernetes on CRI-OKubernetes on CRI-O
Kubernetes on CRI-O
 
The automated tests inside Openshift
The automated tests inside OpenshiftThe automated tests inside Openshift
The automated tests inside Openshift
 
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
 

Similar to LCE13: Test and Validation Mini-Summit: Review Current Linaro Engineering Processes

LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
Linaro
 
SFO15-110: Toolchain Collaboration
SFO15-110: Toolchain CollaborationSFO15-110: Toolchain Collaboration
SFO15-110: Toolchain Collaboration
Linaro
 
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMGLAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG
Linaro
 
LCU14 303- Toolchain Collaboration
LCU14 303- Toolchain CollaborationLCU14 303- Toolchain Collaboration
LCU14 303- Toolchain Collaboration
Linaro
 
BKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcherBKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcher
Linaro
 
BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?
Linaro
 
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
Yaniv cohen
 
Building a Small Datacenter
Building a Small DatacenterBuilding a Small Datacenter
Building a Small Datacenter
ssuser4b98f0
 
Cognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & TricksCognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & Tricks
Senturus
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
APNIC
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
smalltown
 
Expedia 3x3 presentation
Expedia 3x3 presentationExpedia 3x3 presentation
Expedia 3x3 presentation
Drew Hannay
 
HKG18-TR12 - LAVA for LITE Platforms and Tests
HKG18-TR12 - LAVA for LITE Platforms and TestsHKG18-TR12 - LAVA for LITE Platforms and Tests
HKG18-TR12 - LAVA for LITE Platforms and Tests
Linaro
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & Logging
Marian Marinov
 
LCA13: LAVA and CI Component Review
LCA13: LAVA and CI Component ReviewLCA13: LAVA and CI Component Review
LCA13: LAVA and CI Component Review
Linaro
 
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
Neil Armstrong
 
20141111_SOS3_Gallo
20141111_SOS3_Gallo20141111_SOS3_Gallo
20141111_SOS3_Gallo
Andrea Gallo
 
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205
Linaro
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
confluent
 
BUD17-405: Building a reference IoT product with Zephyr
BUD17-405: Building a reference IoT product with Zephyr BUD17-405: Building a reference IoT product with Zephyr
BUD17-405: Building a reference IoT product with Zephyr
Linaro
 

Similar to LCE13: Test and Validation Mini-Summit: Review Current Linaro Engineering Processes (20)

LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)
 
SFO15-110: Toolchain Collaboration
SFO15-110: Toolchain CollaborationSFO15-110: Toolchain Collaboration
SFO15-110: Toolchain Collaboration
 
LAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMGLAS16-209: Finished and Upcoming Projects in LMG
LAS16-209: Finished and Upcoming Projects in LMG
 
LCU14 303- Toolchain Collaboration
LCU14 303- Toolchain CollaborationLCU14 303- Toolchain Collaboration
LCU14 303- Toolchain Collaboration
 
BKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcherBKK16-210 Migrating to the new dispatcher
BKK16-210 Migrating to the new dispatcher
 
BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?BKK16-213 Where's the Hardware?
BKK16-213 Where's the Hardware?
 
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
 
Building a Small Datacenter
Building a Small DatacenterBuilding a Small Datacenter
Building a Small Datacenter
 
Cognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & TricksCognos Performance Tuning Tips & Tricks
Cognos Performance Tuning Tips & Tricks
 
Building a Small DC
Building a Small DCBuilding a Small DC
Building a Small DC
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
 
Expedia 3x3 presentation
Expedia 3x3 presentationExpedia 3x3 presentation
Expedia 3x3 presentation
 
HKG18-TR12 - LAVA for LITE Platforms and Tests
HKG18-TR12 - LAVA for LITE Platforms and TestsHKG18-TR12 - LAVA for LITE Platforms and Tests
HKG18-TR12 - LAVA for LITE Platforms and Tests
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & Logging
 
LCA13: LAVA and CI Component Review
LCA13: LAVA and CI Component ReviewLCA13: LAVA and CI Component Review
LCA13: LAVA and CI Component Review
 
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
 
20141111_SOS3_Gallo
20141111_SOS3_Gallo20141111_SOS3_Gallo
20141111_SOS3_Gallo
 
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
 
BUD17-405: Building a reference IoT product with Zephyr
BUD17-405: Building a reference IoT product with Zephyr BUD17-405: Building a reference IoT product with Zephyr
BUD17-405: Building a reference IoT product with Zephyr
 

More from Linaro

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Linaro
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Linaro
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Linaro
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
Linaro
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
Linaro
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
Linaro
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
Linaro
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Linaro
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Linaro
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
Linaro
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
Linaro
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
Linaro
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
Linaro
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
Linaro
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
Linaro
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
Linaro
 

More from Linaro (20)

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
 

Recently uploaded

BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
Larry Smarr
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
Aurora Consulting
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
Matthew Sinclair
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Chris Swan
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Bert Blevins
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
DealBook of Ukraine: 2024 edition
DealBook of Ukraine: 2024 editionDealBook of Ukraine: 2024 edition
DealBook of Ukraine: 2024 edition
Yevgen Sysoyev
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
Larry Smarr
 
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALLBLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
Liveplex
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
Safe Software
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024
Stephanie Beckett
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
Matthew Sinclair
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
Stephanie Beckett
 

Recently uploaded (20)

BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
DealBook of Ukraine: 2024 edition
DealBook of Ukraine: 2024 editionDealBook of Ukraine: 2024 edition
DealBook of Ukraine: 2024 edition
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
 
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALLBLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024What’s New in Teams Calling, Meetings and Devices May 2024
What’s New in Teams Calling, Meetings and Devices May 2024
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
 

LCE13: Test and Validation Mini-Summit: Review Current Linaro Engineering Processes

  • 1. Linaro Test and Validation Summit Linaro Engineering Teams LCE13 - Dublin, July 2013 How Do We Better Test our Engineering 2nd Half
  • 2. PART 1: Linaro Platform Overview LAVA Citius,Altius,Fortius "Faster,Higher,Stronger" Builds&CI BuildYourCodeWhenYou areNotLooking QAServices Coverallbases PART 2: Linaro Engineering KernelDevelopers &Maintainers LandingTeams LinaroGroups LEG/LNG (DotheyuseCI? Manual orAutomatedTesting?)*Howdotheyvalidate/verifytheiroutput? *What/howdotheydevelop? &easiertouse
  • 3. Linaro Test and Validation Summit Mike Holmes LCE13 - Dublin, July 2013 LNG Engineering
  • 4. ● Create: Two Linux kernels (with and without RT) and a yocto filesystem. ● Benchmark: RT + KVM + HugePage + Dataplane APIs Required to test kernel and userspace performance, some tests may be run in both spaces. ● Platforms: Arndale, AM335x Starter Kit ? (LSI & TI boards in future ?) QEMU - versatile Express ? LNG outputs
  • 5. ● Our code is validated using CI, and performance trends monitored. ● Our output is verified on one general purpose ARM platform and against two SoC vendor platforms via a configurable switch to allow for dedicated links between nodes under test. ● Using open source software, we use one realistic network application, a general purpose benchmark and five feature specific test suites. LNG outputs are verified by
  • 6. Automated testing is done using ● custom scripts run via jenkins & lava executing ○ (RT) LTP (Real Time Test Tree) ○ (RT) Cyclictest ○ (RT) Hackbench ○ (KVM) virt-test ○ (Hugepage) sysbench OLTP ○ (KVM, Hugepage, RT) openvswitch (Kernel and userspace) ○ (KVM, Hugepage, RT) netperf ○ Traffic test cases via pcap files and tcpreplay LNG uses these tools
  • 7. ● We test against three branches ○ linux-lng-tip (development) ○ linux-lng-lsk (bug fixes to stable) ○ linux-lng-lsk-RT (bug fixes to stable RT variant) ● LNG specific CFG fragments ○ KVM (or will this be in a lsk kernel per default?) ○ PREEMPT_RT ○ NO_HZ_FULL (or will this be in a lsk kernel per default?) ○ HUGEPAGE (is that a CFG option?) LNG Kernel branches / configuration
  • 8. ● Some of the SoC vendors hardware has up to 16 x 10Gb links, generating this much traffic is non trivial. ● Tests equipment such as IXIA traffic generators are expensive. ● Test equipment needs to be remotely switched between the different hardware under test in an automated way ● Scheduling test runs that take days and requires specific equipment to be dedicated to the task. LNG unique challenges
  • 9. ● Multiple nodes may be needed to test traffic interoperability. ● It is not feasible to replicate the test environment at every developer's desk. ● the applied RT patch even when disabled, alters the execution paths ● Some test run for 24 hours or more LNG unique challenges
  • 10. Questions ○ LAVA is(isn't) working for us ■ Interactive shells in the LAVA environment would speed debugging given that testing can only be performed with the test equipment in the lab ■ Multinode testing, with the reservation and configuration of network switches is required. ■ Long term trends in performance data need to analysed and compared for regression analysis, triggering alerts for deviations. ○ Further thoughts on Friday ○ https://lce-13.zerista. com/event/member/79674 LNG Q&A
  • 11. Linaro Test and Validation Summit Scott Bambrough LCE13 - Dublin, July 2013 Landing Team Engineering
  • 12. ● Bootloaders ● Linux kernels based on mainline or current RC's ● Linux kernels based on LSK (expected) ● Ubuntu member builds ● Android member builds ● ALIP member build Some outputs are public, others confidential. LT Outputs
  • 13. ● Kernel code is validated using CI in the Linaro LAVA Lab, on various member hardware devices and ARM fast models. ● Our kernel code is also validated in member LAVA labs on both current and next gen hardware. ● Our builds at present are a sanity tested by the LT's but most testing is done by piggybacking on QA or automated testing set up by the platform team. Verification of LT outputs
  • 14. ● Currently run only basic compile boot test + default CI tests (LTP, powermgmt) ● This needs to change, we want/need to do more ● We need more SoC level tests, having LT's aware of how to produce tests to run in LAVA will become more important LT and kernel tests
  • 15. 1. Much better LAVA documentation 2. Document the tests themselves 3. Infrastructure for testing 4. Infrastructure for better analysis of results LT & Member Services Needs
  • 16. ● Deployment Guide ○ what are the hardware requirements for a LAB ○ what are the infrastructure requirements for a LAB ○ hardware setup, software installation instructions ● Administrator's Guide ○ basically how Dave Piggot does his job ○ after initial setup, day to day ops and maintenance Better Documentation
  • 17. ● Test Developer's Guide ○ how to integrate tests to be run in lava-test-shell (lava glue) ○ recommendations on how best to write tests for lava- test-shell ● User's Guide for lava-test-shell ○ for developers to use lava-test-shell ○ section devoted to using lava-test-shell in workflow of kernel developer? Better Documentation
  • 18. ● Impossible to answer the question: What tests are available in LAVA? ● http://lava-test.readthedocs.org/en/latest/index.html ○ not sufficient, not up to date ○ problem isn't LAVA team, Linaro needs an acceptance policy on what a test has available before being used in LAVA ● would like to see meta-data in test documentation that can be used in test reports ○ in a format that can be used in report generation Document the tests
  • 19. ● Buddy systems ○ TI LT developed tests that require access to reference material for comparison ■ video frame captures ■ audio filed ○ TI LT audio/video tests required external box to capture hdmi/audio output ○ Need to do more of this type of automated testing to verify that lower level functions work correctly at BSP level ○ GStreamer insanity test suite requires access to multimedia content Infrastructure for Testing
  • 20. ● Web dashboard won't cut it ● need to separate analysis from display ○ rather do an analysis, then decide how to display ● why infrastructure? ○ think there should be a level of reuse for components used to do analysis ○ think these should be separate from LAVA ○ think of this a more of a data mining operation Infrastructure for Analysis
  • 21. example: ● generate test report as PDF ○ perform tests, generate a report ○ include metadata regarding tests ■ metadata from test documentation? example: ● test report comparing: ○ current member BSP kernel ○ current LT kernel based on mainline ● evidence of quality/stability of LT/mainline kernel ● could be used to convince product teams Infrastructure for Analysis
  • 22. example: ● regression analysis of kernel changes ○ perform tests one day, make changes, test next ○ did any test results change? ■ yes, send report of changes via email example: ● generate test report as PDF ○ perform tests, generate a report ○ include metadata regarding tests ■ metadata from test documentation? Infrastructure for Analysis
  • 23. example: ● test report comparing: ○ current member BSP kernel ○ current LT kernel based on mainline ● evidence of quality/stability of LT/mainline kernel ● could be used to convince product teams Infrastructure for Analysis
  • 24. Linaro Test and Validation Summit Kevin Hilman LCE13 - Dublin, July 2013 Kernel Developer/Maintainer
  • 25. Most kernel development is done with little or no automation ● build: local, custom build scripts ● boot: manual boot testing on local hardware ● debug: custom unit-test scripts, manual verification of results ● publish: to public mailing lists ● merged: into maintainer trees, linux-next ● test: manual test of maintainer trees, linux-next ○ but many (most?) developers don't do this Current workflow: development
  • 26. ● Code review on mailing list ● build/boot testing by maintainers ● build testing in linux-next (manual) ○ several developers do manual build tests of their pet platforms in linux-next and report failures ● Intel's 0-day tester (automated, but closed) ○ regular, automatic build tests ○ multi-arch build tests ○ boot tests (x86) ○ automatic git bisect for failures ○ very fast results ○ detailed email reports ○ extremely useful Current workflow: validation
  • 27. This model is "good enough" for most developers and maintainers, so... Why should we use Jenkins/LAVA? Linaro test/validation will have to be ● at least as easy to use (locally and remotely) ● output/results more useful ● faster ○ build time ○ diagnostic time Current workflow: "good enough"
  • 28. ● Local testing: aid in build, boot, test cycle ○ local LAVA install, using local boards ○ reduce duplication of custom scripts/setup ○ encourage writing LAVA-ready tests ○ easy to switch between local, and remote LAVA lab ● Remote CI: broader coverage ○ "I'm about ready to push this, I wonder if broke any other platforms..." ○ automatic, fast (ish) response Potential Usage models
  • 29. ● Has to be easy to install ○ packaged (deb, rpm) ○ or git repo for development (bzr is ......) ● Has to fit into existing developer work flow ○ LAVA does not exclusively own hardware ○ developers have non-Linaro platforms ○ command-line driven ○ must co-exist with existing interactive use of boards ■ existing Apache setup ■ existing TFTP setup ■ existing, customized bootloaders ■ ... Local testing: LAVA
  • 30. ● Broad testing ● multi-arch (not just ARM) ● ARM: all defconfigs (not just Linaro boards) ○ also: allnoconfig, allmodconfig, randconfig, ... ● Continuous builds ○ Linus' tree, linux-next, arm-soc/for-next, ... ○ developers can submit their own branches ● On-demand builds ○ register a tree/branch ○ push triggers a build ● fast, automatic reporting of failures ○ without manual monitoring/clicking through jenkins Remote CI
  • 31. Tracking build breakage in upstream trees ● when did build start breaking ● what are the exact build error messages (without Jenkins click fest) ● which commit (probably) broke the build ○ automated bisect Useful output: build testing
  • 32. Where is the line between Jenkins and LAVA? ● Jenkins == build, LAVA == test? ● when a LAVA test fails how do I know... ○ was this a new/updated test? ○ was this a new/updated kernel? ○ if so, can I get to the Jenkins build? In less than 10 clicks? Issues: Big picture
  • 33. ● "Master image" is not useful ○ LAVA assumes are powered on and running master image (or will reboot into master image) ○ assumptions about SD card existence, partitioning... ○ assumptions about shell prompts linaro-test [rc=0] # ○ etc. etc. ● Goal: LAVA directly controls bootloader ○ netboot: get kernel + DTB + initrd via TFTP ○ extension via board-specific bootloader scripting Tyler's new "bootloader" device support in LAVA appears to have mostly solved this !! Issues: LAVA design
  • 34. ● Terminology learning curve ○ dispatcher, scheduler, dashboard ○ device, device-type ○ What is a bundle? ○ WTF is a bundle stream? ○ Documentation... not helpful (enough said) ● Navigation ○ click intensive ○ how to get from a log to the test results? or... ○ from a test back to the boot log? ○ what about build log (Jenkins?) ○ can I navigate from Jenkins log to the LAVA test? Issues: LAVA usability
  • 35. Kernel + modules: omap2plus_defconfig ● 1 minute ○ hackbox.linaro.org (-j48: 12 x 3.5GHz Xeon, 24G) ● 1.5 minutes ○ khilman local (-j24: 6 x 3.3GHz i7, 16G RAM) ● 8 minutes ○ Macbook Air (-j8: 2 x 1.8GHz i7, 4G) ● 14 minutes ○ Thinkpad T61 (-j4: 2 x Core2Duo, 4G RAM) ● 16 minutes ○ Linaro Jenkins (-j8: EC2 node, built in tmpfs) ● 17 minutes ○ ARM chromebook (-j4: 2 x 1.7 GHz A15, 2G RAM) Issues: Jenkins performance
  • 36. Linaro Test and Validation Summit Grant Likely LCE13 - Dublin, July 2013 LEG Engineering