Sd times-june-24-2015

June 24, 2015
“Make Testing a First Class Citizen in your Development Process”
Dan Boutin – Senior Product Evangelist
dboutin@soasta.com
Mobile (404) 304-9529
@DanBoutinSOASTA

© 2014 SOASTA. All rights reserved. March 3, 2015 2
Agenda - Overview
o First, why is that important? What’s Missing?
o So, when did testing become second class?
o How do we find it?
o How do we keep it?
o Action Plan

3
Revenue
Brand
Competitive advantage

So Where did the “e” go, anyway?
o It was first reported missing in 1995
o Actually dates back to 1989
o Testing has been more about:
• Coding
• Infrastructure & harness set-up
• Logs & spreadsheets
o What is the “E”? Engineering!!

Performance In General – Where we lost the “e”
Linear approaches include delays between testing
Ut
Unit Testing
Ft
Functional
Testing
Development Testing Operations
Pt
Performance
Testing
Mo
Monitoring
Legacy Approach: Silos and waiting
TIME
o Who’s the voice of the NFR’s? And, where?
o Let’s look within the testing window to see what is really going on

Retest
Process changes = new opportunities
Test Development Infra setup
Repeat
?
Fix
Tes
t
Analyze
The Legacy Approach
TIME
Where we lost the “e” And why!
• Test “development” became its own SDLC
• Complete with coding, peer reviews, etc.
• Why? Script coding!
• Infrastructure “set-up” became expensive
• …and did not scale to production
• Why? Costs.
• Analysis was manual
• Actual “testing” window was small
• Still no NFR flow & traceability

So, how do we find the “e”?
o Performance Testing is NOT Performance Engineering
o Performance Engineering consists of many key processes
Modeling
Testin
g
Application
Developmen
t Life Cycle
IT
Operation
s
Capacit
yManagement
Performance
Management
Performance
Engineering

So, how do we find the “e”?
o Performance Engineering is all about the Analysis
o It is not about “scripting” or “coding”
o It should not be about infrastructure set-up

What has made the “e” so important again?

So, how do we get back the “e”?
o Tear down the silos
• Take the discussion out of “testing” & put the focus in ENGINEERING.
• Embed performance engineering in each silo
• It starts in the Dev silo => Performance IS a design requirement
• In an Agile world, the UX IS all about performance

Testing in Production – Why Not?
o “We don’t test in production.”
o This is a direct result of lack of visibility into actual test execution
o So what should the process look like?

Eliminating Delays Compresses Time
Enabling more testing to be done earlier and more often
Ut
Unit Testing
Ft
Functional
Testing
Testing
Pt
Performance
Testing
Mo
Monitoring
Legacy Approach: Silos and waiting
Agile Approach: Automation and Continuous Testing
NFR’s built into the entire lifecycle
OperationsDevelopment
Dev Ops
Testing
TIME

Continuous Delivery presents new opportunities
The ENGINEERING Approach
TIME
The Legacy Approach
Run more tests in less time.
Start earlier.
Cover more ground.xn
/ Test
Analyz
e
Fix
Test Dev
Performance Engineering is Not a Singular
Event
Test Development Infra setup
Tes
t
Analyze Retest
Repeat
?
Fix

Testing with Specific Objectives
Tests at different stages deliver higher coverage
Component
scalability
Performance
baseline
Integration testing
End to End
w/ Service
Virtualization
Unit scalability
Database testing
3rd party
services
Live E2E
tests
Stress
testing Soak test
(reliability)
Failover
Capacity
Latency
Performance
Regression
(smoke tests)
Network
Testing (staging) Operations (prod)Development (sandbox)

Full Lifecycle Testing - Lab to Production
Tests at different stages delivers higher coverage
API Testing
(REST/SOAP
& now
WebSockets)
Testing (staging) Operations (prod)Development (sandbox)
Run smoke
tests with
Continuous
Integration
frameworks
Run larger tests
sooner with
Virtual Private
Clouds
Introduce
external load
sources with
the cloud
Test to
beyond peak
expectations
Establish true
user SLA’s (and
monitor them)
Directly test
your
databases
to scale
Test the “hard
to test”
(Java custom
modules)

1. Build performance tests which will
model real user behavior on your site
2. Tie business metrics (ROI) to website
performance metrics and real user data
3. Correlate data analytics so you can
troubleshoot bottlenecks and improve
performance.
Now we tie it all together:

Performance
Analytics
Optimize performance across your online business
Requires a Continuous Approach

© 2014 SOASTA. All rights reserved. 19
Does it work as
designed?
Does it perform
under extreme
circumstances?
Approve materials Real Drivers
What happens
in actual
conditions?
Assembly Line

© 2014 SOASTA. All rights reserved. 20
Functional validation
Performance,
reliability,
scalability tests
Real Users
Build
What happens
in actual
conditions?
Unit-tested code

Continuous Quality
SOASTA’s Lifecycle
How we do it at SOASTA

Continuous Quality
Jenkins runs any
unit tests as usual

Continuous Quality
Run iterative load
tests via Jenkins
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual

Continuous Quality
Run iterative load
tests via Jenkins
Mobile? Run functional
automation on real devices
for functional & performance
validation
cloud or the lab
Jenkins runs any
unit tests as usual

Continuous Quality
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
validation
cloud or the lab
Jenkins runs any
unit tests as usual

Continuous Quality
Run iterative load
tests via Jenkins
Embedded failure
data isolate issues
Manage to a
performance
baseline
validation
cloud or the lab
Jenkins runs any
unit tests as usual

Continuous Quality
Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Embedded failure
data isolate issues
Manage to a
performance
baseline
validation
cloud or the lab
Jenkins runs any
unit tests as usual

CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015
Build performance tests that model user behavior
on your site
First you need to determine the workload for the test.

How are companies without the “e” faring today?
What if I have a new process?

o What information I need?
● When is your peak second/minute/hour/day?
● What is your peak volume? (active sessions, unique users, page visits, orders, ...)
● What is the median session length?
● What is the distribution of user paths?
● What is the distribution of devices?
● What is the distribution of locations?
o Good Test Data
● Search Terms
● Product Distribution
● Browse Distribution
● Cart Size
● Order Size
How to get a realistic load test
Factors to consider – Now that we have the “e” back!

Let your customers be your guide…

Analyze the most common session paths of real
users

Test how you want, when you want, where you want

1. Let your users help you determine your
performance goals
2. Tie business metrics, (conversion rates, length of
engagement, etc) to calculate costs of
performance vs revenue gain
3. Use data analytics to review site performance
data to isolate bottlenecks, which can guide your
performance investments
Summary & Takeaways

We will close with some Trivia
@DanBoutinSOASTA

Dan Boutin – Senior Product Evangelist
dboutin@soasta.com
Mobile (404) 304-9529
@DanBoutinSOASTA
oSOASTA Products
•FREE DownLoad:
http://goo.gl/6CErMN
oVisit our Blog: http://goo.gl/TA2HKP
oQuestions? @DanBoutinSOASTA

Sd times-june-24-2015

Related slideshows

More Related Content

Sd times-june-24-2015