Maximize the Benefits from Your Test Automation Investment

Maximize the Benefits from
Your Test Automation
Investment

14 November 2013
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

1

Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improve the Quality?
• Return on Investment with Test Automation
• Different App Verticals – What is Important?
• 10 Tips for Professional Mobile App Testing
• Testdroid Update
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

3

Agenda
• Q&A

4

Challenges in Today’s App Dev
Fact: Most typical application fails when tested
across different device variants used by consumers
• Failures: Both SW and HW cause failures
• Emulators do not offer realistic platform for dev
• Already installation of app fails – not tested at all
• Application crashes, leaks memory, consumes too
much resources, behaves not as meant to behave
• UI scaling/display/resolution – common problem
• Application performance is bad, draining battery

5

Why Do They Fail?


6

Agenda
• Q&A

7

How to Test: Manual vs. Automation
• Costs & Assets
– Time, People, Infrastructure, Tools, Training

• App Development & Testing Cycle – Together
– Testing after Daily Builds (Continuous Integration)

• Tools & Technology – Frameworks
– Myriad of frameworks available to help you

• Test Coverage & Reusability
– Global coverage is CRITICAL to SUCCESS

• Outcome – Learning, Improving, Faster

8

Manual

Automation
Large
coverage, quick
ly
completed, Less
money & time
wasted, Exact
results.

Smaller coverage, More
money burnt & time
wasted, Error-prone

9

Improving Time-to-Market

Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


10

Tips to Improve the App Quality
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets

11

Test Automation Will Deliver You…
• App Compatibility & Global coverage
Test requirements / Planning

Smoke testing

Regression testing

Regression testing

Functional & Interoperability
testing

Regression testing

testing

testing

Performance & Stress testing

Regression testing

Compatibility testing

Performance & Stress testing

Test Automation during the Mobile App Development Process

• Reusability
• Test efficiency & effectiveness
• Faster time-to-market

12

Agenda
• Q&A

13

Example: Cost of Manual Testing
• Test setup for each device – 5 minutes
• 100 test cases, each take about 15 seconds to
execute – 25 minutes
• 4 different application variants: tablet, phone and
2 localized versions (English & German)
• Avg. 2 bugs (of 100 test cases) are found on each
test run – 20 minutes to report bugs
• 15 devices will be used for testing
TIME FOR TESTING: 3000 min (50 hours)
COST FOR TESTING: $60/h x 50 = $3,000 (x4) = $12K

14

Example: Cost of Test Automation
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minutes (25h)
COST FOR TEST CREATION: 25 hours x $60/h = $1,500
• Tools for test automation with the following setup:
– Unlimited devices (250+ Android devices)
– Unlimited execution times

TESTDROID CLOUD: $1,500/month
TOTAL COSTS FOR TEST AUTOMATION: $3,000/month
Direct savings compared to manual: $9,000
ROI ratio: (Gains-Investm.) / Investm.: $9K-$3K/$3K = 2.00

15

Benefits with Testdroid Cloud
• Instant availability of hardware and no need to
buy any device by yourself
• Testing gets done much faster – typically the
same outcome can be achieved in 10% of time
• Test cases are reusable for your next project
• Immediate feedback – all reports as
logs, screenshots, performance stats etc.
• QA can focus on quality – not pressing keys!

16

Example: On-Premise Setup
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minutes (25h)
COST FOR TEST CREATION: 25 hours x $60/h = $1,500
TESTDROID ENTERPRISE: $2,000/month (15 devices)

TOTAL COSTS FOR TEST AUTOMATION: $3,500/month
Direct savings compared to manual: $8,500
ROI ratio:
(Gains-Investm.) / Investm.: $8.5K-$3.5K/$3.5K = 1.43

17

Agenda
• Q&A

18

Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Travel

• Testdroid has helped
thousands of app
developers in these
subverticals!
• What are the critical
elements in each these
verticals?

Mobile

19

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• User Experience!
• Resource consumption &
validation (CPU, Mem…)
• Fully utilizing hardware –
e.g. Touch screen
• Mobile games require
access to graphics APIs
(e.g. OpenGL ES)


20

• Usability!
• Functional and
Utilities & Tools
Behavioral testing
• Metrics analysis for all
captured data (e.g.
Banking & Payment
logs, screenshots, perf
Retail & Travel
stats)
Mobile
• Connection with Social
Media + other apps
Games


21

• Performance!
• Connectivity, robustness
Utilities & Tools
and durability
• Screen orientation
(portrait vs. landscape)
Banking & Payment
• Graphics
Retail & Travel
quality, streaming
Mobile
capabilities
• Technologies Ltd. 2013
© Copyrights by Bitbar User profiles
Games


22

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• Security!
• Secure connectivity with
back-end systems
• Top Quality – extremely
brand sensitive vertical
• Compliances and
verification between real
devices and infrastructure


23

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• Data!
• Connectivity and data
connection with back-end
systems
• Configurability of the app
• Bad quality can hurt the
brand and leave you
without customers


24

Agenda
• Q&A

25

Tip #1: Test early, Test often


27

Tip #2: Plan What to Automate
Time

MANUAL

Infrastructure
Tools

AUTOMATED

People

Training


28

Tip #3: Use only Real Devices


29

Tip #4: Use Atomic Test Units

Test iteration #1

Test iteration #2


30

Tip #5: Create Hermetic Tests


31

Tip #6: Use All Possible Devices

Testdroid Cloud’s 250+ Android devices
= 93-95% global Android volumes!

32

Tip #7: Separate Apps and Tests

Your App


33

Tip #8: Output Everything to Logs


34

Tip #9: End-to-End Testing


35

Tip #10: Integrate Dev & Testing
Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


36

Agenda
• Q&A

37

Testdroid Products
Complete Solution for Mobile Apps/Games Testing


38

Testdroid
– Because it is important to know what app testing costs!


39

Agenda
• Q&A

40


41

Maximize the Benefits from Your Test Automation Investment

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (16)

Similar to Maximize the Benefits from Your Test Automation Investment

Similar to Maximize the Benefits from Your Test Automation Investment (20)

More from Bitbar

More from Bitbar (17)

Recently uploaded

Recently uploaded (20)

Maximize the Benefits from Your Test Automation Investment