It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
- 1. It seemed a good idea at the time:
Intelligent Mistakes
in Test Automation
Prepared and presented by
Dorothy Graham
email: info@dorothygraham.co.uk
@DorothyGraham
www.DorothyGraham.co.uk
© Dorothy Graham 2013
1
Intelligent mistakes?
• mistake:
– an action resulting from defective judgment,
deficient knowledge or carelessness, a
misconception or misunderstanding
• intelligent
– exercising good judgment, showing mental
resourcefulness, being well-informed
• intelligent mistake
– action based on a faulty premise, with the best of
intentions, but with adverse consequences
2
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 2. Contents
•
•
•
•
•
Automation should find lots of bugs
Automation comes out of a box (or download)
Let’s automate all of our manual tests
ROI is essential for automation
Testing tools are testers’ tools
3
Good idea? Automation should find lots
of bugs
• Seems very sensible
– we are automating tests
– tests find bugs
– automation is more thorough and faster
– therefore, automation should find more bugs
- and quicker
• Problems
– what does automation actually give us?
– what tests are we automating?
4
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 3. What is automated?
most often
automated
likelihood of
finding bugs
regression tests
exploratory testing
5
Automation success = find lots of bugs?
• tests find bugs, not automation
• automation is a mechanism for running tests
• the bug-finding ability of a test is not affected
by the manner in which it is executed
• this can be a dangerous objective
– especially for regression automation!
Automated tests
Manual Scripted
Exploratory
Fix Verification
9.3%
24.0%
58.2%
8.4%
Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman
info@dorothygraham.co.uk
6
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 4. Efficiency and effectiveness
better
good
slow
testing
Manual testing
High
good
fast
testing
Automated
Efficiency
poor
fast
testing
poor
slow
testing
worst
greatest
benefit
Effectiveness
not good but
common
Low
7
When is “find more bugs” a good
objective for automation?
• when the first run of a given test is automated
– Model-Based Testing (MBT), exploratory test
automation, automated test design, monkey testing
– keyword-driven (e.g. users populate spreadsheet)
• find bugs in parts we wouldn’t have tested?
– indirect result of automation
– direct result of running more tests
8
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 5. Contents
•
•
•
•
•
Automation should find lots of bugs
Automation comes out of a box (or download)
Let’s automate all of our manual tests
ROI is essential for automation
Testing tools are testers’ tools
9
Good idea? Automation comes out of the
box (or download)
• Seems very sensible
– automation gives great benefit, success stories
– spending a lot on the tool must guarantee great
benefits
– all we need to do is “get the right tool”
• Problems
– technical issue: what testware architecture is right
for your organisation? (hint: not the tool’s)
– management issues: staffing, support, resources
10
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 6. Automation should fit your organisation
• your own testware architecture is critical
– a poor architecture gives
high maintenance cost
Testers&&
write&tests&(in&DSTL)&
• most frequent cause of
abandoned automation/
shelfware
HL Keywords
testware&
architecture&
– two layers of abstraction
– technical: for long life
– human: for wide use
– using the tool’s architecture
ties you to that tool [version]
Structured
Scripts
Test&Execu+on&Tool&
runs&scripts&
11
It’s not about the tool
• no such thing as “the right tool” or “best tool”
– what’s “the best car”?
commercial tools?
investment in
good automation
good benefits
high cost
poor benefits
low cost
benefits
good benefits
low cost cost
moderate
poor benefits
high cost
tool cost
open source tools?
info@dorothygraham.co.uk
budget
12
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 7. It takes time and effort to succeed
• management support is critical
– high level managers need to understand
automation capability & limitations, and have
realistic expectations
– building good automation is a learning process
– “people issues” – people use the automation,
people develop the automation
13
Automated tests/automated testing
Automated tests
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• analyse test failures
• report defect(s)
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Manual process
info@dorothygraham.co.uk
Automated testing
Select / identify test cases to run
Set-up test environment:
• create test environment
• load test data
Repeat for each test case:
• set-up test pre-requisites
• execute
• compare results
• log results
• clear-up after test case
Clear-up test environment:
• delete unwanted data
• save important data
Summarise results
Analyse test failures
Report defects
Automated process
14
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 8. Contents
•
•
•
•
•
Automation should find lots of bugs
Automation comes out of a box (or download)
Let’s automate all of our manual tests
ROI is essential for automation
Testing tools are testers’ tools
15
Good idea? Let’s automate all of our
tests
• Seems very sensible
– we already have lots of tests being done manually
– automation would be much quicker
– we can be completely automated, no need for
manual tests [testers?!]
• Problems
– what tests are you automating?
– what are you missing by automating [only]
existing tests?
16
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 9. Automate manual tests?
manual
tests
automated
tests
tests not
automated
yet
tests not
worth
automating
tests (&
verification)
not possible to
do manually
manual tests
automated
(% manual)
new ways of
automating, e.g.
exploratory test
automation
17
Tools will replace testers?
• “we can reduce the number of testers once we
have the tool”
– what are your testers like?
• mindless morons, or
• intelligent investigators?
– need more skills, not fewer
– automation can free testers to do more test
design, exploratory testing
• and find more bugs
– tools don’t replace testers, they support them
18
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 10. Contents
•
•
•
•
•
Automation should find lots of bugs
Automation comes out of a box (or download)
Let’s automate all of our manual tests
ROI is essential for automation
Testing tools are testers’ tools
19
Good idea? ROI is essential for
automation
• Seems very sensible
– we will be spending time and possibly lots of
money on tools – need to be sure it will be worth it
– a “water-tight” business case will convince our
managers to invest in automation
• Problems
– ROI is not the whole story
– people are not convinced by numbers
– calculating ROI is difficult and can be dangerous
20
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 11. Is this Return on Investment (ROI)?
•
•
•
•
•
tests are run more often
tests take less time to run
it takes less human effort to run tests
we can test (cover) more of the system
we can run the equivalent of days / weeks of
manual testing in a few minutes / hours
• faster time to market
ROI = (benefit – cost)
cost
these are (good) benefits
but are not ROI
21
How important is ROI?
• ROI can be dangerous
– may give impression that people are replaced by
tools (see http://dorothygraham.blogspot.co.uk/)
• “automation is an enabler for success, not a
cost reduction tool” – Yoram Mizrachi
• many achieve lasting success without
measuring ROI (depends on your context)
– need to be aware of benefits (and publicize them)
22
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 12. An example comparative benefits chart
80
70
60
50
40
man
aut
30
20
10
0
exec speed
14 x faster
times run
data variety tester work
5 x more often 4 x more data 12 x less effort
ROI spreadsheet – email me for a copy
23
Contents
•
•
•
•
•
Automation should find lots of bugs
Automation comes out of a box (or download)
Let’s automate all of our manual tests
ROI is essential for automation
Testing tools are testers’ tools
24
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 13. Good idea? Testing tools are testers’
tools
• Seems very sensible
– testers do testing – testing tools help with testing
– tools will make them better testers, because they
will be able to run more tests, faster
• Problems
– tools are software packages using programming
languages; tester skills are different to
development skills
– spending time on / with tools means less testing,
not more testing (initially at least)
25
Is it the tester’s job to automate tests?
– test automation is software development
• needs programming skills
– not all testers want to become developers
• or would be good at it
– automators need testing skills?
– if testers are automators a conflict of interest
• do you run tests or do you automate tests?
• automation is better long-term, BUT
• deadline pressure pushes you back into manual testing
26
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 14. Why are these intelligent mistakes?
• Automation should find bugs?
– tests find bugs; automated tests are still tests
• Automation comes “out of the box”
– you think you buy a solution that will just work
• Automate all manual tests?
– a starting point for tests to be automated
• Automation has to achieve ROI?
– can’t expect investment with showing some return
• Testing tools are tools for testers?
– testers write and run tests; tools run tests
27
Why are these intelligent mistakes?
• Automation should find bugs?
– tests find bugs, automation runs tests
• Automation comes “out of the box”
– effort and time, e.g. tailor your testware architecture
• Automate all manual tests?
– not all should be, automate more than manual
• Automation has to achieve ROI?
– may be dangerous, need to show benefits
• Testing tools are tools for testers?
– different skills, not all testers should code
info@dorothygraham.co.uk
28
© Dorothy Graham 2013
www.DorothyGraham.co.uk
- 15. Summary
• Many ideas seem sensible at first / on the
surface, but have serious problems
– automation should find lots of bugs
– automation comes out of a box (or download)
– let’s automate all of our manual tests
– ROI is essential for automation
– testing tools are testers’ tools
• Recognise these “intelligent mistakes” and
avoid them
29
More information
• downloads www.DorothyGraham.co.uk
– articles and papers
• email info@DorothyGraham.co.uk for
– Framework and test execution tool list
– ROI calculator
– my random newsletter
• blog http://dorothygraham.blogspot.com
– including automation, DDP, certification
• twitter
– @DorothyGraham
30
info@dorothygraham.co.uk
© Dorothy Graham 2013
www.DorothyGraham.co.uk