unit-5 SPM.pptx

TEST AUTOMATION
Software test automation – skill needed for
automation – scope of automation – design and
architecture for automation – requirements for
a test tool – challenges in automation – Test
metrics and measurements – project, progress
and productivity metrics.

Software test automation
• Manual software testing - Trying various usage
and input combinations, comparing the results
to the expected behavior and recording their
observations.
• Automation Testing means using an
automation tool to execute your test case
suite.

• The automation software can also enter
test data , compare expected and actual
results and generate detailed test reports.
• Test Automation demands considerable
investments of money and resources.

Skills needed for automation
The skills required depends on what
generation of automation the company is
in.
1) Capture / playback and test harness
tools (first generation).
2) Data driven tools (second generation).
3) Action driven (third generation).

1) Capture / Playback and Test Harness Tools:
• One of the most boring and time-consuming
activity – rerun manual tests number of
times.
• capture/playback tools -These tools do
this by recording and replaying the
test input scripts.
• Tests can be replayed without attendant for
long hours specially during regression testing.
• Also these recorded test scripts can be edited
as per need i.e., whenever changes are made
to the software.
• These tools can even capture human
operations e.g., mouse activity, keystrokes
etc.

2) Data-driven Tools:
• This method help in developing test scripts
that generates the set of input conditions and
corresponding expected output.
• The approach takes as much time and effort
as the product.
• This generation of automation focuses on
input and output conditions using the black
box testing approach.

3) Action-driven Tools:
• This technique enables a layman to create
automated tests.
• There are no input and expected output
conditions required for running the tests.
• All actions that appear on the application are
automatically tested, based on a generic set of
controls defined for automation.

Scope of Automation in Testing
• Automation is the process of evaluating the
AUT(Application under Test) against the
specification with the help of a tool.
Depending on the nature of testing there are
two main branches under automation.
• Functional testing with automation.
• Performance testing with automation.

Functional Testing with the help of Automation.
• The main area where the functional testing tools are
used is for regression test case execution.
• Mostly in the agile scrum methodology where frequent
releases are happening, it is almost impossible to
execute all the regression test cases manually with the
short span of time.
• Automation gives a high ROI(Return of Investment) in
this area since it is a one time effort for generating the
scripts.
• It is advisable to keep at least 60 to 70 % of regression
cases to be automated for being adherent to the
timelines of test case execution. Automating test cases
is not desirable if it is a onetime testing.

Performance Testing with the help of Automation.
• Performance testing is the process of evaluating the
application performance which is being a critical
requirement of every application now days.
• Performance testing is almost impossible by manual
means.
• There are different tools used across organizations
for evaluating application performance.

Load and Stress testing
• Load testing is performed to confirm that
application can withstand the maximum load
which is mentioned in the SRS.
• Here the application will be tested with the
specified load without overloading the
application.
• In stress testing, here the intention is to find out
the maximum load the application can
accommodate.
• Application will be tested with more load than
the specified limit in order to find out the break
point

• For load and stress testing, large number of concurrent
users is needed and multiple computers are required.
• In manual testing it is diffcult and almost impossible to
run the severe load for several days to identify
problems in the application.
• While using automation the testing software (tool)
monitors the CPU and memory usage of each test
computer.
• The tool generates artificial loads by creating virtual
users and which will produce a guaranteed test
accuracy.
• The tools can generate number of virtual users on a
single computer.
• Automation help us to find out defects as we can run
the test using multiple users, multiple environments on
several days.

• A test automation group's purpose should be
to develop automated support for testing
efforts.
• This group should be responsible for the
design and implementation of a data-driven
automated testing framework.

Assumptions
• An integrated tool suite must be the primary test
management, planning, development, and
implementation vehicle.
• The tool suite must be used to direct and control
test execution, to store and retrieve test artifacts,
and to capture/analyze/report test results.
• The tool suite must include a tool of choice for
defect tracking and resolution.
• The tool suite must include a configuration
management tool of choice

• The proper automated testing workspaces
must be created on test servers that are
separate from development servers.
• Testing standards must be documented and
followed.

Constraints
• The automated tools group resources must
remain independent of any manual testing group.
• There may not be a large enough number of
available staff on the automation team.
• The level of cooperation of the software
development group and automated tool
management may be too low.
• There is uncertainty associated with the GUI
updates in AUT.
• There may be corporate mandates on what tools
must be used.

Critical Success Factors
• Test automation must be implemented as a
full-time effort, not a sideline.
• The test design process and the test
automation framework must be developed as
separate entities.
• The test framework must be easy to expand,
maintain, and enhance.

Strategic Objectives
• Implement a testing framework that both technical and
nontechnical testers can use.
• Document and publish the framework.
• Develop automated regression tests for
 GUI objects and events
 Application functions
 Application special features
 Application performance and scalability
 Application reliability
 Application compatibility
 Application performance
 Database verification

SOFTWARE AUTOMATION
FRAMEWORK
What is Framework?
• A framework is considered to be a
combination of set protocols, rules, standards
and guidelines that can be incorporated or
followed as a whole so as to leverage the
benefits of the scaffolding provided by the
Framework.

Let us consider a real-life scenario.
We very often use lifts or elevators. There are a few guidelines
those are mentioned within the elevator to be followed and taken
care off so as to leverage the maximum benefit and prolonged
service from the system.
Thus, the users might have noticed the following guidelines:
• Keep a check on the maximum capacity of the elevator and do not
get onto an elevator if the maximum capacity has reached.
• Press the alarm button in case of any emergency or trouble.
• Allow the passenger to get off the elevator if any before entering
the elevator .
• In case of fire in the building or if there is any haphazard situation,
avoid the use of the elevator.
• Do not play or jump inside the elevator.
• Do not smoke inside the elevator.
• Call for the help/assistance if the door doesn’t open or if the
elevator doesn’t work at all. Do not try to open the doors forcefully.

Test Automation Framework
• A “Test Automation Framework” is scaffolding
that is laid to provide an execution environment
for the automation test scripts.
• The framework provides the user with various
benefits that help them to develop, execute and
report the automation test scripts efficiently.
• It is more like a system that has created
specifically to automate our tests.

Advantage of Test Automation framework
• Reusability of code
• Maximum coverage
• Recovery scenario
• Low-cost maintenance
• Minimal manual intervention
• Easy Reporting

Types of Test Automation Framework
• Module Based Testing Framework
• Library Architecture Testing Framework
• Data Driven Testing Framework
• Keyword Driven Testing Framework
• Hybrid Testing Framework
• Behavior Driven Development Framework

• 1) Module Based Testing Framework
• Module based Testing Framework is based on
one of the popularly known OOPs concept –
Abstraction.
• The framework divides the entire “Application
Under Test” into a number of logical and isolated
modules.
• For each module, we create a separate and
independent test script.
• Thus, when these test scripts took together builds
a larger test script representing more than one
modules.

Pros:
• The framework introduces the high level of
modularization which leads to easier and cost-efficient
maintenance.
• The framework is pretty much scalable
• If the changes are implemented in one part of the
application, only the test script representing that part
of the application needs to be fixed to leave all the
other parts untouched.
Cons:
• While implementing test scripts for each module
separately, we embed the test data (Data with which
we are supposed to perform testing) into the test
scripts.

2) Library Architecture Testing Framework
• The Library Architecture Testing Framework is
fundamentally and foundationally built on Module
Based Testing Framework with some additional
advantages.
• Instead of dividing the application under test into test
scripts, we segregate the application into functions or
rather common functions can be used by the other
parts of the application as well.
• Thus we create a common library constituting of
common functions for the application under test.
Therefore, these libraries can be called from the test
scripts whenever required.
• The basic fundamental behind the framework is to
determine the common steps and group them into
functions under a library and call those functions in the
test scripts whenever required.

• Example: The login steps can be combined
into a function and kept into a library. Thus all
the test scripts those require to login the
application can call that function instead of
writing the code all over again.

Pros:
• Introduces the high level of modularization which leads
to easier and cost-efficient maintenance and scalability
too.
• As we create common functions that can be efficiently
used by the various test scripts across the Framework.
Thus, the framework introduces a great degree of re-
usability.
Cons:
• Like Module Based Framework, the test data is lodged
into the test scripts, thus any change in the test data
would require changes in the test script as well.
• With the introduction of libraries, the framework
becomes a little complicated.

3) Data Driven Testing Framework
• Data Driven Testing Framework helps the user
segregate the test script logic and the test data
from each other.
• It lets the user store the test data into an
external database.
• The external databases can be property files, xml
files, excel files, text files, CSV files, ODBC
repositories etc.
• The data is conventionally stored in “Key-Value”
pairs. Thus, the key can be used to access and
populate the data within the test scripts.

• 4) Keyword Driven Testing Framework
• It also keeps the certain set of code belonging to
the test script into an external data file.
• These set of code are known as Keywords and
hence the framework is so named. Keywords are
self-guiding as to what actions need to be
performed on the application.
• The keywords and the test data are stored in a
tabular like structure and thus it is also popularly
regarded as Table driven Framework. Take a
notice that keywords and test data are entities
independent of the automation tool being used.

5) Hybrid Testing Framework
• As the name suggests, the Hybrid Testing
Framework is a combination of more than one
above mentioned frameworks. The best thing
about such a setup is that it leverages the
benefits of all kinds of associated frameworks.

Components of Automation Testing
Framework

• Object Repository: Object Repository acronym as OR is constituted
of the set of locators types associated with web elements.
• Test Data: The input data with which the scenario would be tested
and it can be the expected values with which the actual results
would be compared.
• Configuration File/Constants/ Environment Settings: The file stores
the information regarding the application URL, browser-specific
information etc. It is generally the information that remains static
throughout the framework.
• Generics/ Program logics/ Readers: These are the classes that store
the functions which can be commonly used across the entire
framework.
• Build tools and Continuous Integration: These are the tools that
aids to the capabilities of the framework to generate test reports,
email notifications and logging information.

Generic requirements for test
tool/framework
1. No hard coding in the test suite
Keep all variables in separate file
2.Adding test case/suite expandability
Adding new test case should not affect the other
TC.
should not result in retesting the complete test
suite.
3. Reuse of code for different types of testing, test
cases

4. Automatic setup and clean
Each test program should have a setup
program that will create the necessary setup
before executing the test cases.
A set up for one test case may work negatively
for another test case. So it is important not
only to create the setup but also “undo” the
setup soon after the test execution for the test
case.

5. Independent test case
To execute a particular TC , it should not
expect any other TC to have been executed
before nor should it implicitly assume that
certain other TC will be run after it.
EX: TC 2 to be executed after TC-1

6. Dependent TC
Particular TC should be executed before or
after a dependent TC
7. Insulating TC during execution
Interruption of test case execution by some
events or signals – Ex automatic pop up
screens on web browsers. – It affects TC
execution- Framework provides an option for
users to block some of the events.

8. Coding standards and directory structure
CS and DS for a test suite may help the new Er
in understanding he test suite fast and help in
maintaining the test suite.
9. Selective execution of test cases
Framework may have multiple test suites.
Test suite may have multiple test programs.
Test program may have multiple test cases.
The test tool or a framework should have a
facility for the test engineer to select a
particular test cases or set of test cases.

10. Random execution of test cases
Ex:1
random
test-program-name 2,1,5
Test engineer wants the test tool to select one out of test
cases 2,1,5 and executed.
Ex: 2
random
test-program 1(2,1,5)
test-program2
test-program5
Test Engineer wants one out of test programs 1,2, and 5.
If program 1 is selected then one out of test cases 2,1 and 5
to be randomly executed.
if program 2 or 3 are selected, then all the test cases in
those programs are executed.

11. Parallel execution of test cases
It simulates the behavior of several machines running the
same test and hence is very useful for performance and
load testing.
12. Looping the test cases
Reliability testing requires the test cases to be executed in a
loop.
Iteration loop
Repeat_loop, 50
Test –program 1(3) – test case 3 form test program1 is
repeated 50 times.
Timed loop
Time_loop, 5 hours
test- program1 (2,1,5)
test_program 2
Test cases 2,1 and 5 from test program1 and all the test cases
of program 2 are executed in order, in a loop for five hours.

13. Grouping of test cases
The grouping of scenarios allows several tests to be
executed in a predetermined combination of scenarios.
it allows the selected test cases to be executed in order,
random, in a loop all at the same time.
Ex:
Group scenario 1
Parallel, 2 AND repeat, 10@scene1
Scen1
test_program1(2,1,5)
test_program2
test_program3
All test programs are executed by two instances
simultaneously in an iteration loop for 10 times.

14. Test case execution based on previous result
Regression test methodology requires that
test cases be selected based on previous
result and history.
Ex:
Rerun all the test cases which were executed
previously
Rerun only failed/not run test cases
Execute all test cases that were executed on “
Jan 26, 2015”.

SELECTING A TEST TOOL
• Broad categories for classifying the criteria
are
1. Meeting requirements
2. Technology expectations
3. Training/skills
4. Management aspects.

Meeting requirements
• There are many tools available in the market today
but rarely do they meet all the requirements of
given product or a given organization.
• Evaluating different tools for different requirements
involves lot of effort, money and time.
• Huge delay is involved in selecting and implanting
test tools.
• Test tools may not provide backward or forward
compatibility with the product-under-test (PUT).
• Test tools may not go through the same amount of
evaluation for new requirements.

• A number of test tools cannot distinguish
between a product failure and a test failure.
• This increases analysis time and manual
testing.
• The test tools may not provide the required
amount of trouble-shooting/debug/ error
messages to help in analysis.

Technology Expectations
• In test tools may not allow test developers to
extend / modify the functionality of the
framework.
• It involves going back to the tool vendor with
additional cost and effort.
• Very few tools available in market provide source
code for extending functionality or fixing some
problems.
• Extensibility and customization are important
expectations of a test tool.

• A good number of test tools require their
libraries to be linked with product binaries.
• When these libraries are linked with the source code
of the product, it is called as the "instrumented
code".
• Finally, test tools are not 100% cross-platform.
• They are supported only on some O.S. platforms and
the scripts generated from these tools may not be
compatible on other platforms.

Training Skills
• Test tools require plenty of training, but very few
vendors provide the training to the required
level.
• Organization-level training is needed to deploy the
test tools, as the users of the test suite are not only
the test team but also the development team and
other areas like SCM (Software Configuration
Management).
• Test tools expect the users to learn new
language/scripts and may not use
standard languages/scripts.
• This increases skill requirements for automation
and increases the need for a learning curve inside
the organization.

4) Management Aspects
• When selecting the test tool, it is important to
note the system requirements and the cost
involved in upgrading the software and
hardware needs to be included with the cost
of the tool.
• Migrating from one test tool to another may
be difficult and requires a lot of effort.

• Not only is this difficult, as the test suite that
is written cannot be used with other test tools
but also because of the cost involved.
• As the tools are expensive and unless the
management feels that the returns on
investment (ROI) are justified, changing tools
are generally not permitted.

Following are the seven steps to select and deploy
a test tool in an organization.
• Step - 1: Identify your test suite requirements
among the generic requirements discussed. Add
other requirements, if any.
• Step - 2: Make sure experiences discussed in previous
sections are taken care of.
• Step - 3: Collect the experiences of other
organizations, which used similar test tools.
• Step - 4: Keep a checklist of questions to be asked to
the vendors on cost / effort / support.
• Step - 5: Identify list of tools that meet the above
requirements and give priority for the tool, which is
available with the source code.
• Step - 6: Evaluate and shortlist one / set of tools
and train all test developers on the tool.
• Step - 7: Deploy the tool across test teams
after training all potential users of the tool.

CHALLENGES IN AUTOMATION
1) Testing the complete application:
It’s not possible to test each and every combination
both in the manual as well as in automation testing. If
you try all these combinations you will never ship the
product
2) Misunderstanding of company processes:
Sometimes you just don’t pay proper attention what
the company-defined processes are and these are for
what purposes. There are some myths in testers that
they should only go with company processes even
these processes are not applicable for their current
testing scenario. This results in incomplete and
inappropriate application testing.

3) Relationship with developers:
Big challenge. Requires very skilled tester to
handle this relation positively and even by
completing the work in testers way. There are
simply hundreds of excuses developers or
testers can make when they do not agree with
analyzing skill. some points. For this tester
also requires good communication,
troubleshooting

4) Regression testing:
When a project goes on expanding the regression
testing work simply becomes uncontrolled.
Pressure to handle the current functionality
changes, previous working functionality checks,
and bug tracking.
5) Lack of skilled testers:
It can be called this as ‘wrong management
decision’ while selecting or training testers for
their project task in hand. These unskilled fellows
may add more chaos than simplifying the testing
work. This results in incomplete, insufficient and
ad-hoc testing throughout the testing life cycle.

6) Testing always under time constraint:
When this order comes from the boss, tester
simply focuses on task completion and not on
the test coverage and quality of work. There is
a huge list of tasks that you need to complete
within specified time. This includes writing,
executing, automating and reviewing the test
cases.

7) Which tests to execute first?
If you are facing the challenge stated in point no 6,
then how will you take a decision which test cases
should be executed and with what priority? Which
tests are important over others? This requires good
experience to work under pressure.
8 ) Understanding the requirements:
Sometimes testers are responsible for communicating
with customers for understanding the requirements.
What if tester fails to understand the requirements?
Will he be able to test the application properly?
Definitely No! Testers require good listening and
understanding capabilities.

9) Automation testing:
Many sub-challenges –
• Should automate the testing work?
• Till what level automation should be done?
• Do you have sufficient and skilled resources for
automation?
• Is time permissible for automating the test cases?
The decision of automation or manual testing will
need to address the pros and cons of each process.

10) The decision to stop the testing:
When to stop testing?
Very difficult decision. Requires core judgment
of testing processes and importance of each
process. Also requires ‘on the fly’ decision
ability.

• 11) One test team under multiple projects:
Challenging to keep track of each task.
Communication challenges. Many times results in
failure of one or both the projects.
• 12) Reuse of Test scripts:
Application development methods are changing
rapidly, making it difficult to manage the test
tools and test scripts.
• Test script migration or reuse is very essential but
difficult task.

13) Testers focusing on finding easy bugs:
If the organization is rewarding testers based
on a number of bugs (very bad approach to
judge testers performance) then some testers
only concentrate on finding easy bugs those
don’t require deep understanding and testing.
A hard or subtle bug remains unnoticed in
such testing approach.

• 14) To cope with attrition:
Increasing salaries and benefits making many
employees leave the company at very short
career intervals. Management is facing hard
problems to cope with attrition rate.
• Challenges – New testers require project
training from the beginning, complex projects
are difficult to understand, delay in shipping
date!

unit-5 SPM.pptx

Related slideshows

More Related Content

unit-5 SPM.pptx