Chapter 6 - Test Tools and Automation

Test Tools & Automation
1 TTA’s Task in
Risk Based Testing
2 White Box Test
Techniques
3 Analytical
Techniques
Software Testing - ISTQB Advance
Technical Test Analyst Exam Preparation
Chapter 6
Neeraj Kumar Singh
4 Quality
Characteristics
5 Reviews
6 Test Tools &
Automation

Contents
6.1 Defining the Test Automation Project
6.2 Specific Test Tools
Neeraj Kumar Singh

Defining the Test Automation Project
Introduction
In order to be cost-effective, test tools (and particularly those which support test execution), must be carefully architected and
designed.
A test automation project should be considered a software development project. This includes the need for architecture
documentation, detailed design documentation, design and code reviews, component and component integration testing, as well
as final system testing. There are multiple tasks that the Technical Test Analyst can perform regarding test execution automation.
These include:
 Determining who will be responsible for the test execution (possibly in coordination with a Test Manager)
 Selecting the appropriate tool for the organization, timeline, skills of the team, and maintenance requirements (note this
could mean deciding to create a tool to use rather than acquiring one)
 Defining the interface requirements between the automation tool and other tools such as the test management, defect
management and tools used for continuous integration
 Developing any adapters which may be required to create an interface between the test execution tool and the software
under test
 Selecting the automation approach, i.e., keyword-driven or data-driven
 Scheduling the automation project and allocating the time for maintenance
 Training the Test Analysts and Business Analysts to use and supply data for the automation
 Determining how and when the automated tests will be executed
 Determining how the automated test results will be combined with the manual test results
Neeraj Kumar Singh

Selecting the Automation Approach
This section considers the following factors which impact the test automation approach:
 Automating through the GUI
 Applying a data-driven approach
 Applying a keyword-driven approach
 Handling software failures
 Considering system state
Automating through the GUI
 Test automation is not limited to testing through the GUI.
 Can run tests at API level, through a Command Line Interface (CLI) & other interfaces
 Determining the most effective interface to be accessed to automate the testing
 One of the difficulties of testing through the GUI is the tendency for the GUI to change as the software evolves
Neeraj Kumar Singh

Applying a Data-driven Approach
 The data used for each test may be different although the executed test steps are virtually identical
 It is inefficient to develop and maintain an automated test script for each of these values to be tested. A common
technical solution to this problem is to move the data from the scripts to an external store such as a spreadsheet or
a database.
 When using this approach, in addition to the test scripts that process the supplied data, a harness and infrastructure
are needed to support the execution of the script or set of scripts.
 In most cases, the Test Analyst will be responsible for executing the test scripts once the automation is implemented
and tested.
Neeraj Kumar Singh

Applying a Keyword-driven Approach
 Another approach, called keyword- or action word-driven, goes a step further by also separating the action to be
performed on the supplied data from the test script
 Each statement of this language describes a full or partial business process of the domain that may require testing
 A keyword describes a high-level action that will be performed in the application domain. Lower level actions which
denote interaction with the software interface itself
 Once the keywords and data to be used have been defined, the test automator translates the business process
keywords and lower level actions into test automation code.
 The keywords and actions, along with the data to be used, may be stored in spreadsheets or entered using specific
tools which support keyword-driven test automation.
 Programming skills are needed to implement these modular scripts.
Neeraj Kumar Singh

Handling Software Failures
In any test automation design, it is important to anticipate and handle software failures.
If a failure occurs, the test automator must determine what the software should do.
 Should the failure be recorded and the tests continue?
 Should the tests be terminated?
 Can the failure be handled with a specific action or perhaps by adding a delay in the test?
Unhandled software failures may corrupt subsequent test results as well as causing a problem with the test that was
executing when the failure occurred.
Considering System State
 It is also important to consider the state of the system at the start and end of the tests.
 It may be necessary to ensure the system is returned to a pre-defined state after the test execution is completed.
 This will allow a suite of automated tests to be run repeatedly without manual intervention to reset the system to a
known state.
Neeraj Kumar Singh

Modeling Business Processes for Automation
In order to implement a keyword-driven approach for test automation, the business processes to be tested must be
modeled in the high-level keyword language. It is important that the language is intuitive to its users who are likely to be
the Test Analysts working on the project or, in the case of Agile projects, the business representative.
Keywords are generally used to represent high-level business interactions with a system. For example, “Cancel_Order”
may require checking the existence of the order, verifying the access rights of the person requesting the cancellation,
displaying the order to be cancelled and requesting confirmation of the cancellation. Sequences of keywords (e.g.,
“Login”, “Select_Order”, “Cancel_Order”), and the relevant test data are used by the Test Analyst to specify test cases.
The following is a simple keyworddriven input table that could be used to test the ability of the software to add, reset
and delete user accounts:
Neeraj Kumar Singh

Modeling Business Processes for Automation
Issues to consider include the following:
 The more granular the keywords, the more specific the scenarios that can be covered, but the high-level language
may become more complex to maintain.
 Allowing Test Analysts to specify low-level actions ("ClickButton", "SelectFromList", etc.) makes the keyword tests
much more capable of handling different situations. However, because these actions are tied directly to the GUI, it
also may cause the tests to require more maintenance when changes occur.
 Use of aggregated keywords may simplify development but complicate maintenance. For example, there may be six
different keywords that collectively create a record. Should a keyword that actually calls all six keywords
consecutively be created to simplify that action?
 No matter how much analysis goes into the keyword language, there will often be times when new and different
keywords will be needed. There are two separate domains to a keyword (i.e., the business logic behind it and the
automation functionality to execute it). Therefore, a process must be created to deal with both domains.
Keyword-based test automation can significantly reduce the maintenance costs of test automation, but it is more costly,
more difficult to develop, and takes more time to design correctly in order to gain the expected return on investment.
Neeraj Kumar Singh

Specific Test Tools
Introduction
This section contains overview information on tools that are likely to be used by a Technical Test Analyst beyond what is
discussed in the Foundation Level syllabus.
 Fault Seeding/Fault Injection Tools
 Performance Testing Tools
 Tools for Web-Based Testing
 Tools to Support Model-Based Testing
 Component Testing and Build Tools
 Tools to Support Mobile Application Testing
Neeraj Kumar Singh

Specific Test Tools
Fault Seeding/Fault Injection Tools
Fault seeding tools actually modify the code under test (possibly using predefined algorithms) in order to check the
coverage achieved by specified tests. When applied in a systematic way this enables the quality of the tests (i.e., their
ability to detect the inserted defects) to be evaluated and, where necessary, improved.
Fault injection tools deliberately supply incorrect inputs to the software to ensure the software can cope with the fault.
The inputs are injected to disrupt the normal execution flow of the code and enable test coverage to be extended (e.g.,
to cover more negative test conditions and test error handling mechanisms).
Both of these types of tools are generally used by the Technical Test Analyst, but may also be used by the developer when
testing newly developed code.
Neeraj Kumar Singh

Specific Test Tools
Performance Testing Tools
Performance testing tools have the following main functions:
 Generating load
 Providing measurement, monitoring, visualization and analysis of the system response to a given load
 Giving insights into the resource behavior of system and network components
Performance tools generate a load by simulating large numbers of multiple users (“virtual” users) following their
designated operational profiles to accomplish tasks including generating specific volumes of input data.
In comparison with individual test execution automation scripts, many performance testing scripts reproduce user
interaction with the system at the communications protocol level and not by simulating user interaction via a graphical
user interface.
This usually reduces the number of separate "sessions" needed during the testing. Some load generation tools can also
drive the application using its user interface to more closely measure response time while the system is under load.
Neeraj Kumar Singh

Specific Test Tools
Performance Testing Tools
A wide range of measurements are taken by a performance test tool to enable analysis during or after execution of the
test. Typical metrics taken and reports provided include:
 Number of simulated users throughout the test
 Number and type of transactions generated by the simulated users and the arrival rate of the transactions
 Response times to particular transaction requests made by the users
 Reports and graphs of load against response times
 Reports on resource usage (e.g., usage over time with minimum and maximum values)
Significant factors to consider in the implementation of performance test tools include:
 The hardware and network bandwidth required to generate the load
 The compatibility of the tool with the communications protocol used by the system under test
 The flexibility of the tool to allow different operational profiles to be easily implemented
 The monitoring, analysis and reporting facilities required
Performance test tools are typically acquired rather than developed in-house due to the effort required to develop them.
Neeraj Kumar Singh

Specific Test Tools
Tools for Web-Based Testing
A variety of open source and commercial specialized tools are available for web testing. The following list shows the
purpose of some of the common web-based testing tools:
 Hyperlink test tools are used to scan and check that no broken or missing hyperlinks are present on a web site
 HTML and XML checkers are tools which check compliance to the HTML and XML standards of the pages that are
created by a web site
 Load simulators to test how the server will react when large numbers of users connect
 Lightweight automation execution tools that work with different browsers
 Tools to scan through the server, checking for orphaned (unlinked) files
 HTML specific spell checkers
 Cascading Style Sheet (CSS) checking tools
 Tools to check for standards violations e.g., Section 508 accessibility standards in the U.S. or M/376 in Europe
 Tools that find a variety of security issues
Neeraj Kumar Singh

Specific Test Tools
Tools for Web-Based Testing
The following are good sources of open source web testing tools
 The World Wide Web Consortium (W3C) [Web-3] This organization sets standards for the Internet and supplies a
variety of tools to check for errors against those standards.
 The Web Hypertext Application Technology Working Group (WHATWG) [Web-5]. This organization sets HTML
standards. They have a tool which performs HTML validation [Web-6].
Some tools that include a web spider engine can also provide information on the size of the pages and on the time
necessary to download them, and on whether a page is present or not (e.g., HTTP error 404). This provides useful
information for the developer, the webmaster and the tester.
Neeraj Kumar Singh

Specific Test Tools
Tools to Support Model-Based Testing
Model-Based Testing (MBT) is a technique whereby a formal model such as a finite state machine is used to describe the
intended execution-time behavior of a software-controlled system. Commercial MBT tools often provide an engine that
allows a user to “execute” the model. Interesting threads of execution can be saved and used as test cases. Other
executable models such as Petri Nets and Statecharts also support MBT.
MBT models (and tools) can be used to generate large sets of distinct execution threads. MBT tools also can help reduce
the very large number of possible paths that can be generated in a model. Testing using these tools can provide a
different view of the software to be tested. This can result in the discovery of defects that might have been missed by
functional testing.
Neeraj Kumar Singh
Petri Nets State Charts

Specific Test Tools
Component Testing and Build Tools
While component testing and build automation tools are developer tools, in many instances, they are used and
maintained by Technical Test Analysts, especially in the context of Agile development.
Component testing tools are often specific to the language that is used for programming a module. For example, if Java
was used as the programming language, JUnit might be used to automate the unit testing.
Debugging tools facilitate manual component testing at a very low level, allowing developers and Technical Test Analysts
to change variable values during execution and step through the code line by line while testing.
Build automation tools often allow a new build to be automatically triggered any time a component is changed. After the
build is completed, other tools automatically execute the component tests. This level of automation around the build
process is usually seen in a continuous integration environment.
Neeraj Kumar Singh

Specific Test Tools
Tools to Support Mobile Application Testing
Simulators
A mobile simulator models the mobile platform’s runtime environment. Applications tested on a simulator are compiled
into a dedicated version, which works in the simulator but not on a real device. Simulators are sometimes used as
replacements for real devices in testing. However, the application tested on a simulator differs from the application that
will be distributed.
Emulators
A mobile emulator models the hardware and utilizes the same runtime environment as the physical hardware.
Applications compiled to be deployed and tested on an emulator could also be used by the real device.
Common Aspects
Simulators and emulators are useful in the early stage of development as these typically integrate with development
environments and allow quick deployment, testing, and monitoring of applications. Using an emulator or simulator
requires launching it, installing the necessary app on it and then testing the app as if it were on the actual device. Each
mobile operating system development environment typically comes with its own bundled emulator and simulator. Third
party emulators and simulators are also available.
Neeraj Kumar Singh

Exam Pattern
Neeraj Kumar Singh

Sample Questions
1. Which of the following describes a common technical issue that causes automation projects to fail to achieve
the planned return on investment?
Select ONE Option
A. Elimination of duplication of information across tools.
B. Removal of manual checking of data exchanges between tools.
C. Use of an integrated development environment to simplify integration between tools.
D. Lack of separation between code and changeable data in the testware.
Neeraj Kumar Singh

Sample Questions
2. Which of the following statements about fault seeding tools is correct?
Select ONE Option
A. These tools insert defects into the source code to test the input checking capabilities of the software.
B. These tools insert defects into the source code to check the level of fault tolerance of the software.
C. These tools insert defects into the source code to test the effectiveness of the test suite.
D. These tools are generally used by the developer.
Neeraj Kumar Singh

Sample Questions
3. Which of the following statements best captures the difference between emulators and simulators when used in
a mobile application testing context?
Select ONE Option
A. A mobile emulator models the mobile platform’s runtime environment and a simulator utilizes the same
runtime environment as the physical hardware.
B. Applications compiled to be deployed and tested on a simulator could be also used by the real device. This is
not the case for emulators.
C. Simulators are useful in the early stage of development and emulators are useful in the later stages.
D. Emulators and simulators allow the setting of various usage parameters
Neeraj Kumar Singh

By
NeerajKumar
Chapter-wise Break Up
Chapter 1 – 2 Questions
ISTQB Advance
Technical Test Analyst
All the
Best!!

Chapter 6 - Test Tools and Automation

More Related Content

What's hot

What's hot (20)

Similar to Chapter 6 - Test Tools and Automation

Similar to Chapter 6 - Test Tools and Automation (20)

More from Neeraj Kumar Singh

More from Neeraj Kumar Singh (20)

Recently uploaded

Recently uploaded (20)

Chapter 6 - Test Tools and Automation