Fundamental test process

FUNDAMENTAL TEST
PROCESS
By Graham et.al
(2011)
UNIVERSITAS SULTAN SYARIF
KASIM RIAU
Alex Swandi

In this section, we will describe the fundamental test
process and activities. These start with test planning and
continue through to test closure. For each part of the test
process, we'll discuss the main tasks of each test activity.
In this section, you'll also encounter the glossary terms
confirmation testing, exit criteria, incident, regression
testing, test basis, test condition, test coverage, test
data, test execution, test log, test plan, test strategy,
test summary report and testware.
Introduction

As we have seen, although executing tests is important, we also need a plan of
action and a report on the outcome of testing. Project and test plans should
include time to be spent on planning the tests, designing test cases, preparing for
execution and evaluating status. The idea of a fundamental test process for all
levels of test has developed over the years. Whatever the level of testing, we see
the same type of main activities happening, although there may be a different
amount of formality at the different levels, for example, component tests might
be carried out less formally than system tests in most organizations with a less
documented test process. The decision about the level of formality of the
processes will depend on the system and software context and the level of risk
associated with the software. So we can divide the activities within the
fundamental test process into the following basic steps:
 planning and control;
 analysis and design;
 implementation and execution;
 evaluating exit criteria and reporting;
 test closure activities.
Cont…

During test planning, we make sure we understand the goals and
objectives of the customers, stakeholders, and the project, and the
risks which testing is intended to address. This will give us what is
sometimes called the mission of testing or the test assignment. Based
on this understanding, we set the goals and objectives for the testing
itself, and derive an approach and plan for the tests, including
specification of test activities. To help us we may have organization or
program test policies and a test strategy. Test policy gives rules for
testing, e.g. 'we always review the design documents'; test strategy is
the overall high-level approach, e.g. 'system testing is carried out by an
independent team reporting to the program quality manager. It will be
risk-based and proceeds from a product (quality) risk analysis' (see
Chapter 5). If policy and strategy are defined already they drive our
planning but if not we should ask for them to be stated and defined.
Test planning has the following major tasks, given approximately in
order, which help us build a test plan:
Test planning and
control
1

 Determine the scope and risks and identify the
objectives of testing: we consider what software,
components, systems or other products are in scope for
testing; the business, product, project and technical
risks which need to be addressed; and whether we are
testing primarily to uncover defects, to show that the
software meets requirements, to demonstrate that the
system is fit for purpose or to measure the qualities and
attributes of the software.
 Determine the test approach (techniques, test items,
coverage, identifying and interfacing with the teams
involved in testing, testware): we consider how we will
carry out the testing, the techniques to use, what needs
testing and how extensively (i.e. what extent of
coverage). We'll look at who needs to get involved and
when (this could include developers, users, IT infrastruc
Cont…

 Determine the required test resources (e.g. people, test
environment, PCs): from the planning we have already
done we can now go into detail; we decide on our team
make-up and we also set up all the supporting hardware
and software we require for the test environment.
 Schedule test analysis and design tasks, test
implementation, execution and evaluation: we will
need a schedule of all the tasks and activities, so that
we can track them and make sure we can complete the
testing on time.
 Determine the exit criteria: we need to set criteria
such as coverage criteria (for example, the percentage
of statements in the software that must be executed
during testing) that will help us track whether we are
completing the test activ ities correctly. They will show
us which tasks and checks we must complete for a
Cont…

Test analysis and design is the activity where general testing objectives are trans- formed into
tangible test conditions and test designs. During test analysis and design, we take general testing
objectives identified during planning and build test designs and test procedures (scripts). You'll
see how to do this in Chapter 4. Test analysis and design has the following major tasks, in
approximately the following order:
 Review the test basis (such as the product risk analysis, requirements, architecture, design
specifications, and interfaces), examining the specifications for the software we are
testing. We use the test basis to help us build our tests. We can start designing certain
kinds of tests (called black-box tests) before the code exists, as we can use the test basis
documents to understand what the system should do once built. As we study the test basis,
we often identify gaps and ambiguities in the specifications, because we are trying to
identify precisely what happens at each point in the system, and this also pre- vents
defects appearing in the code.
 Identify test conditions based on analysis of test items, their specifications, and what we
know about their behavior and structure. This gives us a highlevel list of what we are
interested in testing. If we return to our driving example, the examiner might have a list of
test conditions including 'behav ior at road junctions', 'use of indicators', 'ability to
maneuver the car' and so on. In testing, we use the test techniques to help us define the
test condi tions. From this we can start to identify the type of generic test data we might
need.
Test analysis and
design
2

 Design the tests (you'll see how to do this in Chapter 4), using techniques to help
select representative tests that relate to particular aspects of the soft ware which
carry risks or which are of particular interest, based on the test conditions and
going into more detail. For example, the driving examiner might look at a list of
test conditions and decide that junctions need to include T-junctions, cross roads
and so on. In testing, we'll define the test case and test procedures.
 Evaluate testability of the requirements and system. The requirements may be
written in a way that allows a tester to design tests; for example, if the per
formance of the software is important, that should be specified in a testable way.
If the requirements just say 'the software needs to respond quickly enough' that is
not testable, because 'quick enough' may mean different things to different people.
A more testable requirement would be 'the soft ware needs to respond in 5 seconds
with 20 people logged on'. The testabil ity of the system depends on aspects such
as whether it is possible to set up the system in an environment that matches the
operational environment and whether all the ways the system can be configured or
used can be understood and tested. For example, if we test a website, it may not
be possible to iden tify and recreate all the configurations of hardware, operating
system, browser, connection, firewall and other factors that the website might
encounter.
 Design the test environment set-up and identify any required infrastructure and
tools. This includes testing tools (see Chapter 6) and support tools such as
Cont…

During test implementation and execution, we take the
test conditions and make them into test cases and
testware and set up the test environment. This means that,
having put together a high-level design for our tests, we
now start to build them. We transform our test conditions
into test cases and procedures, other testware such as
scripts for automation. We also need to set up an envi-
ronment where we will run the tests and build our test
data. Setting up environments and data often involves
significant time and effort, so you should plan and monitor
this work carefully. Test implementation and execution
have the following major tasks, in approximately the
following order:
Test implementation and
execution
3

 Implementation:
 Develop and prioritize our test cases, using the techniques you'll see in Chapter 4, and create test data
for those tests. We will also write instructions for carrying out the tests (test procedures). For the
driving examiner this might mean changing the test condition 'junc tions' to 'take the route down
Mayfield Road to the junction with Summer Road and ask the driver to turn left into Summer Road and
then right into Green Road, expecting that the driver checks mirrors, signals and maneuvers correctly,
while remaining aware of other road users.' We may need to automate some tests using test harnesses
and automated test scripts. We'll talk about automation more in Chapter 6.
 Create test suites from the test cases for efficient test execution. A test suite is a logical collection of
test cases which naturally work together. Test suites often share data and a common high-level set of
objectives. We'll also set up a test execution schedule.
 Implement and verify the environment. We make sure the test envi ronment has been set up correctly,
possibly even running specific tests on it.
Cont…

 Execution:
 Execute the test suites and individual test cases, following our test proce dures. We might do this manually
or by using test execution tools, accord ing to the planned sequence.
 Log the outcome of test execution and record the identities and versions of the software under test, test
tools and testware. We must know exactly what tests we used against what version of the software; we
must report defects against specific versions; and the test log we keep provides an audit trail.
 Compare actual results (what happened when we ran the tests) with expected results (what we anticipated
would happen).
 Where there are differences between actual and expected results, report discrepancies as incidents. We
analyze them to gather further details about the defect, reporting additional information on the problem,
identify the causes of the defect, and differentiate between problems in the software and other products
under test and any defects in test data, in test documents, or mistakes in the way we exe cuted the test.
We would want to log the latter in order to improve the testing itself.
 Repeat test activities as a result of action taken for each discrepancy. We need to re-execute tests that
previously failed in order to confirm a fix (confirmation testing or re-testing). We execute corrected tests
and suites if there were defects in our tests. We test corrected software again to ensure that the defect
was indeed fixed correctly (confirmation test) and that the programmers did not introduce defects in
unchanged areas of the software and that fixing a defect did not uncover other defects (regression
testing).
Cont…

Evaluating exit criteria is the activity where test execution is assessed
against the defined objectives. This should be done for each test level,
as for each we need to know whether we have done enough testing.
Based on our risk assessment, we'll have set criteria against which
we'll measure 'enough'. These criteria vary for each project and are
known as exit criteria. They tell us whether we can declare a given
testing activity or level complete. We may have a mix of coverage or
completion criteria (which tell us about test cases that must be
included, e.g. 'the driving test must include an emergency stop' or 'the
software test must include a response measurement'), acceptance
criteria (which tell us how we know whether the software has passed or
failed overall, e.g. 'only pass the driver if they have completed the
emergency stop correctly' or 'only pass the software for release if it
meets the priority 1 requirements list') and process exit criteria (which
tell us whether we have completed all the tasks we need to do,
e.g. 'the examiner/tester has not finished until they have written and
filed the end of test report'). Exit criteria should be set and evaluated
for each test level. Evaluating exit criteria has the following major
tasks:
Evaluating exit criteria and
reporting
4

 Check test logs against the exit criteria specified in test planning:
We look to see what evidence we have for which tests have been
executed and checked, and what defects have been raised, fixed,
confirmation tested, or are out standing.
�� Assess if more tests are needed or if the exit criteria specified
should be changed: We may need to run more tests if we have not
run all the tests we designed, or if we realize we have not reached
the coverage we expected, or if the risks have increased for the
project. We may need to change the exit criteria to lower them, if
the business and project risks rise in impor tance and the product
or technical risks drop in importance. Note that this is not easy to
do and must be agreed with stakeholders. The test manage ment
tools and test coverage tools that we'll discuss in Chapter 6 help us
with this assessment.
 Write a test summary report for stakeholders: It is not enough
that the testers know the outcome of the test. All the stakeholders
need to know what testing has been done and the outcome of the
Cont…

During test closure activities, we collect data from completed test activities to
consolidate experience, including checking and filing testware, and analyzing
facts and numbers. We may need to do this when software is delivered. We also
might close testing for other reasons, such as when we have gathered the infor-
mation needed from testing, when the project is cancelled, when a particular
milestone is achieved, or when a maintenance release or update is done. Test
closure activities include the following major tasks:
 Check which planned deliverables we actually delivered and
ensure all incident reports have been resolved through defect
repair or deferral. For deferred defects, in other words those
that remain open, we may request a change in a future release.
We document the-acceptance or rejection of the software
system.
 Finalize and archive testware, such as scripts, the test
environment, and any other test infrastructure, for later reuse.
It is important to reuse whatever we can of testware; we will
inevitable carry out maintenance testing, and it saves time and
effort if our testware can be pulled out from a library of existing
tests. It also allows us to compare the results of testing between
software versions.
Test closure
activities
5

 Hand over testware to the maintenance organization who will support the
software and make any bug fixes or maintenance changes, for use in con
firmation testing and regression testing. This group may be a separate group to
the people who build and test the software; the maintenance testers are one
of the customers of the development testers; they will use the library of tests.
 Evaluate how the testing went and analyze lessons learned for future releases
and projects. This might include process improvements for the soft ware
development life cycle as a whole and also improvement of the test processes.
If you reflect on Figure 1.3 again, we might use the test results to set targets
for improving reviews and testing with a goal of reducing the number of
defects in live use. We might look at the number of incidents which were test
problems, with the goal of improving the way we design, execute and check
our tests or the management of the test environments and data. This helps us
make our testing more mature and cost-effective for the organization. This is
documented in a test summary report or might be part of an overall project
evaluation report.
Cont…

Fundamental test process

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Fundamental test process

Similar to Fundamental test process (20)

Recently uploaded

Recently uploaded (20)

Fundamental test process