Test Case Point Analysis
- 2. Agenda
• Objectives
• Background and Motivation
• Existing Estimation Methods
• Test Case Point Analysis
• Effort Estimation Methods using TCP
• Conclusion and Future Work
2
- 3. Objectives
• Discuss importance of software estimation,
especially for testing projects
• Discuss existing methods for estimating testing
activities
• Introduce an approach to estimating testing
projects
– Test Case Point Analysis
– Effort Estimation using TCP
3
- 4. Agenda
• Objectives
• Background and Motivation
• Existing Estimation Methods
• Test Case Point Analysis
• Effort Estimation Methods using TCP
• Conclusion and Future Work
4
- 5. Software Estimation and Its Importance
• Software estimation
– process of determining the cost, time, staff, and other related
attributes of software projects, often before work is performed
• Estimation is important for the success or failure of software
projects
• Provides inputs for
– Making investment decisions
– Budget and staff allocation
– Tradeoff and risk analysis
– Project planning
– Stakeholder negotiation
– etc.
5
- 6. Popular Software Estimation Methods
• Sizing Methods
– Source Lines of Code (SLOC)
– Function Points Analysis
– Use Case Points
• Effort Estimation Methods
– Expert judgment/experience
– Productivity index
– COCOMO, SEER-SIM, SLIM models
• Software testing estimation methods
– Using a test distribution percentage
– Test Case Points Analysis (by Cognizant Technology Solutions) [2]
• Mainly based on the number of steps
6
- 7. Motivation
• Testing accounts for up to 50% of project effort [1]
• Current problems
– estimates are done for the whole project rather than
testing specific
– lack of reliable methods designed for estimating size and
effort of software testing
– vague definitions of testing productivity
• due to the lack of a size measure for software testing
• There are needs of
– accurately estimating effort of testing activities
– measuring size and productivity of testing activities
– measuring effectiveness and efficiency of software
testing
7
- 8. Motivation (cont’d)
• Our aim to
– Defining a method for estimating the size of testing
activities
– Defining methods to estimate testing effort, schedule,
and staff accurately using this size measure
– Using this size measure as a basis for measuring
• productivity
• effectiveness
• other testing metrics
8
- 9. Agenda
• Objectives
• Background and Motivation
• Existing Estimation Methods
• Test Case Point Analysis
• Effort Estimation Methods using TCP
• Conclusion and Future Work
9
- 10. Test Case Point Analysis
• Principles
– Size must reflect the mass and complexity of the testing project
– Size should correlate with testing effort
– Test case point is measured using test cases as main input
– Test case complexity is based on
• Checkpoints
• Precondition
• Test Data
• Type of test case
10
- 11. Test Case Point Analysis (cont’d)
• Process overview
Project
Attributes
Test Case Estimate Estimated
Test Case Test Case
Points Testing Effort
Point Analysis
Effort
11
- 12. Test Case Point Analysis (cont’d)
• TCPA estimates the size of testing projects using test
cases as input
• Steps
1. Measure test case complexity
• Count checkpoints
• Measure the complexity of precondition
• Measure the complexity of data
2. Adjust Test Case Point by type of test
• Consider 11 types of test case
o Each type has a weight
o Adjust TCP using the weight
• Use functional test cases as baseline with the weight of 1.0
12
- 13. Checkpoint
• Checkpoint
– Is the condition in which the tester verifies whether the
result produced by the target function matches the
expected criterion
– One test case consists of one or many checkpoints
• Counting rule 1
One checkpoint is counted as one Test Case Point
13
- 14. Test Case Precondition
• Precondition
– specifies the condition to execute the test case
– mainly affects the cost to execute the test case
– may be related to data prepared for the test case
• Four levels of complexity
– None
– Low
– Medium
– High
14
- 15. Test Case Precondition (cont’d)
• None
– The precondition is not applicable or important to execute the test case
– Or, the precondition is just reused from the previous test case to continue
the current test case
• Low
– The condition for executing the test case is available with some simple
modifications required
– Or, some simple set-up steps are needed
• Medium
– Some explicit preparation is needed to execute the test case
– The condition for executing is available with some additional modifications
required
– Or, some additional set-up steps are needed
• High
– Heavy hardware and/or software configurations are needed to execute the
test case
15
- 16. Test Case Precondition (cont’d)
• Counting rule 2A: Counting Unadjusted Test Case Points for
Precondition:
Each complexity level of precondition is assigned a
number of Test Case Points
Based on our survey of 18 senior QA engineers
16
- 17. Test Data
• Test Data
– used to execute the test case
– can be generated at the test case execution time, sourced from
previous tests, or generated by test scripts
– Test data is test case specific, or general to a group of test cases, or
for the whole system
• Four levels of complexity
– None
– Low
– Medium
– High
17
- 18. Test Data (cont’d)
• None
– No test data preparation is needed
• Low
– Simple test data is needed and can be created during the test case execution time
– Or, the test case uses a slightly modified version of existing test data and requires little
or no effort to modify the test data
• Medium
– Test data is deliberately prepared in advance with extra effort to ensure its
completeness, comprehensiveness, and consistency
• High
– Test data is prepared in advance with considerable effort to ensure its completeness,
comprehensiveness, and consistency
– This could include using support tools to generate data and a database to store and
manage test data
– Scripts may be required to generate test data
18
- 19. Test Data (cont’d)
• Counting rule 2B: Counting Unadjusted Test Case Points for
Test Data:
Each complexity level of Test Data is assigned a
number of Test Case Points
Based on our survey of 18 senior QA engineers
19
- 20. Adjust Test Case Point
• Test Case Point counted till this point is considered
Unadjusted Test Case Point (UTCP)
• UTCP is adjusted by considering types of test case
– Each type of test case is assigned a weight
– Adjusted Test Case Point (ATCP):
n
ATCP = ∑UTCPi * Wi
i=1
• UTCPi - the number of UTCP counted for the test case ith.
• Wi - the weight of the test case ith, taking into account its test type
20
- 21. Weight by Type of Test
Based on our survey of 18 senior QA engineers
21
- 23. Test Case Point Analysis - Summary
• Process overview
Count
Checkpoints
Determine
Test Case UTCP Adjust with TCP
Precondition
Test Type
Complexity
Determine
Test Data
Complexity
23
- 24. Agenda
• Objectives
• Background and Motivation
• Existing Estimation Methods
• Test Case Point Analysis
• Effort Estimation Methods using TCP
• Conclusion and Future Work
24
- 25. Estimate Testing Effort
• Estimate testing effort using TCP
• Test effort distribution, four phases
– Test Planning
– Test Analysis and Design Each of these phases may be
performed multiple times
– Test Execution
– Test Tracking and Reporting
• Effort estimation methods
– Productivity index
– Regression
25
- 26. Estimate Testing Effort (cont’d)
• When estimating TCP, assume that each phase is
performed ONCE
• Effort is estimated dependent on how many times a
phase is performed
26
- 27. Productivity Index
• Effort is computed using productivity index of
completed project
• Productivity index is measured as person-month per
TCP
Effort = TCP * Productivity Index
Simple method
27
- 28. Regression
• Estimate effort of new projects using size and effort
of completed projects
100
90
Regression
80
formula
70
y = 0.072x + 1.640
Effort (PM)
60
50
40
30
20
10
0
0 100 200 300 400 500 600 700 800 900 1000
Adjusted TCP
28
- 29. Tools Support by QASymphony [3]
• qTest – Quality Management Suite
– Manage requirements and test cases
– Plan testing activities and resources
– Monitor and control testing activities
– Perform measurement, analysis, and reporting
• Test case point analysis
• Effort estimation
• qTrace – defect and scenario recording tool
– Record defect
– Record user scenarios
29
- 30. Conclusion and Future Work
• It is important to estimate size and effort of testing
projects
• Test Case Point Analysis (TCPA) was proposed to
estimate size. Elements include
– Checkpoints
– Precondition
– Test Data
– Test Type
• Simple two methods for estimating effort were
introduced
– Productivity index
– Regression
30
- 31. Conclusion and Future Work
• Advantages of TCPA
– Easy to implement
– Reflecting real complexity of test cases
– Independent with the number of steps
• Limitations and future work
– Need empirical validations
– Recalibrate constants used in the method
– Support only manual testing
– Lack of support for performance testing
31
- 33. References
• [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software
maintenance effort, International Conference on Software Maintenance, 2008, pp. 456-459
• [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant
Technology Solutions, White Paper, 2001
• [3] QASymphony: www.qasymphony.com