Metrics for Test Reporting: Analysis and Reporting for Effective Test Management

Preface....5

Contents....12

1 Definition, History, and Benefits of Metrics....14

Abstract....14

1.1 Criteria for Metrics....15

1.2 Quality Criteria for Metrics....15

1.3 Success Criteria for Metrics....16

1.4 Validation of Measures....16

1.5 Problems in the Software Development Process....17

1.6 History of Test Metrics....17

1.7 Steps to Set Up Test Metrics....18

1.8 Lifecycle of Test Metrics....19

References....19

2 Classification of Metrics....20

Abstract....20

2.1 Key Figures....20

2.2 Indicators ....22

2.3 Classification of Test Metrics....22

2.4 Classification According to IEEE Standard 1061....26

References....28

3 Complexity Measures....29

Abstract....29

3.1 Evaluation of Project Complexity ....29

3.2 Calculation of Project Complexity....31

3.3 Halstead Metric....32

3.4 Number of Delivered Bugs (B)....36

3.5 McCabe Metric (Cyclomatic Complexity)....37

3.6 Complexity Measure According to Rechenberg....39

3.7 Maintainability Index....40

3.8 Hybrid Metrics....40

3.9 Object-Oriented Software Metrics....41

3.10 Number of Children (NOC)....42

3.11 RFC (Response for a Class)....42

3.12 CBO (Coupling Between Objects)....42

3.13 CBC (Coupling Between Classes)....43

3.14 LCOM (Lack of Cohesion in Methods)....43

3.15 WMC (Weighted Methods per Class)....43

3.16 Kiviat Diagrams....43

3.17 Metrics for Verifying Test Design and Evaluating the Test Process....44

References....45

4 Standards and Norms for Measurements....47

Abstract....47

4.1 PSM—Practical Software and Systems Measurement....47

4.2 ISO/IEC15939 Software Measurement Process....48

4.3 ISO14598....48

4.4 CMMI Process Area Measurement and Analysis....49

4.5 Measurement with ISO15504 and Automotive SPICE ®....49

4.6 Analysis Methods....51

4.7 Analysis Tools....52

4.8 Quality Improvement Paradigm (QIP Model)....56

References....57

5 Use of Different Representations....58

Abstract....58

5.1 Axis Diagrams....58

5.2 Block Diagrams....63

5.3 Design of Reporting....63

References....64

6 Application of Metrics....65

Abstract....65

6.1 Metrics and Software Industrialization....67

References....69

7 Software Metrics....70

Abstract....70

8 Metrics for Software Quality....72

Abstract....72

8.1 Boehm’s Property Tree....72

8.2 Function Measurement according to Gilb....74

8.3 McCall’s Quality Matrix....76

8.4 Automated Quality Measurement....78

References....79

9 Requirement-Based Test Metrics....80

Abstract....80

9.1 Requirements/specified requirements....81

10 Metrics for Effort Estimation....84

Abstract....84

10.1 Evaluation of Testability and Test Complexity....84

References....88

11 Metrics for Test Preparation and Follow-up....89

Abstract....89

12 Metrics for Test Planning....91

Abstract....91

References....94

13 Metrics for Different Test Levels....95

Abstract....95

13.1 Module Level Testing....95

13.2 Integration Level Testing....96

13.3 System Level Testing....96

14 Metrics for Measuring Test Coverage....99

Abstract....99

14.1 Concept Coverage....99

14.2 User Manual Coverage....99

14.3 Code Coverage....100

14.4 Design Coverage....100

14.5 Module Coverage....100

14.6 Architecture Coverage....100

14.7 Data Coverage....101

14.8 Function Coverage....101

14.9 Test Coverage Metrics....102

14.10 Test Case Coverage....102

14.11 Test Case Completeness....103

14.12 Lines of Code....103

14.13 Reliability as a Central Test Metric....105

14.14 Function Point Analysis (FPA)....105

14.15 Use Case Points....108

14.16 Test Case Points....109

14.17 Systematic and Deterministic Testing....110

14.18 Test Coverage Metrics....112

14.19 Stochastic Testing....113

References....113

15 Test Case-Based Metrics....115

Abstract....115

15.1 Chapin's Q Metric....116

15.2 Test Data Density....117

15.3 Test Data Volume....117

15.4 Test Case Efficiency....118

15.5 Test Case Reusability....118

15.6 Test Coverage....118

15.7 Test Conformity....118

15.8 Test Design Ratio....118

15.9 Test Coverage....119

References....120

16 Test Object-Based Metrics....121

Abstract....121

16.1 Code Coverage....121

16.2 Dialogue Coverage....122

16.3 Installation Variants....122

16.4 Coverage Measures....122

16.5 Instruction Coverage....123

16.6 Decision/Branch Coverage....123

16.7 Path Coverage....123

16.8 Condition Coverage....124

16.9 Tests in the Black-Box Method....125

16.10 Test Progress Metric....127

References....129

17 Metrics for Test Execution....130

Abstract....130

References....133

18 Metrics for Performance and Load Tests....134

Abstract....134

18.1 Requests per Second....134

18.2 Processor Utilization....135

18.3 Queued Requests....135

References....137

19 Metrics for Test Automation....138

Abstract....138

20 Cost-Based Test Metrics....140

Abstract....140

20.1 Test Effort....141

21 Test Productivity Metrics....143

Abstract....143

References....145

22 Goal Question Metric....146

Abstract....146

22.1 ISO9126....149

References....150

23 Defect-Based Test Metrics....152

Abstract....152

23.1 Error Classes....153

23.2 Defects Over Time....154

23.3 Metrics for Risks....154

23.4 Defect Reduction Forecasts....155

23.5 Examples of Defect Reduction Metrics....157

23.6 Error Seeding Method....160

23.7 Running Average of Errors....161

23.8 Defects by Project Phase....163

23.9 Test Quality Metric....164

23.10 Confidence Level....165

23.11 Escape Defects....166

References....168

24 Metrics for Test Documents....169

Abstract....169

24.1 Reviews....169

24.2 Status of the Documents....170

25 Metrics for Usability Tests....172

Abstract....172

25.1 A/B Testing and Multivariate Testing....175

References....177

26 TPI Next....178

Abstract....178

26.1 Criteria for Maturity Levels....181

26.2 Test Maturity Matrix....182

26.3 Maturity Levels per Criterion....184

26.4 Metrics in TPI Next....187

References....190

27 Metrics for Test Completion Criteria....191

Abstract....191

27.1 Defects per Unit of Time....192

References....194

28 Metrics and Test Reporting....195

Abstract....195

28.1 Perspectives on Test Reporting....195

28.2 Period of Consideration....196

28.3 Adjusting History of Metrics....197

28.4 Metric Defect....197

28.5 Problems with Test Progress Consideration....199

28.6 Other Viewing Periods....199

28.7 Recommended Reporting Rhythm....199

28.8 Summary of Metrics in a Metric....201

28.9 Other Metrics....203

28.10 Project Traffic Light....203

References....205

29 Comparison of Test Metrics....206

Abstract....206

29.1 Comparison of Different Metrics....207

29.2 Standardization of Test Metrics....207

29.3 Metrics No Longer Needed....209

29.4 Increasing Data Volume....209

References....211

30 Problem Solving for the Use of Test Metrics in Test Operations....212

Abstract....212

30.1 Number of Metrics Used....212

30.2 Selection of Metrics....213

30.3 Extension of Metrics....214

30.4 Creating Realistic Forecasts....214

30.5 Subsequent Change of Values....215

30.6 Evaluation Framework for Metrics....215

30.7 Meaningless Correlations....216

References....218

31 Impact of Reporting with the Help of Test Metrics....219

Abstract....219

31.1 Manipulative Effect of Statistics....226

31.2 Relative Values....226

31.3 Arithmetic Mean....226

31.4 Manipulated Graphics....226

Afterword....230

Weiterführende Literatur....235

Part of any software development is to regularly inform management about the progress and any problems of the project. This book presents various test parameters and metrics that can be used to vividly illustrate the progress of a software test and easily identify any need for action. The key parameters and index values that are essential for successful reporting are explained in detail.

Starting with the historical development of test reporting, the author explains the fundamental benefits of metrics and provides an overview of the different types of metrics and how they can be used effectively in software testing. He shows how a particular metric can be individually adapted to the software to be tested, starting with the test specification and test execution through to the development of test coverage, and how this can lead to test automation. He also presents the advantages and disadvantages of those test metrics that are based on defects, i.e. that measure the number of defects, the defect density and the development of defects over time.

With this detailed examination of test reporting, the author provides an optimal basis for evaluation, which not only allows the assessment of which test metric should be used for which purpose or which individual project situation, but also what the particular problems of an individual metric for test reporting may be and how these problems can best be solved. This practical guide is therefore primarily aimed at employees in IT projects, such as project managers, software testers and developers, but business and technology consultants as well as lecturers at colleges and universities will also find an exciting insight into various software testing methods.