Your test coverage is a lie!
•
1 like•640 views
Mutation testing in software development surfaced in academia during the 70's and has recently seen a resurgence in popularity as a legitimate tool in your testing arsenal. In this session we review the conventional testing pyramid, modern approaches to testing software and look at how mutation testing can help fill in those blind spots.
Report
Share
Related slideshows
Your test coverage is a lie!
- 1. The Cake Your Test Coverage is a Lie! Chris Shepherd © 2020 ThoughtWorks
- 2. Hi!
- 3. Photo by luis gomes from Pexels Testing!
- 6. Photo by Min An from Pexels
- 7. ● Common testing strategies ● Exciting errors (and where to find them) ● Mutation testing and its history ● Stryker Mutator - A fun framework for finding mutants ● Some other bits and bobs So what's the plan, Stan?
- 9. © 2020 ThoughtWorks© NASA
- 10. © NASA
- 11. © NASA
- 12. © NASA
- 13. © NASA
- 14. © Photo by Rebekah Howell on Unsplash
- 15. © NASA
- 18. Manual, Exploratory Tests Unit Tests Service Tests UI / E2E Tests
- 19. Unit Tests Service Tests UI / E2E Tests Manual, Exploratory Tests
- 20. Unit Tests Service Tests UI / E2E Tests Manual, Exploratory Tests
- 21. Unit Tests Service Tests UI / E2E Tests Manual, Exploratory Tests
- 23. Photo by Jake Davies on Unsplash
- 24. Photo by Giorgio Trovato on Unsplash Static Type System Unit Tests Integration Tests E2E Tests
- 25. Photo by Sanjay Shivakumar on Unsplash Integrated Tests Integration Tests Implementation Details
- 27. Photo by Somben Chea from Pexels Manual, Exploratory Tests Unit Tests :(
- 28. Photo by Sarah Kilian on Unsplash
- 32. Problem One The Oracle Problem
- 33. “In order to use testing to validate a program it is necessary to assume the existence of a test oracle which can be used to check the correctness of test output.” - William Howden, 1978
- 34. Problem Two Photo by Gustavo Fring from Pexels The Reliable Test Set Problem
- 35. ● Surfaced in 1971 ● An answer to validating a Reliable Test Set ● Seed defects in your software to create Mutants ● Re-evaluate your tests to verify their efficacy ● Profit In a Nutshell Photo by Dollar Gill on Unsplash
- 36. Photo by Jean-Baptiste Burbaud from Pexels The Coupling Effect Hypothesis.
- 39. execute(operation, args) { if (operation //= "ADD") { this.toDoList.push(args); } return this.toDoList; } it("Adds Items", () /> { const toDoList = new ToDoList(); const list = toDoList.execute("ADD", "Write the Tests"); expect(list).toEqual(["Write the Tests"]); });
- 40. File % Stmts % Funcs % Lines All files 100 100 100 utils.js 100 100 100
- 41. execute(operation, args) { if (operation //= "ADD") { this.toDoList.push(args); } return this.toDoList; } it("Adds Items", () /> { const toDoList = new ToDoList(); const list = toDoList.execute("ADD", "Write the Tests"); expect(list).toEqual(["Write the Tests"]); }); execute(operation, args) { if (true) { this.toDoList.push(args); } return this.toDoList; } <- Before After ->
- 42. File % Stmts % Funcs % Lines All files 100 100 100 utils.js 100 100 100
- 44. //. if (this.toDoList[i] != item) { return this.toDoList; } //. Decision Mutations //. if (true) { return this.toDoList; } //.
- 45. //. execute(operation, args) { if (operation //= "ADD") { this.toDoList.push(args); } return this.toDoList; } //. Statement Mutations //. execute(operation, args) { if (operation //= "ADD") { this.toDoList.push(args); this.toDoList.push(args); } return this.toDoList; } //.
- 46. … for (i = 0; i < toDoList.length; i/+) { if (this.toDoList[i] //= item) { return this.toDoList; } this.toDoList.push(item); } //. Value Mutations … for (i = 0; i /= toDoList.length; i/+) { if (this.toDoList[i] //= item) { return this.toDoList; } this.toDoList.push(item); } //.
- 47. Photo by Andrea Piacquadio from Pexels
- 48. Photo by Simon Berger on Unsplash
- 49. Photo by Federica Galli on Unsplash
- 50. Photo by Marvin Meyer on Unsplash
- 52. Photo by David Levêque on Unsplash Welcome To The Robo-Bar! http://qrco.de/bbZArc
- 53. $ git clone https://github.com/stryker-mutator/robobar-example $ cd robobar-example $ npm install $ npm test $ npm run stryker Stryker: Getting Started
- 54. Coverage Report
- 55. Photo by Jamie Street on Unsplash
- 56. Mutation Report
- 58. Photo by Element5 Digital from Pexels
- 59. Residual Defect Density (RDD) Photo by William Iven on Unsplash
- 60. Defect Count Total # Lines of Code
- 62. Using Mutation Testing to Estimate RDD Learned RDD X Mutation Score
- 63. Code Coverage is a Lie Mutation Scores Highlight Blindspots Residual Defect Density is a Thing Mutation Scores Can Predict RDD
- 64. Photo by Ian Espinosa on Unsplash
- 65. Photo by Ian Espinosa on Unsplash
- 66. Photo by bruno neurath-wilson on Unsplash
- 67. Photo by Lama Roscu on Unsplash
- 69. Photo by taylor hernandez on UnsplashPhoto by taylor hernandez on Unsplash
- 71. Photo by Brian McGowan on Unsplash
- 72. Chris Shepherd Ta © 2020 ThoughtWorks
- 73. References ● Metamorphic Testing: A Review of Challenges and Opportunities - Tsong Yueh Chen et al ● Investigations of the Software Testing Coupling Effect - A. Jefferson Offutt ● Mutant census: An empirical examination of the competent programmer hypothesis - Gopinath, Rahul et al ● Mutation Testing - Stuart Anderson ● Fault-Based Testing - Mauro Pezzè & Michal Young ● Fault-Based Testing Without the Need of Oracles - TY Chen ● Theoretical and Empirical Studies of Program Testing - William Howden ● https://stryker-mutator.io/ - This is really good, you should check it out! @the_sheps me@thesheps.dev thesheps.dev © 2020 ThoughtWorks
- 74. Continue the conversation on Slack © 2020 ThoughtWorks XConfEurope2020 xconfeurope2020.slack.com #talk6-your-test-coverage-is-a-lie #XConfOnline