A presentation at Clermont’ech - API Hour #72 in in Clermont-Ferrand, France by Romain Rochegude
Romain Rochegude Software Architect IT Technical Manager RoRoche
“Program testing can be used to show the presence of bugs, but never to show their absence!” — Edsger W. Dijkstra
Who here believes they have good test coverage? 70%? 80%? 90%?
And now, who is confident that their tests actually catch real bugs?
The False Sense of Safety ✅ Green tests ✅ CI pipeline is fine ✅ High coverage 🐛 And yet… bugs still reach production
Our tests pass… but do they really test anything?
What if, instead of testing our code, we started testing our tests?
🧯 The Problem: What Coverage Doesn’t Tell You Coverage ≠ Test Quality Coverage measures: executed lines not validated behavior A test suite can look “good” even if: assertions are too weak edge cases are missing important conditions are never challenged
Coverage answers “what was executed?” It does not answer “what is actually protected?” 👉 Code coverage is comforting — but it proves nothing.
🤔 The Real Question “If I break the code… will my tests notice?”
✅ The Solution: Mutation Testing What Is Mutation Testing? Mutation testing is about: Intentionally breaking the production code to check whether the tests fail Or more bluntly: “If your tests don’t fail, they’re useless.”
🔬 How It Works 1. 🧬 Inject mutants into the production code conditional changes boolean inversions arithmetic operator changes 2. ▶️ Re-run the unit tests 3. 📊 Analyze the results ❌ Mutant killed → ✅ test is effective ✅ Mutant survives → 🚨 test is weak or missing 👉 We are no longer verifying code execution, but behavior protection.
📈 The Mutation Score Mutation score = percentage of mutants killed by the test suite This gives us a new signal: Not “did tests run?” But “did tests react?” A higher mutation score means more meaningful tests.
🖥️ Live Demo — Making Tests Fail on Purpose Goal: prove that passing tests do NOT necessarily mean protected code
💡 Concrete Benefits Why Mutation Testing Is Powerful ✅ Evaluate test robustness ✅ Identify missing or insufficient tests ✅ Improve the overall quality of the test suite ✅ Go beyond raw code coverage Coverage tells you where to look. Mutation testing tells you what to fix.
🎁 Positive Side Effects Better test design Cleaner, simpler production code Less dead or defensive logic More valuable technical discussions 💬 “Why does this mutant survive?” ➡️ A great question about design, intent, and architecture.
⚠️ Limitations The Main Drawback ⏱️ Execution time Tests are executed many times Computationally expensive Not suitable for all code, all the time Mutation testing forces trade-offs.
🧠 Use It Smartly ✅ Apply mutation testing on: Critical business logic Core domain rules Algorithms ✅ Run it: Periodically On dedicated CI jobs During refactoring or hardening phases You don’t need a perfect score. An imperfect but understood mutation score is far better than perfect but misleading coverage.
🏁 Conclusion ❌ Testing code execution is not enough ✅ We must test the effectiveness of our tests Mutation testing doesn’t replace your tests. It reveals whether they actually matter.
🚀 Call to Action Key takeaways: Coverage reassures Mutation testing challenges Quality improves when we embrace discomfort “Your tests pass? Good. Now let’s try to make them fail.”
Thank you Powered by
“Program testing can be used to show the presence of bugs, but never to show their absence!” — Edsger W. Dijkstra
Automated tests give us confidence in our applications’ behavior, but do they really cover what matters?
What if we tested… our tests?
This talk offers a practical introduction to mutation testing, illustrated by a real-world case where mutation testing uncovered a weakness in the unit testing framework. It also highlights the resulting improvements in software quality and confidence.
