Testing on Thin Ice: Chipping Away at Test Unpredictability

Testing on Thin Ice Chipping Away at Test Unpredictability

DevNexus 2025 | Testing on Thin Ice LOTS OF RERUN TESTS Brian Demers François Martin Java Champion Developer Advocate @bdemers Senior Full Stack Software Engineer @martinfrancois

DevNexus 2025 | Testing on Thin Ice $36 Billion /year (USD) Loss Productivity $40 Billion End World Hunger by 2030 🦋bdemers.io | bdemers /year (USD)

DevNexus 2025 | Testing on Thin Ice Topics What is a Flaky Test? How to find them How to deal with them How to fix them Slides & eBook 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice What is a Flaky Test? No Flake News! 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Definition Test that exhibits both a passing and failing test result with the same code if run multiple times 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Timing related Example: 1. 2. 3. 4. Enter search term into textbox Click on button “Search” Wait for one second Click on first result 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Test data related Examples: ● Input field for prename doesn’t allow special characters ○ Some randomly generated prenames in tests contain special characters (“François”) ● Tests reliant on predefined test data in the database ○ Dependencies between tests ○ Failing tests cause inconsistencies 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Infrastructure related Unstable infrastructure Tests time out Most difficult to reproduce Examples: ● Build runners under heavy load ● Microservices with networking issues 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Resource contention related Mostly when running tests in parallel Examples: Multiple tests trying to… ● … write to the same file at the same time ○ File lock ○ Concurrent modification errors ● … create a user with the same username at the same time ○ Unique constraint violations 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Why Worry about Flaky Tests? ● Re-running tests wastes a lot of time ● Repetitive investigation efforts to determine if flaky or legitimate failure ● Team starts losing trust in test results ● Bad software quality ● Pilots frequently ignoring an alarm in cockpits of a Boeing 737 resulted in fatal plane crash in 2005 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Unit Tests vs Integration Tests UT vs IT 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice End-to-End Tests Integration Tests Unit Tests Testing Pyramid 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Unit Tests Integration Tests Flaky Foundations End-to-End Tests 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Complexity Flakiness 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice How to find Flaky Tests Flake Busters 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Running tests in parallel Can reveal many issues: ● Resource contention ● Dependency on test data ● Dependency on test execution order Side benefit: Tests execute faster ● Playwright & WebdriverIO: parallel by default ● Cypress: only in cloud (paid) ● JUnit and Gradle: through configuration 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Rerun Tests Most test frameworks allow for executing a test multiple times. Slides & eBook 🦋bdemers.io | bdemers

Maven Surefire Plugin DevNexus 2025 | Testing on Thin Ice <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <rerunFailingTestsCount>3//rerunFailingTestsCount> <failOnFlakeCount>0//failOnFlakeCount> </configuration> </plugin> 🦋bdemers.io | bdemers

Rerun Gotcha Increased time to fail 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Dealing with Flaky Tests 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Prevention In Pull / Merge Requests, run new or changed tests 3, 10, 50, or 1000 times (depending on resources and test type) Alternatively run all tests multiple times nightly 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Tell Your Team! Don’t let them waste time Uncover deeper issues Saves morale 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice New Sprint Checklist Fix a flaky test Pull a new issue 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Flaky Test Days 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Temporarily Quarantine Tests // JUnit @Disabled(“quarantine: reason or link to issue here”) @Test void flakyTest() { } // Jest </ quarantine: reason or link to issue here it.skip(‘should throw an error’, () <> { expect(response).toThrowError(expected_error); }); 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Quarantine ● Quarantine test after first failure ● Fix the quarantined tests timely ● Set a limit for tests in quarantine ● Define time limit for quarantined tests 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Delete Flaky Tests Does the test provide value? Can it be fixed? Keeping a flaky test provides negative value 🦋bdemers.io | bdemers

SSH into CI Agent $ ssh ci-agent cd /your-project # rerun build ./mvnw clean install 🦋bdemers.io | bdemers DevNexus 2025 | Testing on Thin Ice

DevNexus 2025 | Testing on Thin Ice How to fix Flaky Tests Defeating Murphy’s Law 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Test Data Consistency // JUnit 5 // Jest @BeforeEach void setup() { </ Create test data } beforeEach(() <> { </ Create test data }); @AfterEach void teardown() { </ Delete test data } afterEach(() <> { </ Delete test data }); 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Test Data Consistency // Using Testcontainers </ build.gradle testImplementation “org.testcontainers:postgres:1.19.8” testImplementation “org.testcontainers:junit-jupiter:1.19.8” </ JUnit Test @Testcontainers public class IntegrationTest { @Container PostgreSQLContainer<?> postgresContainer = new PostgreSQLContainer<>(“postgres:16.3”); } 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Ports Source Data Tests Data 🦋bdemers.io | bdemers Isolate Tests

DevNexus 2025 | Testing on Thin Ice Don’t Reuse System Resources ● Setting System Properties System.setProperty(“key”, “value”); ● Use unique files for each test temp directories, output files, etc. ● Use a random free ephemeral port (49152 - 65535) new ServerSocket(0).getLocalPort() 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice In E2E Tests ● Use deep links to navigate directly to the page to be tested ● Start the test with the test user already logged in ● Make it possible to run each test spec independently ● Do not wait in predefined time intervals, wait for a condition to be fulfilled with a timeout ● Use stable test ids in selectors ● Consider mocking external APIs 🦋fmartin.ch | FrançoisMartin

DevNexus 2025 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Convert ITs and E2E into UTs Flakiness 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Convert ITs and E2E into UTs Stability 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice Fix the Culture of Flaky Tests ● Brown Bag / Lunch & Learn ● Make flaky tests visible ● Track them ● Use manager terms 💰 🦋bdemers.io | bdemers

DevNexus 2025 | Testing on Thin Ice THANKS! QUESTIONS? bdemers.io fmartin.ch bdemers FrançoisMartin CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik and images generated by OpenAI’s DALL-E model via ChatGPT Slides & eBook