Testing on Thin Ice: Chipping Away at Test Unpredictability

Testing on Thin Ice Chipping Away at Test Unpredictability Questions

JCON Europe 2024 | Testing on Thin Ice LOTS OF RERUN TESTS Brian Demers François Martin Java Champion Developer Advocate @bdemers Senior Full Stack Software Engineer @martinfrancois

JCON Europe 2024 | Testing on Thin Ice $36 Billion /year (USD) Loss Productivity $40 Billion End World Hunger by 2030 @BrianDemers | bdemers /year (USD)

JCON Europe 2024 | Testing on Thin Ice Topics What is a Flaky Test? How to find them How to deal with them How to fix them Questions @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice What is a Flaky Test? No Flake News! @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Definition Test that exhibits both a passing and failing test result with the same code if run multiple times @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Timing related Example: 1. 2. 3. 4. Enter search term into textbox Click on button “Search” Wait for one second Click on first result @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Test data related Examples: ● Input field for prename doesn’t allow special characters ○ Some randomly generated prenames in tests contain special characters (“François”) ● Tests reliant on predefined test data in the database ○ Dependencies between tests ○ Failing tests cause inconsistencies @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Infrastructure related Unstable infrastructure Tests time out Most difficult to reproduce Examples: ● Build runners under heavy load ● Microservices with networking issues @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Resource contention related Mostly when running tests in parallel Examples: Multiple tests trying to… ● … write to the same file at the same time ○ File lock ○ Concurrent modification errors ● … create a user with the same username at the same time ○ Unique constraint violations @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Why Worry about Flaky Tests? ● Re-running tests wastes a lot of time ● Repetitive investigation efforts to determine if flaky or legitimate failure ● Team starts losing trust in test results ● Bad software quality ● Pilots frequently ignoring an alarm in cockpits of a Boeing 737 resulted in fatal plane crash in 2005 @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Unit Tests vs Integration Tests UT vs IT @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice End-to-End Tests Integration Tests Unit Tests Testing Pyramid @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Unit Tests Integration Tests Flaky Foundations End-to-End Tests @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Complexity Flakiness @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice How to find Flaky Tests Flake Busters @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Running tests in parallel Can reveal many issues: ● Resource contention ● Dependency on test data ● Dependency on test execution order Side benefit: Tests execute faster ● WebdriverIO: parallel by default ● Cypress: only in cloud (paid) ● JUnit and Gradle: through configuration @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Rerun Tests Most test frameworks allow for executing a test multiple times. @BrianDemers | bdemers

Maven Surefire Plugin JCON Europe 2024 | Testing on Thin Ice <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <rerunFailingTestsCount>3//rerunFailingTestsCount> <failOnFlakeCount>0//failOnFlakeCount> </configuration> </plugin> @BrianDemers | bdemers

Rerun Gotcha Increased time to fail

JCON Europe 2024 | Testing on Thin Ice @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Dealing with Flaky Tests @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Prevention In Pull / Merge Requests, run new or changed tests 3, 10, 50, or 1000 times (depending on resources and test type) Alternatively run all tests multiple times nightly @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Tell Your Team! Don’t let them waste time Uncover deeper issues Saves morale @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice New Sprint Checklist Fix a flaky test Pull a new issue @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Flaky Test Days @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Quarantine ● Quarantine test after first failure ● Fix the quarantined tests timely ● Set a limit for tests in quarantine ● Define time limit for quarantined tests @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Quarantine // JUnit @Disabled(“quarantine: reason or link to issue here”) @Test void flakyTest() { } // Jest </ quarantine: reason or link to issue here it.skip(‘should throw an error’, () <> { expect(response).toThrowError(expected_error) }); @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Delete Flaky Tests Does the test provide value? Can it be fixed? Keeping a flaky test provides negative value @BrianDemers | bdemers

SSH into CI Agent $ ssh ci-agent cd /your-project # rerun build ./mvnw clean install @BrianDemers | bdemers JCON Europe 2024 | Testing on Thin Ice

JCON Europe 2024 | Testing on Thin Ice How to fix Flaky Tests Defeating Murphy’s Law @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Test Data Consistency // JUnit 5 // Jest @BeforeEach void setup() { </ Create test data } beforeEach(() <> { </ Create test data }); @AfterEach void teardown() { </ Delete test data } afterEach(() <> { </ Delete test data }); @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Test Data Consistency // Using Testcontainers </ build.gradle testImplementation “org.testcontainers:postgres:1.19.8” testImplementation “org.testcontainers:junit-jupiter:1.19.8” </ JUnit Test @Testcontainers public class IntegrationTest { @Container PostgreSQLContainer<?> postgresContainer = new PostgreSQLContainer<>(“postgres:16.3”); } @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Ports Source Data Tests Data @BrianDemers | bdemers Isolate Tests

JCON Europe 2024 | Testing on Thin Ice Don’t Reuse System Resources ● Setting System Properties System.setProperty(“key”, “value”); ● Use unique files for each test temp directories, output files, etc. ● Use a random free ephemeral port (49152 - 65535) new ServerSocket(0).getLocalPort() @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice In E2E Tests ● Use deep links to navigate directly to the page to be tested ● Start the test with the test user already logged in ● Make it possible to run each test spec independently ● Do not wait in predefined time intervals, wait for a condition to be fulfilled with a timeout ● Use stable test ids in selectors ● Consider mocking external APIs @fmartin_ | FrançoisMartin

JCON Europe 2024 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Convert ITs and E2E into UTs Flakiness @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Te nd -E to dEn io n at gr te In Un it Te st s Te s st s ts Convert ITs and E2E into UTs Stability @BrianDemers | bdemers

JCON Europe 2024 | Testing on Thin Ice Fix the Culture of Flaky Tests @BrianDemers | bdemers ● Brown Bag / Lunch & Learn ● Make flaky tests visible ● Track them ● Use manager terms 💰 @BrianDemers | bdemers

Slides JCON Europe 2024 | Testing on Thin Ice THANKS! QUESTIONS? @BrianDemers @fmartin_ bdemers FrançoisMartin CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik and images generated by OpenAI’s DALL-E model via ChatGPT Questions