Testing Your Testing Strategy

Testing your testing strategy Trent Willis revo.js

Testing your testing strategy

cd existing-product Trent Willis revo.js

Testing your testing strategy

cd existing-product > init new-product Trent Willis revo.js

Testing your testing strategy

cd existing-product > init new-product > npm test Trent Willis revo.js

Testing your testing strategy

cd existing-product > init new-product > npm test Running… Trent Willis revo.js

Testing your testing strategy “ Trent Willis Why do I need to write tests when I might have to rewrite them in a few weeks anyway? revo.js

Testing your testing strategy “ Trent Willis This approach worked for the other project. We can’t ship features without tests! revo.js

Testing your testing strategy Trent Willis revo.js

Rethinking Testing your testing strategy Trent Willis revo.js

Testing your testing strategy Trent Willis Staff Software Engineer, Netflix Former QUnit project lead @trentmwillis Trent Willis revo.js

Testing your testing strategy Expectation vs. Reality of Testing Trent Willis revo.js

Testing your testing strategy Many applications remain under-tested* even though testing is widely regarded as a best practice *according to their authors Trent Willis revo.js

Testing your testing strategy Why testing feels inefficient Trent Willis revo.js

Testing your testing strategy Why testing feels inefficient: 1. Unnecessarily high expectations Trent Willis revo.js

Testing your testing strategy Popular testing wisdom advocates for multi-layered approaches for all projects Trent Willis revo.js

Testing your testing strategy There is an (over)abundance of testing options Trent Willis revo.js

Testing your testing strategy Why testing feels inefficient: 1. Unnecessarily high expectations 2. Underdeveloped skills Trent Willis revo.js

Testing your testing strategy Learning to write good (i.e., efficient and helpful) tests is a skill that must be developed Trent Willis revo.js

Testing your testing strategy Why testing feels inefficient: 1. Unnecessarily high expectations 2. Underdeveloped skills 3. Unhelpful tests Trent Willis revo.js

Testing your testing strategy const submitButton = screen.getByText(‘Submit’); expect(submitButton) .not.toHaveAttribute(‘disabled’); // Oops, submitButton is <span> not <button>! // So the test never fails! Trent Willis revo.js

Testing your testing strategy Why testing feels inefficient: 1. Unnecessarily high expectations 2. Underdeveloped skills 3. Unhelpful tests Trent Willis revo.js

Testing your testing strategy The value of testing is in quality not quantity Trent Willis revo.js

Testing your testing strategy How do you create a valuable testing strategy? Trent Willis revo.js

Testing your testing strategy Start by defining your goals Trent Willis revo.js

Testing your testing strategy Requirements met Trent Willis revo.js

Testing your testing strategy Time Trent Willis revo.js

Testing your testing strategy The best way to not break an application is to not change it, but we change software to make it better Trent Willis revo.js

Testing your testing strategy Iteration Speed vs. Confidence Trent Willis revo.js

Testing your testing strategy Confidence “No deploy Fridays” “Move fast and break things” Iteration Speed Trent Willis revo.js

Confidence Testing your testing strategy Time Spent On Testing Trent Willis revo.js

Testing your testing strategy The goal of a testing strategy should be to optimize the amount of confidence you get for the time you invest in testing Trent Willis revo.js

Testing your testing strategy How much confidence do you need? Trent Willis revo.js

Testing your testing strategy How much confidence do you need? 1. What stage of development are you in? 2. What are your users’ expectations? 3. What are the market expectations? Trent Willis revo.js

Testing your testing strategy How much confidence do you need? 1. What stage of development are you in? 2. What are your users’ expectations? 3. What are the market expectations? Trent Willis revo.js

Testing your testing strategy As a product “matures”, you need more confidence Trent Willis revo.js

Testing your testing strategy Your confidence level should mirror your confidence that requirements are stable Trent Willis revo.js

Testing your testing strategy How much confidence do you need? 1. What stage of development are you in? 2. What are your users’ expectations? 3. What are the market expectations? Trent Willis revo.js

Testing your testing strategy The cost to access and criticality of a product are directly related to how much confidence you need Trent Willis revo.js

Testing your testing strategy You can codify user expectations with service-level agreements (SLAs) Trent Willis revo.js

Testing your testing strategy How much confidence do you need? 1. What stage of development are you in? 2. What are your users’ expectations? 3. What are the market expectations? Trent Willis revo.js

Testing your testing strategy Good understanding of your product market expectations avoids over- or under-investing in your testing strategy Trent Willis revo.js

Testing your testing strategy Example Early development stage + Low user expectations + New market space = Iteration Speed > Confidence Trent Willis revo.js

Testing your testing strategy Evaluate your options for building confidence Trent Willis revo.js

Testing your testing strategy Non-Testing Options Trent Willis revo.js

Testing your testing strategy Non-Testing Options Static Analysis Trent Willis Linting Type checking revo.js

Testing your testing strategy Non-Testing Options Static Analysis Ideological Changes Trent Willis Reduce size + complexity Encapsulation SOLID (Single Responsibility) Presentational + Container revo.js

Testing your testing strategy Non-Testing Options Static Analysis Ideological Changes Processes Trent Willis Code reviews Easy + fast deployments Canaries / AB Tests Observability + monitoring revo.js

Testing your testing strategy Non-Testing Options Static Analysis Ideological Changes Processes Trent Willis revo.js

Testing your testing strategy Non-Testing Options can build confidence without significantly impacting iteration speed, especially with changing requirements Trent Willis revo.js

Testing your testing strategy “Actual” Tests Unit, end-to-end, integration, smoke, acceptance, regression, functional, behavioral, etc. Trent Willis revo.js

Testing your testing strategy Tests that validate a user flow end-to-end Tests that validate an isolated unit of functionality Trent Willis revo.js

Testing your testing strategy end-to-end tests Trent Willis unit tests revo.js

Testing your testing strategy end-to-end tests Mimic user flow Complicated, slow, flakey Confidence a user flow works Trent Willis unit tests revo.js

Testing your testing strategy end-to-end tests Mimic user flow Complicated, slow, flakey Confidence a user flow works Trent Willis unit tests Isolated functionality Simple, fast, stable No guarantee user flow works revo.js

Testing your testing strategy end-to-end tests Mimic user flow Complicated, slow, flakey Confidence a user flow works Trent Willis unit tests Isolated functionality Simple, fast, stable No guarantee user flow works revo.js

Testing your testing strategy Ideally, end-to-end tests are more valuable Really, unit tests are often a better value Trent Willis revo.js

Testing your testing strategy Doing one or the other well is better than doing both poorly Trent Willis revo.js

Testing your testing strategy Valuable tests are those that catch issues Trent Willis revo.js

Testing your testing strategy Confidence 100% Unit Tests End-to-End Tests Non-Testing Time Investment Trent Willis revo.js

Testing your testing strategy Put it all together Trent Willis revo.js

Testing your testing strategy Iteration Speed > Confidence What can you do within the time constraints? Trent Willis revo.js

Testing your testing strategy Ex: Barely enough time to deliver product as-is Static analysis (linting + type checking) Processes (fast deployments) Add unit/end-to-end tests later Trent Willis revo.js

Testing your testing strategy Ex: In between Alpha and Beta phases Ideological changes (refactor monolithic modules) Add unit tests for complex data logic Trent Willis revo.js

Testing your testing strategy Confidence > Iteration Speed What options give us the confidence needed? Trent Willis revo.js

Testing your testing strategy Ex: Read-only app End-to-end (visual regression) tests Static analysis (linting + type checking) Trent Willis revo.js

Testing your testing strategy Ex: Highly configurable dashboard app Unit tests (covering every option) End-to-end tests (for basic combinations of options) Ideological changes (small modules) Processes (fast deployments, code reviews) Static analysis (linting + type checking) Trent Willis revo.js

Testing your testing strategy Balance your time investment with the level of confidence you need at a given period in time Trent Willis revo.js

Testing your testing strategy Strategy should be revisited periodically Trent Willis revo.js

Testing your testing strategy Your mix of choices can/should change over time Trent Willis revo.js

Testing your testing strategy TL;DL (Too Long; Didn’t Listen) Trent Willis revo.js

Testing your testing strategy TL;DL (Too Long; Didn’t Listen) Define your goals Evaluate your options Make choices based on your goals Trent Willis revo.js

Testing your testing strategy A custom testing strategy benefits your users, company, and team Trent Willis revo.js

Testing your testing strategy You don’t need to reinvent the wheel, but you also don’t need to use the same wheels as everyone else Trent Willis revo.js