Breeding 10x Developers with Developer Productivity Engineering

G U J t i o r t e D Breeding 10x Developers With Developer Productivity Engineering

Who is this guy? @BrianDemers | bdemers

source: Silicon Valley @BrianDemers | bdemers

VS @BrianDemers | bdemers

VS source:epicurious.com/recipes/food/views/detroit-style-pizza @BrianDemers | bdemers source: allrecipes.com/recipe/240376/homemade-pepperoni-pizza/

VS @BrianDemers | bdemers

Developer Productivity Engineering @BrianDemers | bdemers

Myth Origin (probably) The Coding War Games

The “best” programmers outperformed the worst by roughly a 10:1 ratio There were some interesting “non-factors”: Language Years of Experience Number of Defects Salary

What Mattered? ⬢ Paired programmers from the same organization performed at roughly the same level ⬢ The average difference was only 21% between paired participants ⬢ They didn’t work together on the task, but they came from the same organization ⬢ The best organizations performed 11.1x better than the worst “While this productivity differential among programmers is understandable, there is also a 10 to 1 difference in productivity among software organizations.” -Harlan D. Mills, Software Productivity

The best performers are clustering in some organizations while the worst performers are clustering in others. Some companies are doing a lot worse than others. Something about their environment and corporate culture is failing to attract and keep good people or is making it impossible for even good people to work effectively.

Average performance of those in the top quarter was 2.6 times better than that of those in the bottom quarter. Though the phrase had not yet been coined, increased productivity came down to developer experience.

10x Organizations are Manufactured, Not Born

This Matters Even More Now

… But Most Organizations Aren’t Aligned In a study dated April 27, 2022, between Microsoft and the University of Victoria in British Columbia, Developers and Managers were surveyed on their interpretation of the SPACE framework

When surveyed with the following questions, Developers and Managers answered much differently Developers Managers When thinking about your work, how do you define productivity? When thinking about your team, how do you define productivity? https://arxiv.org/pdf/2111.04302.pdf

DevOps, 12-Factor, Agile, etc, have still not captured all bottlenecks, friction, and obstacles to throughput Many are hiding in plain sight, in the developer experience itself A 10x organization should think about reducing build and test feedback times, and improving the consistency and reliability of builds

The only initiatives that will positively impact performance are ones which increase throughput while simultaneously decreasing cost.

It’s Time for Developer Productivity Engineering

Code Code Wait Time for Local Build Debug Build Failure Lunch Code Wait Time for Local Build Sprint Waiting time for CI Build Investigate/Fix Flaky Tests 24

xkcd.com/303

Testing xkcd.com/303

CHAT GPT IS DOWN CHAT GPT IS STILL DOWN xkcd.com/303

What Problems Does DPE Solve? Creative Flow “ It isn’t about being the best, because once you tap into the creative flow, the best doesn’t exist. It’s about experiencing the magic. Its about being in that place where your heart is free. It’s about grabbing hold of the intangible, and making something beautiful. This takes too long! This takes too long to fix This should have been observable

• Laura Jaworski, Artist and Author

The anatomy and importance of fast feedback cycles PRODUCTIVITY Less idle/ wait time Less context switching More focused developers QUALITY FASTER FEEDBACK CYCLES More frequent builds KEY: Earlier quality checks Fewer downstream incidents Smaller change sets Few merge conflicts New behavior Effect KEY BENEFIT More efficient troubleshooting Faster MTTR

Build caching delivers fast build and test feedback cycles

Build Caching ⬢ Introduced to the Java world by Gradle in 2017 ⬢ Maven has an open source build ⬢ Build caches are complementary to dependency caches, not mutually exclusive: ○ dependencies cache too ⬢ Used by leading technology ○ Can support both user local and remote caching for distributed teams A build cache accelerates building a single source repository companies like Google and Facebook ⬢ A dependency cache caches fully compiled ○ A build cache caches build actions (e.g. Gradle tasks or Maven goals)

What is a Build Cache? Inputs ● Gradle Tasks ● Maven Goal Executions Outputs When the inputs have not changed, the output can be reused from a previous run.

Cache Key/Value Calculation The cacheKey for Gradle Tasks/Maven Goals is based on the Inputs: cacheKey(javaCompile) = hash(sourceFiles, jdk version, classpath, compiler args) The cacheEntry contains the output: cacheEntry[cacheKey(javaCompile)] = fileTree(classFiles) For more information, see: https://docs.gradle.org/current/userguide/build_cache.html

When not using the build cache, with Maven any change will require a full build. For Gradle this is the case when doing clean builds and switching between branches.

Changing an public method in the export-api module

Changing an implementation detail of a method in the service module

Remote Build Cache ⬢ Shared among different machines ⬢ Speeds up development for the whole team ⬢ Reuses build results among CI agents/jobs and individual developers

@BrianDemers | bdemers

Test distribution can make tests even faster

How it works Autoscaler

Existing solutions - CI fanout Test execution is distributed by manually partitioning the test set and then running partitions in parallel on several CI nodes. pipeline { stage(‘compile’) { … } parallelStage(‘test’) { step { sh ‘./gradlew :testGroup1’ } step { sh ‘./gradlew :testGroup2’ } step { sh ‘./gradlew :testGroup3’ } } } See https://builds.gradle.org/project/Gradle for an example of this strategy

Assessment of existing solutions ⬢ Build Caching is great in many cases but doesn’t help when test inputs have changed. ⬢ Single machine parallelism is limited by that machine’s resources. ⬢ CI fanout does not help during local development, is inefficient (in particular on ephemeral CI agents or without build cache), requires manual setup and test partitioning, and result collection/aggregation

Netflix reduced a 62-minute test cycle time down to just under 5 minutes!

fi Predictive Test Selection leads to greater ef ciencies

https://research.facebook.com/publications/predictive-test-selection/

Conventional Test Selection Approach

Predictive Test Selection Approach 4% 3% 2% 2% 98% 96%

Build Scans speeds up troubleshooting

Without focus, problems can sneak back in… ⬢ ⬢ Infrastructure changes ○ Binary management ○ Caching ○ CI agents New annotation processors or versions of annotation processors ⬢ Build logic configurations settings ○ Build tool version and plugins ○ Compiler and/or Memory settings ⬢ Code refactoring ⬢ New office locations ⬢ Without observability, it is impossible to have a great and fast developer experience.

“You can observe a lot by just watching” - Yogi Berra, Catcher and Philosopher

Performance Insights Are you tracking local build and test times?

Is the cycle fast enough? Is the cycle as fast as it can possibly be?

DPE Organizations Eliminate Avoidable Failures

DPE Organizations Track Failure Rates

Dealing with Flaky Tests The test is flaky. What do you do now? a. Try it again b. Re-run it c. Re-run it again d. Ignore it and approve PR e. All of the above

DPE Organizations Analyze Flaky Tests

All Of This Will Improve CI

DPE Will Become Standard Practice Because the World Should Foster Developer Joy

Questions? BrianDemers bdemers Learn more & get free wag

THANKS!!