Breeding 10x Developers with Developer Productivity Engineering

A presentation at Tampa JUG in September 2023 in Tampa, FL, USA by Brian Demers

Slide 1

Breeding 10x Developers With Developer Productivity Engineering

Slide 2

Who is this guy? @BrianDemers | bdemers

Slide 3

source: Silicon Valley @BrianDemers | bdemers

Slide 4

VS @BrianDemers | bdemers

Slide 5

VS @BrianDemers | bdemers

Slide 6

Developer Productivity Engineering @BrianDemers | bdemers

Slide 7

Slide 8

Myth Origin (probably) The Coding War Games

Slide 9

Slide 10

The “best” programmers outperformed the worst by roughly a 10:1 ratio There were some interesting “non-factors”: Language Years of Experience Number of Defects Salary

Slide 11

What Mattered? ⬢ Paired programmers from the same organization performed at roughly the same level ⬢ The average difference was only 21% between paired participants ⬢ They didn’t work together on the task, but they came from the same organization ⬢ The best organizations performed 11.1x better than the worst “While this productivity differential among programmers is understandable, there is also a 10 to 1 difference in productivity among software organizations.” -Harlan D. Mills, Software Productivity

Slide 12

The best performers are clustering in some organizations while the worst performers are clustering in others. Some companies are doing a lot worse than others. Something about their environment and corporate culture is failing to attract and keep good people or is making it impossible for even good people to work effectively.

Slide 13

Average performance of those in the top quarter was 2.6 times better than that of those in the bottom quarter. Though the phrase had not yet been coined, increased productivity came down to developer experience.

Slide 14

10x Organizations are Manufactured, Not Born

Slide 15

@BrianDemers | bdemers

Slide 16

… But Most Organizations Aren’t Aligned In a study dated April 27, 2022, between Microsoft and the University of Victoria in British Columbia, Developers and Managers were surveyed on their interpretation of the SPACE framework

Slide 17

When surveyed with the following questions, Developers and Managers answered much differently Developers Managers When thinking about your work, how do you define productivity? When thinking about your team, how do you define productivity? https://arxiv.org/pdf/2111.04302.pdf

Slide 18

DevOps, 12-Factor, Agile, etc, have still not captured all bottlenecks, friction, and obstacles to throughput Many are hiding in plain sight, in the developer experience itself A 10x organization should think about reducing build and test feedback times, and improving the consistency and reliability of builds

Slide 19

The only initiatives that will positively impact performance are ones which increase throughput while simultaneously decreasing cost.

Slide 20

It’s Time for Developer Productivity Engineering

Slide 21

Slide 22

Slide 23

Code Code Wait Time for Local Build Debug Build Failure Lunch Code Wait Time for Local Build Sprint Waiting time for CI Build Investigate/Fix Flaky Tests 23

Slide 24

xkcd.com/303

Slide 25

Testing xkcd.com/303

Slide 26

CHAT GPT IS DOWN CHAT GPT IS STILL DOWN xkcd.com/303

Slide 27

What Problems Does DPE Solve? Creative Flow “ It isn’t about being the best, because once you tap into the creative flow, the best doesn’t exist. It’s about experiencing the magic. Its about being in that place where your heart is free. It’s about grabbing hold of the intangible, and making something beautiful. This takes too long! This takes too long to fix This should have been observable

Laura Jaworski, Artist and Author

Slide 28

The anatomy and importance of fast feedback cycles PRODUCTIVITY Less idle/ wait time Less context switching More focused developers QUALITY FASTER FEEDBACK CYCLES More frequent builds KEY: Earlier quality checks Fewer downstream incidents Smaller change sets Few merge conflicts New behavior Effect KEY BENEFIT More efficient troubleshooting Faster MTTR

Slide 29

Build caching delivers fast build and test feedback cycles

Slide 30

Build Caching ⬢ Introduced to the Java world by Gradle in 2017 ⬢ Maven has an open source build ⬢ Build caches are complementary to dependency caches, not mutually exclusive: ○ dependencies cache too ⬢ Used by leading technology ○ Can support both user local and remote caching for distributed teams A build cache accelerates building a single source repository companies like Google and Facebook ⬢ A dependency cache caches fully compiled ○ A build cache caches build actions (e.g. Gradle tasks or Maven goals)

Slide 31

What is a Build Cache? Inputs ● Gradle Tasks ● Maven Goal Executions Outputs When the inputs have not changed, the output can be reused from a previous run.

Slide 32

Cache Key/Value Calculation The cacheKey for Gradle Tasks/Maven Goals is based on the Inputs: cacheKey(javaCompile) = hash(sourceFiles, jdk version, classpath, compiler args) The cacheEntry contains the output: cacheEntry[cacheKey(javaCompile)] = fileTree(classFiles) For more information, see: https://docs.gradle.org/current/userguide/build_cache.html

Slide 33

When not using the build cache, with Maven any change will require a full build. For Gradle this is the case when doing clean builds and switching between branches.

Slide 34

Changing an public method in the export-api module

Slide 35

Changing an implementation detail of a method in the service module

Slide 36

Remote Build Cache ⬢ Shared among different machines ⬢ Speeds up development for the whole team ⬢ Reuses build results among CI agents/jobs and individual developers

Slide 37

reproducible-builds.org @BrianDemers | bdemers

Slide 38

Test distribution can make tests even faster

Slide 39

How it works Autoscaler

Slide 40

Existing solutions - CI fanout Test execution is distributed by manually partitioning the test set and then running partitions in parallel on several CI nodes. pipeline { stage(‘compile’) { … } parallelStage(‘test’) { step { sh ‘./gradlew :testGroup1’ } step { sh ‘./gradlew :testGroup2’ } step { sh ‘./gradlew :testGroup3’ } } } See https://builds.gradle.org/project/Gradle for an example of this strategy

Slide 41

Assessment of existing solutions ⬢ Build Caching is great in many cases but doesn’t help when test inputs have changed. ⬢ Single machine parallelism is limited by that machine’s resources. ⬢ CI fanout does not help during local development, is inefficient (in particular on ephemeral CI agents or without build cache), requires manual setup and test partitioning, and result collection/aggregation

Slide 42

Netflix reduced a 62-minute test cycle time down to just under 5 minutes!

Slide 43

fi Predictive Test Selection leads to greater ef ciencies

Slide 44

https://research.facebook.com/publications/predictive-test-selection/

Slide 45

Conventional Test Selection Approach

Slide 46

Predictive Test Selection Approach 4% 3% 2% 2% 98% 96%

Slide 47

Build Scans speeds up troubleshooting

Slide 48

Slide 49

Without focus, problems can sneak back in… ⬢ ⬢ Infrastructure changes ○ Binary management ○ Caching ○ CI agents New annotation processors or versions of annotation processors ⬢ Build logic configurations settings ○ Build tool version and plugins ○ Compiler and/or Memory settings ⬢ Code refactoring ⬢ New office locations ⬢ Without observability, it is impossible to have a great and fast developer experience.

Slide 50

“You can observe a lot by just watching” - Yogi Berra, Catcher and Philosopher

Slide 51

Performance Insights Are you tracking local build and test times?

Slide 52

Is the cycle fast enough? Is the cycle as fast as it can possibly be?

Slide 53

Slide 54

DPE Organizations Eliminate Avoidable Failures

Slide 55

DPE Organizations Track Failure Rates

Slide 56

Dealing with Flaky Tests The test is flaky. What do you do now? a. Try it again b. Re-run it c. Re-run it again d. Ignore it and approve PR e. All of the above

Slide 57

DPE Organizations Analyze Flaky Tests

Slide 58

All Of This Will Improve CI

Slide 59

DPE Will Become Standard Practice Because the World Should Foster Developer Joy

Slide 60

Questions? BrianDemers bdemers Learn more & get free swag

Slide 61

THANKS!!