A presentation at Tampa JUG in in Tampa, FL, USA by Brian Demers
Breeding 10x Developers With Developer Productivity Engineering
Who is this guy? @BrianDemers | bdemers
source: Silicon Valley @BrianDemers | bdemers
VS @BrianDemers | bdemers
VS @BrianDemers | bdemers
Developer Productivity Engineering @BrianDemers | bdemers
Myth Origin (probably) The Coding War Games
The “best” programmers outperformed the worst by roughly a 10:1 ratio There were some interesting “non-factors”: Language Years of Experience Number of Defects Salary
What Mattered? ⬢ Paired programmers from the same organization performed at roughly the same level ⬢ The average difference was only 21% between paired participants ⬢ They didn’t work together on the task, but they came from the same organization ⬢ The best organizations performed 11.1x better than the worst “While this productivity differential among programmers is understandable, there is also a 10 to 1 difference in productivity among software organizations.” -Harlan D. Mills, Software Productivity
The best performers are clustering in some organizations while the worst performers are clustering in others. Some companies are doing a lot worse than others. Something about their environment and corporate culture is failing to attract and keep good people or is making it impossible for even good people to work effectively.
Average performance of those in the top quarter was 2.6 times better than that of those in the bottom quarter. Though the phrase had not yet been coined, increased productivity came down to developer experience.
10x Organizations are Manufactured, Not Born
@BrianDemers | bdemers
… But Most Organizations Aren’t Aligned In a study dated April 27, 2022, between Microsoft and the University of Victoria in British Columbia, Developers and Managers were surveyed on their interpretation of the SPACE framework
When surveyed with the following questions, Developers and Managers answered much differently Developers Managers When thinking about your work, how do you define productivity? When thinking about your team, how do you define productivity? https://arxiv.org/pdf/2111.04302.pdf
DevOps, 12-Factor, Agile, etc, have still not captured all bottlenecks, friction, and obstacles to throughput Many are hiding in plain sight, in the developer experience itself A 10x organization should think about reducing build and test feedback times, and improving the consistency and reliability of builds
The only initiatives that will positively impact performance are ones which increase throughput while simultaneously decreasing cost.
It’s Time for Developer Productivity Engineering
Code Code Wait Time for Local Build Debug Build Failure Lunch Code Wait Time for Local Build Sprint Waiting time for CI Build Investigate/Fix Flaky Tests 23
xkcd.com/303
Testing xkcd.com/303
CHAT GPT IS DOWN CHAT GPT IS STILL DOWN xkcd.com/303
What Problems Does DPE Solve? Creative Flow “ It isn’t about being the best, because once you tap into the creative flow, the best doesn’t exist. It’s about experiencing the magic. Its about being in that place where your heart is free. It’s about grabbing hold of the intangible, and making something beautiful. This takes too long! This takes too long to fix This should have been observable
The anatomy and importance of fast feedback cycles PRODUCTIVITY Less idle/ wait time Less context switching More focused developers QUALITY FASTER FEEDBACK CYCLES More frequent builds KEY: Earlier quality checks Fewer downstream incidents Smaller change sets Few merge conflicts New behavior Effect KEY BENEFIT More efficient troubleshooting Faster MTTR
Build caching delivers fast build and test feedback cycles
Build Caching ⬢ Introduced to the Java world by Gradle in 2017 ⬢ Maven has an open source build ⬢ Build caches are complementary to dependency caches, not mutually exclusive: ○ dependencies cache too ⬢ Used by leading technology ○ Can support both user local and remote caching for distributed teams A build cache accelerates building a single source repository companies like Google and Facebook ⬢ A dependency cache caches fully compiled ○ A build cache caches build actions (e.g. Gradle tasks or Maven goals)
What is a Build Cache? Inputs ● Gradle Tasks ● Maven Goal Executions Outputs When the inputs have not changed, the output can be reused from a previous run.
Cache Key/Value Calculation The cacheKey for Gradle Tasks/Maven Goals is based on the Inputs: cacheKey(javaCompile) = hash(sourceFiles, jdk version, classpath, compiler args) The cacheEntry contains the output: cacheEntry[cacheKey(javaCompile)] = fileTree(classFiles) For more information, see: https://docs.gradle.org/current/userguide/build_cache.html
When not using the build cache, with Maven any change will require a full build. For Gradle this is the case when doing clean builds and switching between branches.
Changing an public method in the export-api module
Changing an implementation detail of a method in the service module
Remote Build Cache ⬢ Shared among different machines ⬢ Speeds up development for the whole team ⬢ Reuses build results among CI agents/jobs and individual developers
reproducible-builds.org @BrianDemers | bdemers
Test distribution can make tests even faster
How it works Autoscaler
Existing solutions - CI fanout Test execution is distributed by manually partitioning the test set and then running partitions in parallel on several CI nodes. pipeline { stage(‘compile’) { … } parallelStage(‘test’) { step { sh ‘./gradlew :testGroup1’ } step { sh ‘./gradlew :testGroup2’ } step { sh ‘./gradlew :testGroup3’ } } } See https://builds.gradle.org/project/Gradle for an example of this strategy
Assessment of existing solutions ⬢ Build Caching is great in many cases but doesn’t help when test inputs have changed. ⬢ Single machine parallelism is limited by that machine’s resources. ⬢ CI fanout does not help during local development, is inefficient (in particular on ephemeral CI agents or without build cache), requires manual setup and test partitioning, and result collection/aggregation
Netflix reduced a 62-minute test cycle time down to just under 5 minutes!
fi Predictive Test Selection leads to greater ef ciencies
https://research.facebook.com/publications/predictive-test-selection/
Conventional Test Selection Approach
Predictive Test Selection Approach 4% 3% 2% 2% 98% 96%
Build Scans speeds up troubleshooting
Without focus, problems can sneak back in… ⬢ ⬢ Infrastructure changes ○ Binary management ○ Caching ○ CI agents New annotation processors or versions of annotation processors ⬢ Build logic configurations settings ○ Build tool version and plugins ○ Compiler and/or Memory settings ⬢ Code refactoring ⬢ New office locations ⬢ Without observability, it is impossible to have a great and fast developer experience.
“You can observe a lot by just watching” - Yogi Berra, Catcher and Philosopher
Performance Insights Are you tracking local build and test times?
Is the cycle fast enough? Is the cycle as fast as it can possibly be?
DPE Organizations Eliminate Avoidable Failures
DPE Organizations Track Failure Rates
Dealing with Flaky Tests The test is flaky. What do you do now? a. Try it again b. Re-run it c. Re-run it again d. Ignore it and approve PR e. All of the above
DPE Organizations Analyze Flaky Tests
All Of This Will Improve CI
DPE Will Become Standard Practice Because the World Should Foster Developer Joy
Questions? BrianDemers bdemers Learn more & get free swag
THANKS!!
Sasquatch. Yeti. The Loch Ness Monster. The 10x Developer.
You may think of these as mythical creatures that can’t possibly exist, but the 10x Organization is very real. In this session, Gradle’s Brian Demers will explain how a dedicated Developer Productivity Engineering (DPE) organization can breed 10x Developers. By reducing the toil, friction, and frustration of slow builds, flaky tests, and other avoidable failures, a DPE team enables a level of developer productivity that you may have thought impossible. Brian will help you explore DPE technologies, including build and test acceleration, failure analytics, and easily analyzed build records to show how to create an environment in which 10x Developers not only exist, but thrive.
