Developer Productivity Engineering: What’s in it for Java Developers?

Developer Productivity Engineering What’s in it for Java Developers?

Who is this guy? @BrianDemers | bdemers

source: Silicon Valley @BrianDemers | bdemers

VS @BrianDemers | bdemers

VS @BrianDemers | bdemers

Developer Productivity Engineering @BrianDemers | bdemers

@BrianDemers | bdemers

Code Code Wait Time for Local Build Debug Build Failure Lunch Code Sprint Wait Time for Local Build Waiting time for CI Build Investigate/Fix Flaky Tests Bottlenecks to Productivity are Everywhere

“Bottlenecks in the toolchain are holding back the rockstar 10x developers” Pete Smoot, Software Architect, Dell Technologies It’s True at Dell, and Everywhere Else

Developers code 52 minutes per day https://www.software.com/reports/code-time-report @BrianDemers | bdemers

DPE is a new software development practice used by leading software development organizations to maximize developer productivity and happiness. Gradle is Pioneering DPE

xkcd.com/303 This Doesn’t Have To Be Our Reality

source: https://www.cshl.edu/quiz/brain-interrupted/ @BrianDemers | bdemers

What Problems Does DPE Solve? This takes too long! This takes too long to fix This should have been observable

Blank use will Builds background and Tests Take Tooat Long! Blank background use at will

The anatomy and importance of fast feedback cycles PRODUCTIVITY Less idle/ wait time Less context switching More focused developers QUALITY FASTER FEEDBACK CYCLES More frequent builds KEY: Earlier quality checks Fewer downstream incidents Smaller change sets Few merge conflicts New behavior Effect KEY BENEFIT More efficient troubleshooting Faster MTTR

Faster Builds Improve Creative Flow No. of Devs Build Time No. of local builds Team 1 Team 2 11 6 4 mins 1 mins 850 1010

Multiple Acceleration Technologies are Best

Build caching delivers fast build and test feedback cycles

Build Caching ⬢ Introduced to the Java world by Gradle in 2017 ⬢ Maven has an open source build ⬢ Build caches are complementary to dependency caches, not mutually exclusive: ○ dependencies cache too ⬢ Used by leading technology ○ Can support both user local and remote caching for distributed teams A build cache accelerates building a single source repository companies like Google and Facebook ⬢ A dependency cache caches fully compiled ○ A build cache caches build actions (e.g. Gradle tasks or Maven goals)

What is a Build Cache? Inputs ● Gradle Tasks ● Maven Goal Executions Outputs When the inputs have not changed, the output can be reused from a previous run.

Cache Key/Value Calculation The cacheKey for Gradle Tasks/Maven Goals is based on the Inputs: cacheKey(javaCompile) = hash(sourceFiles, jdk version, classpath, compiler args) The cacheEntry contains the output: cacheEntry[cacheKey(javaCompile)] = fileTree(classFiles) For more information, see: https://docs.gradle.org/current/userguide/build_cache.html

Remote Build Cache Savings at Dell 41% Savings from Cache

https://reproducible-builds.org/

fi Predictive Test Selection leads to greater ef ciencies

https://research.facebook.com/publications/predictive-test-selection/

Conventional Test Selection Approach

Predictive Test Selection Approach 4% 3% 2% 2% 98% 96%

Predictive Test Selection Savings at Dell 7 Day Period 41 Days Testing Savings Predicted

Test distribution can make tests even faster

How it works Autoscaler

Existing solutions - CI fanout Test execution is distributed by manually partitioning the test set and then running partitions in parallel on several CI nodes. pipeline { stage(‘compile’) { … } parallelStage(‘test’) { step { sh ‘./gradlew :testGroup1’ } step { sh ‘./gradlew :testGroup2’ } step { sh ‘./gradlew :testGroup3’ } } } https://builds.gradle.org/project/Gradle for an example of this strategy See

Assessment of existing solutions ⬢ Build Caching is great in many cases but doesn’t help when test inputs have changed. ⬢ Single machine parallelism is limited by that machine’s resources. ⬢ CI fanout does not help during local development, is inefficient (in particular on ephemeral CI agents or without build cache), requires manual setup and test partitioning, and result collection/aggregation

Build Scans speeds up troubleshooting

Improved Troubleshooting

Build Scan - scans.gradle.com

Without focus, problems can sneak back in… ⬢ ⬢ Infrastructure changes ○ Binary management ○ Caching ○ CI agents New annotation processors or versions of annotation processors ⬢ Build logic configurations settings ○ Build tool version and plugins ○ Compiler and/or Memory settings ⬢ Code refactoring ⬢ New office locations ⬢ Without observability, it is impossible to have a great and fast developer experience.

“ You canBlank observe a lot by watching.” background usejust at will

Yogi Berra, Catcher and Philosopher

Performance Insights Are you tracking local build and test times?

DPE Organizations Track Build and Test Times

DPE Organizations Track Failure Rates

Flaky Tests Are Everywhere 7 day period Test Flaky 299 times

Dealing with Flaky Tests The test is flaky. What do you do now? a. Try it again b. Re-run it c. Re-run it again d. Ignore it and approve PR e. All of the above

DPE Organizations Analyze Flaky Tests

All Of This Will Improve CI

Tips

@BrianDemers | bdemers

Block out time in your calendar! @BrianDemers | bdemers

DPE Blank will become standard practice background use at will Because the world should foster developer joy

Questions? Thank you! BrianDemers bdemers Learn more & get free swag

Developer Productivity Engineering What’s in it for Java Developers?