Security and Productivity - Pick Two with Reproducible builds

A presentation at Boston Java Users ACM Chapter in January 2024 in Boston, MA, USA by Brian Demers

Slide 1

Slide 1

fl ThunderKiss Photography ickr.com/photos/treycampbell/4512662113 Security and Productivity Pick Two with Reproducible Builds Brian Demers Open Source Developer BrianDemers bdemers

Slide 2

Slide 2

@BrianDemers | bdemers

Slide 3

Slide 3

Who is this guy? @BrianDemers | bdemers

Slide 4

Slide 4

source: Silicon Valley @BrianDemers | bdemers

Slide 5

Slide 5

VS @BrianDemers | bdemers

Slide 6

Slide 6

VS @BrianDemers | bdemers

Slide 7

Slide 7

Developer Productivity Engineering @BrianDemers | bdemers

Slide 8

Slide 8

Topics • • • Reproducible Builds • What is it? • Why should you care? Developer productivity • How are these related? • Build Cache Tips & Tricks @BrianDemers | bdemers

Slide 9

Slide 9

Source Build Build Verify reproducible-builds.org @BrianDemers | bdemers

Slide 10

Slide 10

Source Build Build Verify @BrianDemers | bdemers

Slide 11

Slide 11

03ba204e50d126e4… data @BrianDemers | bdemers hash function

Slide 12

Slide 12

How did we get here? @BrianDemers | bdemers

Slide 13

Slide 13

Old way (META-INF/MANIFEST.MF) Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: jdcasey Build-Jdk: 1.4.2_09 Extension-Name: maven-core Specification-Title: Maven is a project development management and com prehension tool. Based on the concept of a project object model: buil ds, dependency management, documentation creation, site publication, and distribution publication are all controlled from the declarative file. Maven can be extended by plugins to utilise a number of other d evelopment tools for reporting or the build process. Specification-Vendor: Apache Software Foundation Implementation-Vendor: Apache Software Foundation Implementation-Title: maven-core Implementation-Version: 2.0.1 @BrianDemers | bdemers

Slide 14

Slide 14

New Way (META-INF/MANIFEST.MF) Manifest-Version: 1.0 Created-By: Maven JAR Plugin 3.3.0 Build-Jdk-Spec: 17 Specification-Title: Maven Core Specification-Version: 3.9 Specification-Vendor: The Apache Software Foundation Implementation-Title: Maven Core Implementation-Version: 3.9.4 Implementation-Vendor: The Apache Software Foundation @BrianDemers | bdemers

Slide 15

Slide 15

@BrianDemers | bdemers

Slide 16

Slide 16

@BrianDemers | bdemers

Slide 17

Slide 17

Reproducible builds for security • Binaries are not tampered with • Build system not comprimised • Prevent backdoors • Supply Chain attacks @BrianDemers | bdemers

Slide 18

Slide 18

Compromised Toolchain $ cp evil-compiler /usr/bin/compiler $ unzip evil.zip -d /src/project extracting @BrianDemers | bdemers ../../etc/passwd

Slide 19

Slide 19

Who should care? • Open Source Projects • Distributions • Companies • Users

Slide 20

Slide 20

Shouldn’t All Builds be Reproducible? @BrianDemers | bdemers

Slide 21

Slide 21

📆 Dates ⏰ • Current date/time • Time Zone • Locale/Format • Dates in versions

Slide 22

Slide 22

File Dates $ ls -alh target/scim-core-1.0.0-SNAPSHOT.jar -rw-r—r— bdemers staff 56K Jul 27 12:32:43 2023 @BrianDemers | bdemers

Slide 23

Slide 23

Dates in Archives (zip, tar, jar, etc) Archive: target/scim-core-1.0.0-SNAPSHOT.jar Date Time Name ————— ———-04-05-2023 08:23 META-INF/MANIFEST.MF 04-05-2023 08:23 META-INF/DEPENDENCIES 04-05-2023 08:23 META-INF/LICENSE 04-05-2023 08:23 META-INF/NOTICE 04-05-2023 08:23 META-INF/beans.xml 04-05-2023 08:23 META-INF/maven/org.apache.directory.scim/scim-core/pom.xml 04-05-2023 08:23 META-INF/maven/org.apache.directory.scim/scim-core/pom.properties 04-05-2023 08:23 org/apache/directory/scim/core/repository/PatchHandler.class 04-05-2023 08:23 org/apache/directory/scim/core/repository/Repository.class 04-05-2023 08:23 org/apache/directory/scim/core/repository/UpdateRequest.class … @BrianDemers | bdemers

Slide 24

Slide 24

Random bits xkcd.com/221 @BrianDemers | bdemers

Slide 25

Slide 25

OS & Environment • File Encoding • OS • Tool Versions • File Paths • Locales • .DS_Store @BrianDemers | bdemers Save the Environment

Slide 26

Slide 26

xkcd.com/1834 File Input / Output Order • Hash Maps • • Serialized data File order in archives

Slide 27

Slide 27

@BrianDemers | bdemers

Slide 28

Slide 28

How to Verify? $ shasum -a 256 AAA-file.zip 80da7adf80a819db609ac6862931dc6c1cc04bf4c8ba292446021aa805aa3bfa $ shasum -a 256 BBB-file.zip 80da7adf80a819db609ac6862931dc6c1cc04bf4c8ba292446021aa805aa3bfa @BrianDemers | bdemers

Slide 29

Slide 29

Diffoscope @BrianDemers | bdemers

Slide 30

Slide 30

jvm-repo-rebuild/reproducible-central @BrianDemers | bdemers

Slide 31

Slide 31

How to get started? @BrianDemers | bdemers

Slide 32

Slide 32

Maven & Gradle <!— Maven —> <properties> <project.build.outputTimestamp> 2023-01-01T00:00:00Z </project.build.outputTimestamp> </properties> // Gradle tasks.withType(AbstractArchiveTask).configureEach { preserveFileTimestamps = false reproducibleFileOrder = true } @BrianDemers | bdemers

Slide 33

Slide 33

Other Beni ts Quality • Debugging • Smaller deltas in releases • Cacheable fi • B s u n o

Slide 34

Slide 34

Think about a Docker builds @BrianDemers | bdemers

Slide 35

Slide 35

Build Caching ⬢ Introduced to the Java world by Gradle ⬢ Build caches are complementary to dependency in 2017 ⬢ Maven has an open source build cache caches, not mutually exclusive: ○ too ⬢ Used by leading technology dependencies ○ companies like Google and Facebook ⬢ Can support both user local and remote caching for distributed teams @BrianDemers | bdemers A dependency cache caches fully compiled A build cache accelerates building a single source repository ○ A build cache caches build actions (e.g. Gradle tasks or Maven goals)

Slide 36

Slide 36

What is a Build Cache? Inputs ● Gradle Tasks ● Maven Goal Executions Outputs When the inputs have not changed, the output can be reused from a previous run.

Slide 37

Slide 37

Cache Key/Value Calculation The cacheKey for Gradle Tasks/Maven Goals is based on the Inputs: cacheKey(javaCompile) = hash(sourceFiles, jdk version, classpath, compiler args) The cacheEntry contains the output: cacheEntry[cacheKey(javaCompile)] = fileTree(classFiles) For more information, see: https://docs.gradle.org/current/userguide/build_cache.html

Slide 38

Slide 38

When not using the build cache, with Maven any change will require a full build. For Gradle this is the case when doing clean builds and switching between branches.

Slide 39

Slide 39

Changing an public method in the export-api module

Slide 40

Slide 40

Changing an implementation detail of a method in the service module

Slide 41

Slide 41

Remote Build Cache ⬢ Shared among different machines ⬢ Speeds up development for the whole team ⬢ Reuses build results among CI agents/jobs and individual developers

Slide 42

Slide 42

source: https://www.cshl.edu/quiz/brain-interrupted/ @BrianDemers | bdemers

Slide 43

Slide 43

The anatomy of fast feedback cycles PRODUCTIVITY Less idle/ wait time Less context switching More focused developers QUALITY FASTER FEEDBACK CYCLES KEY: New behavior More frequent builds Earlier quality checks Fewer downstream incidents Smaller change sets Few merge conflicts Effect More efficient troubleshooting KEY BENEFIT @BrianDemers | bdemers Faster MTTR

Slide 44

Slide 44

Developer Productivity Engineering (DPE) @BrianDemers | bdemers

Slide 45

Slide 45

@BrianDemers | bdemers

Slide 46

Slide 46

Progression of Productivity @BrianDemers | bdemers

Slide 47

Slide 47

The Future Dwurban, CC BY-SA 4.0 https://commons.wikimedia.org/w/index.php?curid=116834907 @BrianDemers | bdemers

Slide 48

Slide 48

Other Ways to Speed up Builds • Update your build tool • Break project into modules • Predictive Test Selection • Test Distribution

Slide 49

Slide 49

@BrianDemers | bdemers

Slide 50

Slide 50

Questions? Thank you! BrianDemers bdemers Learn more & get free swag