Refactoring to modules: Why and how – all you need to know in (less than) an hour

A presentation at Triangle Golang Meetup August 2019 in August 2019 in Raleigh, NC, USA by Baruch Sadogursky

Slide 1

Slide 1

Refactoring to modules: All you need to know in less than an hour @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 2

Slide 2

Let’s go back in time

Slide 3

Slide 3

Pre 1.0 (2012) 1.2 (2013) Poll time! 1.5 (2015) 1.8 (2017) 1.11 (2018) @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 4

Slide 4

Slide 5

Slide 5

🎩 @jbaruch #dockercon jfrog.com/shownotes @ErinMeyerINSEAD’s “Culture Map”

Slide 6

Slide 6

shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 7

Slide 7

Why we have a problem? @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 8

Slide 8

Why we have a problem? @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 9

Slide 9

Dependencies are sources Remote import is a VCS path Simple solution! Dump everything together into one source tree (GOPATH) Compile Profit @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 10

Slide 10

@jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 11

Slide 11

Know which dependencies do I use? Know which dependencies did you use? Know which dependencies should I use? But… how do i… Know is it our code that I am editing right now? WTF is going on?! @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 12

Slide 12

Yeah… “ To date, we’ve resorted to an email semaphore whenever someone fixes a bug a package, imploring everyone else to run go get -u. You can probably imagine how successful this is, and how much time is being spent chasing bugs that were already fixed. Dave Cheney @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 13

Slide 13

Duplicate your dependencies “ Check your dependencies to your own VCS. Brad Firzpatrick @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 14

Slide 14

Build your own dependency manager “ It’s not the role of the tooling provided by the language to dictate how you manage your code in the production sense. Andrew Gerrand @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 15

Slide 15

We expect you to already have a homegrown dependency manager “ If you need to build any tooling around what Go uses (Git, Mercurial, Bazaar), you already understand those tools, so it should be straightforward to integrate with whatever system you have. Andrew Gerrand @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 16

Slide 16

Don’t trust what we’ve built “ go-get is nice for playing around, but if you do something serious, like deploying to production, your deploy script now involves fetching some random dude’s stuff on GitHub. Brad Firzpatrick @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 17

Slide 17

Slide 18

Slide 18

godeps.json dependencies.tsv govendor, govend, goven, gv quiz time! trash, garbage, rubbish Weapons manufacturer @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 19

Slide 19

GOPATH + VENDORING = 💖 @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 20

Slide 20

GOPATH, The proud son of the monorepo

Slide 21

Slide 21

It only allows a single version of any given package to exist at once (per GOPATH) Two huge problems with gopath @jbaruch We cannot programmatically differentiate between code the user is working on and code they merely depend on #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 22

Slide 22

Vendoring – the worst kind of forking “ Copy all of the files at some version from one version control repository and paste them into a different version control repository @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 23

Slide 23

What’s wrong with it (well, what’s not) @jbaruch #trianglegolang History, branch, and tag information is lost Pulling updates is impossible It invites modification, divergence, and bad fork It wastes space Good luck finding which version of the code you forked #jcenter http://jfrog.com/shownotes

Slide 24

Slide 24

Slide 25

Slide 25

You still have no idea what version are you using You have to connect each dependency as a submodule manually Switching branches and forks LOL Working on modules with other teams ROFL Still wrong! @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 26

Slide 26

@jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 27

Slide 27

The go dep experiment

Slide 28

Slide 28

@jbaruch #dockercon jfrog.com/shownotes

Slide 29

Slide 29

Working in project directories Proper dependency management? Local cache for dependencies Version declarations Conflict resolution @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 30

Slide 30

Conflict on the conflict resolution SAT/SMT vs MVS/SIV

Slide 31

Slide 31

Enter Go modules

Slide 32

Slide 32

Enter go modules @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 33

Slide 33

Backwards compatibility and migration @jbaruch #trianglegolang go mod init go.mod file is created The rest is the same: imports in code just work #jcenter http://jfrog.com/shownotes

Slide 34

Slide 34

That’s some serious magic… @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 35

Slide 35

Go modules convert everything (almost?) @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 36

Slide 36

What happens to go.mod when you add import (and run go get/go build)

Slide 37

Slide 37

Have goimport? no yes Follow goimport URL yes Invalid import path Serve the module no Is it a webpage ? no Is it a VCS? yes Build the module locally Clone the sources Access import path as URL

Slide 38

Slide 38

Latest compatible version tag

Slide 39

Slide 39

@jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 40

Slide 40

Let’s assume SemVer works (LOL) The latest version of v1.x.x is compatible with v1.0.0 and up Compatible?! Premise: import path string should always be backwards compatible @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 41

Slide 41

] Incompatible code can’t use the same import path Add /v2/ to the module path What about version 2?! Use /v2/ in the import path import “github.com/my/module/v2/mypkg” @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 42

Slide 42

What if it doesn’t have any semver tags?! @jbaruch #trianglegolang Pseudo version v0.0.0-yyyymmddhhmmss-abcdefabcdef #jcenter http://jfrog.com/shownotes

Slide 43

Slide 43

You can specify “version X or later”: >= x.y.z What if (when) I want to ban a version?! @jbaruch You can use exclude or replace for better control #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 44

Slide 44

Houston… I think I lost my module?

Slide 45

Slide 45

Modules shoud be immutable The <module>@v<version> construct should be immutable That means that github.com/myuser/myrepo/@v/v1.0.0 Should forever be the same…

Slide 46

Slide 46

But are they really? “ ”Friends don’t let friends do git push -f” Aaron Schlesinger @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 47

Slide 47

Using the goproxy variable export GOPROXY=https://myawesomeproxy.com go get github.com/myuser/myrepo

Slide 48

Slide 48

Have goimport? no yes Access import path as URL Follow goimport URL yes Invalid import path Serve the module no Is it a webpage ? no Is it a VCS? no Is it a module repo? yes Build the module locally Clone the sources yes

Slide 49

Slide 49

Keeping modules Local cache ($GOPATH/pkg/mod) Immediate access, not shared, can be wiped… Organizational cache (private proxy) Fast access, requires infra, shared across devs Public cache (public proxy) Highly available, CDN, no infra, free

Slide 50

Slide 50

Immutable and repeatable builds Immutable dependencies The best way to guarantee issues is force push Lost Dependencies Who doesn’t remember left-pad with Node.js? Internet Issues Even build when GitHub is down!?

Slide 51

Slide 51

And also faster builds…

Slide 52

Slide 52

From vendoring to hierarchy of module repositories

Slide 53

Slide 53

Go modules define an hierarchy of caches Public Modules Repository GoCenter Organizational Modules Repository The Athens Project JFrog Artifactory Local cache on the developer’s machine $GOPATH/pkg/mod

Slide 54

Slide 54

After the mods are resolved (or built) they are cached in $GOPATH/pkg/mod Local cache on the developer’s machine Provides immediate access Not shared Not reliable (can be wiped at any moment) @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 55

Slide 55

JFrog Artifactory or Project Athens Provides faster (Intranet) access Provides reproducible builds as it caches the dependencies used once for build reproduction Requires team infrastructure and maintenance (SaaS offers exist) Organizational modules repository @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 56

Slide 56

GoCenter Google announced a vision for a federation of public repositories Provides fast access Provides reproducible builds as it caches the popular and requested dependencies from version control Highly available, requires no maintenance, free public modules repositories @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 57

Slide 57

Documentation Stats More value to gocenter @jbaruch Nudges Ratings #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 58

Slide 58

Documentation Active Development # of stars Unit test coverage Licensing # of Dependents Passing unit tests # of forks # of maintainers Score of dependencies Security # of downloads Who the judges are?! @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes

Slide 59

Slide 59

Twitter ads and q&A jfrog.com/shownotes @jbaruch #trianglegolang gocenter.io @jbaruch #trianglegolang #jcenter http://jfrog.com/shownotes