#DataDrivenDevOps

A presentation at DevOps Days Boise 2018 in June 2018 in Boise, ID, USA by Baruch Sadogursky

Slide 1

Slide 1

DataDrivenDevOps Setting up your DevOps team to success with data @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 2

Slide 2

shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch @DevOpsDaysBoise #datadrivendevops http://jfrog.com/shownotes

Slide 3

Slide 3

About me @ jbaruch

Slide 4

Slide 4

Cobbler's children have no shoes

Slide 5

Slide 5

Poll time! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes How many of you are software engineers? How many of you are optimists? How many of you are self

confident in their work?

Slide 6

Slide 6

Dunning

Kruger Effect a.k.a. ”optimism” People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes “

Slide 7

Slide 7

Second

system effect a.k.a. “self

confidence” The tendency of small, elegant, and successful systems, to be succeeded by over

engineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man

Month @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes “

Slide 8

Slide 8

The consequences @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 9

Slide 9

disillusion @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Software I like Software I know really well

Slide 10

Slide 10

overconfidence @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Every software is 80% ready 80% of the time Every developer, ever “

Slide 11

Slide 11

Hard to give up @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 12

Slide 12

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 13

Slide 13

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 14

Slide 14

How departments measure @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering

Slide 15

Slide 15

How departments measure @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering

Slide 16

Slide 16

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 17

Slide 17

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 18

Slide 18

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 19

Slide 19

Engineering metrics in pre

devops era @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Dev • Velocity • Architecture • Compliance QA • Incidents • Defects • MTTR • Code coverage • Test suite stability Ops • SLA • Cost env setup • Average customer cost • Tools stability

Slide 20

Slide 20

DEVOPS! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes ?

Slide 21

Slide 21

Velocity to agile is like ? To devops … @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes ?

Slide 22

Slide 22

Velocity to agile is like profit To devops … @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Profit

Slide 23

Slide 23

Very much alike! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗

Slide 24

Slide 24

Very much alike! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗

Slide 25

Slide 25

Very much alike! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗

Slide 26

Slide 26

Poll time! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn't look right?

Slide 27

Slide 27

What’s the problem? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Velocity

Slide 28

Slide 28

True for both! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗

Slide 29

Slide 29

Too coarse! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Profit

Slide 30

Slide 30

Too coarse! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes DevOps

Slide 31

Slide 31

This is devops * @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Ops QA Dev Common goals, tools, culture Deep specialization *Unless you’re netfliX **You’re not

Slide 32

Slide 32

This is devops @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Ops QA Dev

Slide 33

Slide 33

Same for metrics! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Ops QA Dev profit Specialized metrics

Slide 34

Slide 34

Same for metrics! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Ops QA Dev

Slide 35

Slide 35

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 36

Slide 36

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 37

Slide 37

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 38

Slide 38

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 39

Slide 39

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 40

Slide 40

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 41

Slide 41

How do metrics collaborate? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Te s t s u i t e s t a b i l i t y • Incoming defect velocity • Te s t e x e c u t i o n t i m e • Time to create new QA lab influenced affected

Slide 42

Slide 42

Cooking with metrics @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 43

Slide 43

Metrics categorization @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 44

Slide 44

Why? @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 45

Slide 45

ROI – is this activity is the right thing to do? @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 46

Slide 46

Continuous improvement @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 47

Slide 47

trust @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 48

Slide 48

sampling @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 49

Slide 49

Complete sample @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 50

Slide 50

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 51

Slide 51

Representative sample @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 52

Slide 52

How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 53

Slide 53

Predicting the future @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 54

Slide 54

Meet scrum team “avengers” @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 55

Slide 55

anecdotes from the daily scrum stand

ups @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” “

Slide 56

Slide 56

Let’s gather some data! 0 2 4 6 8 10 wee k 1 wee k 2 wee k 3 wee k 4 wee k 5 wee k 6 wee k 7 wee k 8 wee k 9 wee k 10 wee k 11 wee k 12 wee k 13 wee k 14 Environment Utilization Util i zed Availab le @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 57

Slide 57

That doesn’t look good! But why? 0 5 10 15 20 25 Raj Jan e Christina Hel en Le e Baruc h Joh n Sc ot t Nata ly Ruppert Jos h Pa t r ic k Mi c h ae l 1 2 3 6 7 10 13 13 14 15 16 18 25 Days the environment stays checked out Days ch ec ked o u t @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 58

Slide 58

Continuous improvement @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes 0 2 4 6 8 10 12 14 16 wee k 1 wee k 3 wee k 5 wee k 7 wee k 9 wee k 11 wee k 13 wee k 15 wee k 17 wee k 19 wee k 21 wee k 23 Environment Utilization Util i zed Availab le

Slide 59

Slide 59

anecdotes from the daily scrum stand

ups @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” “

Slide 60

Slide 60

Let’s gather some data! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 61

Slide 61

Return on investment @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 62

Slide 62

anecdotes from the daily scrum stand

ups @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” “

Slide 63

Slide 63

Let’s gather some data! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 64

Slide 64

Trust and predictability @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes 0: 0 0 0: 4 0 1: 2 0 2: 0 0 2: 4 0 3: 2 0 4: 0 0 4: 4 0 5: 2 0 6: 0 0 6: 4 0 7: 2 0 8: 0 0 8: 4 0 9: 2 0 10 : 00 10 : 40 11 : 20 12 : 00 12 : 40 13 : 20 14 : 00 14 : 40 15 : 20 16 : 00 16 : 40 17 : 20 18 : 00 18 : 40 19 : 20 20 : 00 20 : 40 21 : 20 22 : 00 22 : 40 23 : 20 GC time GC ti me

Slide 65

Slide 65

Dos and Don’ts

Slide 66

Slide 66

Measuring the right thing @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 67

Slide 67

LoC LOL, anyone? @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 68

Slide 68

Avoid data overload @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 69

Slide 69

Use reliable data @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 70

Slide 70

Common vocabulary is important @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes ”medium” In russia ”medium” In brazil

Slide 71

Slide 71

Metrics generate incentives, be aware! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 72

Slide 72

Tweak as you go @ jb a ru c h @ DevOpsDaysBoise

datadrivendevops http:// jfro g .c o m / show notes

Slide 73

Slide 73

Elevate! @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Raw Data Product Specific Dashboards Integrated Dashboards

Slide 74

Slide 74

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 75

Slide 75

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 76

Slide 76

@ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes

Slide 77

Slide 77

How departments measure @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering

Slide 78

Slide 78

Step into data

driven life @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes Removes blame game Builds accountability and trust Creates common base for discussion

Slide 79

Slide 79

Q&A and twitter aids @ jbaruch @ DevOpsDaysBoise

datadrivendevops http:// jfrog.com / shownotes @ jbaruch @ devopsdaysboise

datadrivendevops http:// jfrog.com / shownotes