Turning the Quality of Your Deployment Pipeline Into a Team Task

A presentation at DevOpsDays Zürich in May 2019 in Zürich, Switzerland by Areti Panou

Slide 1

Slide 1

Turning the Quality of Your Deployment Pipeline Into a Team Task Areti Panou, SAP May 14, 2019 PUBLIC

Slide 2

Slide 2

Who is fixing the failing pipeline steps? unit tests build I guess somebody is already working on this code checks Security tests API tests Should I do something? deploy smoke test Was it me? Performance tests @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

Slide 3

Slide 3

The bystander effect @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3

Slide 4

Slide 4

“The Somebody Else’s Problem field… relies on people’s natural predisposition not to see anything they don’t want to, weren’t expecting, or can’t explain” Douglas Adams @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Life, the Universe and Everything 4

Slide 5

Slide 5

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Slide 6

Slide 6

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Slide 7

Slide 7

QAs were the messengers making any failure your problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 7

Slide 8

Slide 8

QAs didn’t trust developers to make failures their problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8

Slide 9

Slide 9

Unless personally assigned, developers believed that pipeline step failures were somebody else’s problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 9

Slide 10

Slide 10

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Slide 11

Slide 11

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 11

Slide 12

Slide 12

“Keep only those things that speak to the heart, and discard items that no longer spark joy.” What is the KonMari Method? @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 12

Slide 13

Slide 13

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 13

Slide 14

Slide 14

Things you don’t want to see Flaky steps Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 14

Slide 15

Slide 15

“Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Martin Fowler (@martinfowler) Eradicating Non-Determinism in Tests (2011) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 15

Slide 16

Slide 16

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 16

Slide 17

Slide 17

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 17

Slide 18

Slide 18

“A 30 minute code change took 2 weeks to get the acceptance tests working” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 18

Slide 19

Slide 19

“Introduce synthetic monitoring. This replaced our acceptance tests” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 19

Slide 20

Slide 20

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps Simplify or replace @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 20

Slide 21

Slide 21

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 21

Slide 22

Slide 22

Things you don’t expect to see Unresponsive external services New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 22

Slide 23

Slide 23

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 23

Slide 24

Slide 24

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 24

Slide 25

Slide 25

Make transparent who is already working on a fix john.doe@acme.com Claim – Jenkins plugin @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 25

Slide 26

Slide 26

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step Create public-awareness of who is resolving the failure @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 26

Slide 27

Slide 27

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 27

Slide 28

Slide 28

Things you can’t explain Steps with unclear purpose Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 28

Slide 29

Slide 29

A good example of a bad step description @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 29

Slide 30

Slide 30

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 30

Slide 31

Slide 31

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 31

Slide 32

Slide 32

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 32

Slide 33

Slide 33

A better example of a step description Name Description Name Description Security_tests Checks that APIs are secure Runs VS Severity Sec_API_Perm_tests Checks that only roles with the right permissions can access the APIs On commit Super high – Legal problems if broken Threshold 1 broken test = no deployment Wiki //acme/our-wiki/api-perm-tests Contacts jane.doe@acme.com (CET) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 33

Slide 34

Slide 34

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 34

Slide 35

Slide 35

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Slide 36

Slide 36

Automatically assign someone as the owner of a failed step using a deterministic rule. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 36

Slide 37

Slide 37

The owner of a failed step should be responsible for the solution, not necessarily the fix. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 37

Slide 38

Slide 38

Assign an owner of the quality of the pipeline rather than the pipeline steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 38

Slide 39

Slide 39

Educate and enable the whole team to become owner of the quality of the pipeline @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 39

Slide 40

Slide 40

Be prepared for a rough start or maybe not… @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 40

Slide 41

Slide 41

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Slide 42

Slide 42

Turning the quality of your deployment pipeline into a team task Delays fixing failed pipeline steps Improve quality of pipeline steps Automatically assign owner of failed pipeline step solution Monitor improvements Assign pipeline quality owner Pipeline steps fixed always in acceptable time Enable everyone to become pipeline quality owner @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 42

Slide 43

Slide 43

Thank you. Contact information: Areti Panou Cloud Quality Coach areti.panou@sap.com @unremarkableQA unremarkabletester.com