Turning the Quality of Your Deployment Pipeline Into a Team Task

Turning the Quality of Your Deployment Pipeline Into a Team Task Areti Panou, SAP June 12th, 2020 PUBLIC

Who is fixing the failing pipeline steps? unit tests build I guess somebody is already working on this code checks Security tests API tests Should I do something? deploy smoke test Was it me? Performance tests @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

“The Somebody Else’s Problem field… relies on people’s natural predisposition not to see anything they don’t want to, weren’t expecting, or can’t explain” Douglas Adams @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Life, the Universe and Everything 4

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? @unremarkableQA How is the field deactivated? How to keep the field inactive?

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things Remove things Remove things you don’t want to see you don’t expect to see you can’t explain @unremarkableQA

How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things you don’t want to see @unremarkableQA Remove things Remove things you don’t expect to see you can’t explain

“Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Martin Fowler (@martinfowler) Eradicating Non-Determinism in Tests (2011) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 14

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 15

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 16

“A 30 minute code change took 2 weeks to get the acceptance tests working” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 17

“Introduce synthetic monitoring. This replaced our acceptance tests” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 18

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps Simplify or replace @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 19

How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things you don’t want to see @unremarkableQA Remove things you don’t expect to see Remove things you can’t explain

Things you don’t expect to see Unresponsive external services New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 21

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 22

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 23

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step Create public-awareness of who is resolving the failure @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 25

How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things Remove things you don’t want to see you don’t expect to see @unremarkableQA Remove things you can’t explain

Things you can’t explain Steps with unclear purpose Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 27

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 29

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 30

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 31

A better example of a step description Name Description Name Description Security_tests Checks that APIs are secure Runs VS Severity Sec_API_Perm_tests Checks that only roles with the right permissions can access the APIs On commit Super high – Legal problems if broken Threshold 1 broken test = no deployment Wiki //acme/our-wiki/api-perm-tests Contacts jane.doe@acme.com (CET) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 32

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 33

How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things Remove things Remove things you don’t want to see you don’t expect to see you can’t explain @unremarkableQA

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? @unremarkableQA How is the field deactivated? How to keep the field inactive?

Turning the quality of your deployment pipeline into a team task Delays fixing failed pipeline steps Improve quality of pipeline steps Automatically assign owner of failed pipeline step solution Monitor improvements Assign pipeline quality owner Pipeline steps fixed always in acceptable time Enable everyone to become pipeline quality owner @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 42

Thank you. Contact information: Areti Panou Product Owner – Release Management Tools @unremarkableQA unremarkabletester.com areti.panou@sap.com

Appendix

References • https://yorkesoftware.com/2016/06/06/the-effect-of-bystander-syndrome-on-an-agile-team/ • https://en.wikipedia.org/wiki/Bystander_effect • https://www.getdonedone.com/somebody-elses-problem/ • https://konmari.com/ • https://speakerdeck.com/sarahjwells/european-test-conference-2019-quality-for-cloud-nativeswhat-changes-when-your-systems-are-complex-and-distributed • https://plugins.jenkins.io/claim • https://martinfowler.com/articles/nonDeterminism.html • https://blog.liberationist.org/how-to-design-team-rituals-to-accelerate-change-4fc43a1acdc0 © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 45

Turning the Quality of Your Deployment Pipeline Into a Team Task

Slide 1

Slide 2

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

Slide 17

Slide 18

Slide 19

Slide 20

Slide 21

Slide 22

Slide 23

Slide 24

Slide 25

Slide 26

Slide 27

Slide 28

Slide 29

Slide 30

Slide 31

Slide 32

Slide 33

Slide 34

Slide 35

Slide 36

Slide 37

Slide 38

Slide 39

Slide 40

Slide 41

Slide 42

Slide 43

Slide 44

Slide 45