Turning the Quality of Your Deployment Pipeline Into a Team Task Areti Panou, SAP May 14, 2019 PUBLIC

Who is fixing the failing pipeline steps? unit tests build I guess somebody is already working on this code checks Security tests API tests Should I do something? deploy smoke test Was it me? Performance tests @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2

The bystander effect @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3

“The Somebody Else’s Problem field… relies on people’s natural predisposition not to see anything they don’t want to, weren’t expecting, or can’t explain” Douglas Adams @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Life, the Universe and Everything 4

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

QAs were the messengers making any failure your problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 7

QAs didn’t trust developers to make failures their problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8

Unless personally assigned, developers believed that pipeline step failures were somebody else’s problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 9

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 11

“Keep only those things that speak to the heart, and discard items that no longer spark joy.” What is the KonMari Method? @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 12

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 13

Things you don’t want to see Flaky steps Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 14

“Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Martin Fowler (@martinfowler) Eradicating Non-Determinism in Tests (2011) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 15

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 16

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 17

“A 30 minute code change took 2 weeks to get the acceptance tests working” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 18

“Introduce synthetic monitoring. This replaced our acceptance tests” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 19

Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps Simplify or replace @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 20

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 21

Things you don’t expect to see Unresponsive external services New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 22

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 23

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 24

Make transparent who is already working on a fix john.doe@acme.com Claim – Jenkins plugin @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 25

Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step Create public-awareness of who is resolving the failure @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 26

Remove things you don’t want to see How is the field deactivated? Remove things you don’t expect to see Remove things you can’t explain @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 27

Things you can’t explain Steps with unclear purpose Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 28

A good example of a bad step description @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 29

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 30

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 31

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 32

A better example of a step description Name Description Name Description Security_tests Checks that APIs are secure Runs VS Severity Sec_API_Perm_tests Checks that only roles with the right permissions can access the APIs On commit Super high – Legal problems if broken Threshold 1 broken test = no deployment Wiki //acme/our-wiki/api-perm-tests Contacts jane.doe@acme.com (CET) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 33

Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 34

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Automatically assign someone as the owner of a failed step using a deterministic rule. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 36

The owner of a failed step should be responsible for the solution, not necessarily the fix. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 37

Assign an owner of the quality of the pipeline rather than the pipeline steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 38

Educate and enable the whole team to become owner of the quality of the pipeline @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 39

Be prepared for a rough start or maybe not… @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 40

Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?

Turning the quality of your deployment pipeline into a team task Delays fixing failed pipeline steps Improve quality of pipeline steps Automatically assign owner of failed pipeline step solution Monitor improvements Assign pipeline quality owner Pipeline steps fixed always in acceptable time Enable everyone to become pipeline quality owner @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 42

Thank you. Contact information: Areti Panou Cloud Quality Coach areti.panou@sap.com @unremarkableQA unremarkabletester.com