A presentation at TestBash Manchester in in Manchester, UK by Areti Panou
Turning the Quality of Your Deployment Pipeline Into a Team Task Areti Panou, SAP October 3rd 2019 PUBLIC
Who is fixing the failing pipeline steps? unit tests build I guess somebody is already working on this code checks Security tests API tests Should I do something? deploy smoke test Was it me? Performance tests @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 2
The bystander effect @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 3
“The Somebody Else’s Problem field… relies on people’s natural predisposition not to see anything they don’t want to, weren’t expecting, or can’t explain” Douglas Adams @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC Life, the Universe and Everything 4
Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? @unremarkableQA How is the field deactivated? How to keep the field inactive?
Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? @unremarkableQA How is the field deactivated? How to keep the field inactive?
QAs were the messengers making any failure your problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 7
QAs didn’t trust developers to make failures their problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 8
Unless personally assigned, developers believed that pipeline step failures were somebody else’s problem @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 9
Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?
How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things Remove things Remove things you don’t want to see you don’t expect to see you can’t explain @unremarkableQA
How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things you don’t want to see @unremarkableQA Remove things Remove things you don’t expect to see you can’t explain
Things you don’t want to see Flaky steps Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 13
“Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Martin Fowler (@martinfowler) Eradicating Non-Determinism in Tests (2011) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 14
Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 15
Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 16
“A 30 minute code change took 2 weeks to get the acceptance tests working” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 17
“Introduce synthetic monitoring. This replaced our acceptance tests” Sarah Wells (@sarahjwells) Quality for ‘Cloud Natives’: what changes when your systems are complexed and distributed (2019) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 18
Things you don’t want to see Flaky steps Quarantine, stabilise or remove Long running steps Parallelise to save time Complicated-to-fix steps Simplify or replace @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 19
How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things you don’t want to see @unremarkableQA Remove things you don’t expect to see Remove things you can’t explain
Things you don’t expect to see Unresponsive external services New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 21
Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 22
Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 23
Make transparent who is already working on a fix john.doe@acme.com Claim – Jenkins plugin @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 24
Things you don’t expect to see Unresponsive external services Ping external services before starting the pipeline New unannounced steps Establish a new steps ritual Being the n-th person notified of the failed step Create public-awareness of who is resolving the failure @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 25
How is the field deactivated? “Keep only those things that speak to the heart, and discard items that no longer spark joy.” Marie Kondo Remove things Remove things you don’t want to see you don’t expect to see @unremarkableQA Remove things you can’t explain
Things you can’t explain Steps with unclear purpose Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 27
A good example of a bad step description @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 28
Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 29
Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 30
Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 31
A better example of a step description Name Description Name Description Security_tests Checks that APIs are secure Runs VS Severity Sec_API_Perm_tests Checks that only roles with the right permissions can access the APIs On commit Super high – Legal problems if broken Threshold 1 broken test = no deployment Wiki //acme/our-wiki/api-perm-tests Contacts jane.doe@acme.com (CET) @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 32
Things you can’t explain Steps with unclear purpose Make the value and information of the step visible Steps with unclear failure implications Make the impact and the reasons of the failure transparent Steps with unclear fix deadlines Set a time limit for the fixes @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 33
Addressing the “Somebody Else’s Problem” (SEP) field How is the field activated? How is the field deactivated? How to keep the field inactive?
Automatically assign someone as the owner of a failed step using a deterministic rule. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 35
The owner of a failed step should be responsible for the solution, not necessarily the fix. @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 36
Assign an owner of the quality of the pipeline rather than the pipeline steps @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 37
Educate and enable the whole team to become owner of the quality of the pipeline @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 38
Be prepared for a rough start or maybe not… @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 39
Turning the quality of your deployment pipeline into a team task Delays fixing failed pipeline steps Improve quality of pipeline steps Automatically assign owner of failed pipeline step solution Monitor improvements Assign pipeline quality owner Pipeline steps fixed always in acceptable time Enable everyone to become pipeline quality owner @unremarkableQA © 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ PUBLIC 40
Thank you. Contact information: Areti Panou Cloud Quality Coach @unremarkableQA unremarkabletester.com areti.panou@sap.com
It is around 16:00 on a Friday afternoon and somebody from the other end of the office says out loud: “Who is working on fixing the API tests?” A silence, as thick as a morning fog, covers the entire floor. Nobody has done anything.
Slowly, all eyes are turn towards the tester. How could she allow this to happen? When the API tests fail, the deployment pipeline is broken. When the pipeline is broken, there can be no deployment to production. The hope of a work-free weekend is slipping away…
This could be the beginning of a fictional, admittedly a bit corny, story. Unfortunately, in my experience, this is not something unheard of in an Agile software development team, where quality is a whole-team effort and not just the tester’s obligation.
So, how do we create a clear action plan to avoid such drama scenes?
The following resources were mentioned during the presentation or are useful additional information.
Here’s what was said about this presentation on social media.
for free. You
can too.