A presentation at Agile Testing Days in in Potsdam, Germany by Iancho Dimitrov
Iancho Dimitrov linkedin.com/in/iancho @iandim Agile Testing Days, Potsdam November 7th, 2019
GOAL To efficiently and sustainably provide actionable insight on the state of quality through efficient test execution. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 2
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 3
Trophies | Scars | Part-of-my-job Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 4
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 5
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 6
• Test Automation is not always - appropriate / necessary - cost effective • Even when appropriate and cost effective it often fails to deliver results – and to survive Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 7
Possible BENEFITS • reduced – cost of testing – time needed for testing (i.e. “time to market/production”) • more time spent by QA Engineers on meaningful work with higher added value Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 8
Possible BENEFITS • • test EXECUTION coverage o broader (incl. versions, environments, devices) o deeper (variations of test data) o tests that are impossible to do manually DevOps, CI/CD Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 9
Possible BENEFITS • test more often –> catch bugs early –> reduced bug-fixing cost • higher reliability (people make mistakes in repetitive tasks) • motivated, fulfilled and happy QA Engineers • can be used for monitoring on production Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 10
Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 11
…but it comes with its COSTS • commercial tools - licenses • infrastructure resources • team members training • PoC implementation Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 12
…but it comes with its COSTS • new tests design and implementation • (!) framework architecture setup, maintenance & enhancement • (!) tests execution, results analysis, reporting and defect tracking • (!) tests maintenance • test data management Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 13
Major RISKS • focus on tool selection – instead of on strategy, plan and process • slow start, high initial costs – and canceled prematurely • (!) exploding maintenance costs (changes in SUT, platforms, etc.) • lack of needed skills – and high hopes on “snake oil” tools Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 14
Major RISKS • wrong expectations to replace almost all other QA activities • wrong expectations for quick payback • (!) failure to achieve recognizable goals (esp. with none defined) • unhappy QA Engineers • TA assets to become “shelfware” – in full or partially Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 15
Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 16
Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 17
WRAP-UP • Test Automation is NOT a fit for every project • Many possible benefits – but they come at a cost • Many risks – achieving long-term success not trivial • Hard to start and carry on Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 18
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 19
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 20
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 21
a separate long-term project initiative Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 22 22
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 24
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 30
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 31
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 32
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 33
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 34
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 35
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 36
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 37
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 38
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 39
a separate long-term project initiative Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 40 40
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 41
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 45
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 46
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 47
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 48
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 53
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 54
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 55
Track and provide transparency on: • Results achieved • Time spent • Work done • Resources used …regularly – on Sprint/Release basis, but at least once in 2 or 3 weeks. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 56
Sample KPIs for Execution, Results Analysis and Reporting: • Number Test Suite runs • Number Test runs (and % of successful runs) • Number Executed “User actions”(→ man-hours saved) • Percentage “Test Run Results” reports delivered in time • Number Automated Test cycle time Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 57
Sample KPIs for New Tests Design & Implementation: • Percentage Sprint regression TA goal achieved (X out of Y) • Number New tests automated + “user actions” • Number New reusable components created • Number New Smoke tests automated • Number Test Cases designed for automation (and %) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 58
Sample KPI for Maintenance: • Number Existing tests adapted/maintained Framework/Tooling Enhancement: • < highlights on framework enhancements implemented > Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 59
Sample KPIs for Time spent (quantity & percentage) on: • run results analysis and reporting • new tests implementation • test maintenance • framework/tooling enhancement • other – test data preparation, process optimization, etc. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 60
KPIs on Doing Test Automation Right (continuous improvement) • track progress on achieving the goals in your prioritized list • track “health” status of all items already achieved (as they tend to deteriorate if not taken care of) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 61
The regular TA Status Report should be: • short, but comprehensive – 2-3 slides • addressed to and discussed with key stakeholders: - Project Manager / Budget Owner - Test Manager - Scrum Master / Dev Team Leader
TA Status Report – sample elements Effort Distribution (%) Sprint TA Scope DONE 120 100 80 60 40 20 0 Sprint 1 New Tests Impl. Sprint 2 Results Analysis Sprint 3 Maintenance Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing Sprint 4 Framework Enhancements 63
TA Status Report – sample elements Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 64
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 65
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 66
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 67
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 68
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 69
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 70
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 71
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 72
Iancho Dimitrov iancho.d@gmail.com linkedin.com/in/iancho @iandim Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 74
Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 75
Set up Process & Responsibilities for: • planning & case design → technical design → implementation • run → results analysis → reporting & defect tracking • (!) proactive test maintenance • framework/tooling enhancement Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 76
Make the most of TA (execution) results • distribute run results to all relevant team members (mailgroups) • meaningful descriptions – incl. screenshots, input data, logs • regularly update all team members on progress on key TA goals: - could be once in 3, 4 or 6 months - semi-formal gathering (e.g. a Lunch&Learn/Brown-bag session) - gather feedback Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 77
Assure links & traceability between artifacts: • User Story / Feature • Test Case • Test Run Reports • Defects Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 78
Design for automation: • the right test cases • designed for automated testing • decomposed to the needed level of reusable components (vs. automating tests designed for manual testing) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 79
Cross-functional team members involvement in TA • Developers – assuring TA-friendly software • BA/PO/PMs – design & prioritization, esp. “what’s next” • Test Management – integration of TA in the QA Strategy & Plan …and vice versa - TA Engineers – biz domain; features, requirements Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 80
• Do Test Automation Right - so many things to address: - make a goals list – prioritized - reap the low-hanging fruit (things mostly within your control) - work on achieving the goals, track progress • Measure | Communicate | Optimize Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 81
• Don’t expect to achieve everything quickly • Make a custom list for your initiative – and prioritize it • Reap the low-hanging fruit • Set deadlines, track progress – and work on these one step at a time Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 82
☺ Iancho Dimitrov iancho.d@gmail.com LinkedIn.com/in/iancho Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 83
Test Automation is a great way to achieve agility, save time and reduce the amount of tedious tasks your QA team needs to handle. On the other hand, it is neither a silver bullet for assuring quality, nor it is applicable for every project. In order for it to be part of a project it must surely make business sense.
How do you evaluate and justify the introduction of Test Automation in your project – and keeping it in the long-term? How do you integrate it in your QA Strategy and Plan in a way that brings maximized benefits that are recognized by the project team, management and business stakeholders? What measurable goals and metrics can you set, track and report so you transparently show the return of investment in Test Automation? How to regularly present information about work done and results achieved in a way understandable by a broader audience? What prioritization approaches can help you achieve biggest impact? What can you do to keep “costs” under control and not have maintenance efforts “explode”?
Join this session where I’ll share my thoughts and experience on these topics – and I’ll present examples, ideas and approaches that worked well for multi-year automated testing initiatives I have been responsible for.
Structuring your Test Automation initiative in a way that assures maximized benefits, recognized by the team, management and business stakeholders.
How to evaluate and justify the introduction of Test Automation in your project.
Key success factors - and more importantly, most common pitfalls and reasons for Test Automation initiatives failure.
Measurable goals and metrics for tracking and reporting to transparently show the return on investment in Test Automation.
Prioritization and other approaches for what, how and when to automate - to ensure maximized benefits and minimized costs.
Here’s what was said about this presentation on social media.
for free. You
can too.