1 Browser Test Automation (in 2018) @ ColinBendell CTO Office, Cloudinary

Agenda Te s t M e t h o d o l o g i e s We bDr i ve r. i o Mocha tests How Browsers Work & Browser APIs We b Pl at for m Te st Mobile & Device Emulation Accessibility Operational: DNS, Paralyzing, CI Real User Testing 9:00 AM Welcome | 10:30 AM Break | 12:00 AM Lunch | 2:30 PM Break | 4:00 PM Wrap

up & Q&A

Cloudinary's Two Probl ems Felix Zilber New Digital

Asset

Management Front

End Automating CDN Changes

5 Why Do We Test?

10 Te s t i n g : P r e p a r e f o r t h e U n e x p e c t e d

“I just want to buy my Mom’s birthday present.” “I just want to know if it will rain today.” “I just want to share pictures of my vacation”

Te s t i n g = = = M e e t i n g U s e r E x p e c t a t i o n s

Te s t i n g M o d e l s

Kinds of Testing https:// stackoverflow.com /questions/2741832/unit

tests

vs

functional

tests

Functional v. Non

Functional

Unit Testing • Smallest testable component • Usually simple inputs and outputs • Moch objects v. skeleton dev services

Unit Testing const expect

require( "chai" ).expect; const converter

require( "../app/converter" ); describe( "Color Code Converter" , function () { describe( "RGB to Hex conversion" , function () { it( "converts the basic colors" , function () { var redHex

converter.rgbToHex ( 255 , 0 , 0 ); var greenHex

converter.rgbToHex ( 0 , 255 , 0 ); var blueHex

converter.rgbToHex ( 0 , 0 , 255 ); expect( redHex ). to.equal ( "ff0000" ); expect( greenHex ). to.equal ( "00ff00" ); expect( blueHex ). to.equal ( "0000ff" ); }); }); });

Unit Testing • Blurred line to integration tests • Encourages small component logic v. monoliths • Large volume of Tests

Integration Tests

Integration Tests describe( "Color Code Converter API" , function () { describe( "RGB to Hex conversion" , function () { let url

"http://localhost:3000/ rgbToHex?red =255&green=255&blue=255" ; it( "returns status 200" , function () {}); it( "returns the color in hex" , function () {}); }); describe( "Hex to RGB conversion" , function () { let url = "http://localhost:3000/ hexToRgb?hex =00ff00" ; it( "returns status 200" , function () {}); it( "returns the color in RGB" , function () {}); }); });

System Tests • Does it match design? • Meet expectations of design • Often focused on workflows • Vol ume, l oad, stress, securi ty • Usability, Accessibility

System Tests

Acceptance Tests • Meet business objectives • End

to

End • Can the end user accomplish what we wanted it to do? • Alpha/Beta Feedback cycle

Kinds of Testing

“We develop in the Zoo, But release in the Jungle”

Melanie Cey

Kinds of Testing https:// stackoverflow.com /questions/2741832/unit

tests

vs

functional

tests

Why do we need browser testing • Functional User experience • Wo r k f l o ws • Design & Layout • Preserve Presentation • Above the fold, responsive web • Mobi l e experi ences v. Watch v. ?? • Accessibility • Screen readers • audits • Performance measurement • Performance budgets • Request flows

Browser Eco System

2007 2008 2009 2010 Present IE7 iPhone Firefox 2 IE7 iPhone 3G Firefox 3 Android Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad ??

2012: 3,977 Different Android Devices Source: OpenSi gnal 2012: 3,977 Different Android Devices

Source: OpenSignal 2014: 18.7k Different Android Devices

Source: OpenSignal 2015: 24.0k Different Android Devices

How many browsers in a typical Thanksgiving weekend?

How many browsers in a typical Thanksgiving weekend? 572

2007 2008 2009 2010 Present IE7 iPhone Firefox 2 IE7 iPhone 3G Firefox 3 Android Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad 572 iOS Safari MacOS Safari Android Chrome Desktop Chrome Samsung IB Opera …

“…25% of new Android phones have only 512MB of RAM. ”

Jen Fitzpatrick VP of product management for Google Maps

Browser Based Te s t i n g

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

#1 reason checklists fail to catch browser regression: Humans

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

metric MAE baseline.png testoutput.png null

same1.png[0] PNG 640x400 640x400+0+0 8

bit DirectClass 1.64KB 0.010u 0:00.019

same2.jpg[0] JPEG 640x400 640x400+0+0 8

bit DirectClass 3.65KB 0.000u 0:00.009

0.196766 (3.00245e

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

metric MAE baseline.png testoutput.png null

same1.png[0] PNG 640x400 640x400+0+0 8

bit DirectClass 1.64KB 0.010u 0:00.019

same2.jpg[0] JPEG 640x400 640x400+0+0 8

bit DirectClass 3.65KB 0.000u 0:00.009

0.196766 (3.00245e

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

metric MAE baseline.png testoutput.png null

same1.png[0] PNG 640x400 640x400+0+0 8

bit DirectClass 1.64KB 0.010u 0:00.019

same2.jpg[0] JPEG 640x400 640x400+0+0 8

bit DirectClass 3.65KB 0.000u 0:00.009

0.196766 (3.00245e

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

metric MAE baseline.png testoutput.png null

same1.png[0] PNG 640x400 640x400+0+0 8

bit DirectClass 1.64KB 0.010u 0:00.019

same2.jpg[0] JPEG 640x400 640x400+0+0 8

bit DirectClass 3.65KB 0.000u 0:00.009

0.196766 (3.00245e

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

OS API application control • Access the window model directly • Simulate Keyboard strokes and mouse movements • Interact with any application like a human • Screen Readers use this approach to supersede the OS accessibility features

Programmatic Control

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

Selenium Evolution • Selenium • Selenium Core • Selenium RC Selenium 1 • Selenium WebDriver Selenium 2 • We b D r i v e r Selenium 3 Gecko Driver (+Marionette) Chrome Driver EdgeDriver Apple Safari Driver Firefox Driver Safari Driver … https:// www.seleniumhq.org /download/

Selenium 3 (W3C WebDriver) Selenium Bindings Java Ruby C# Python JavaScript ? Selenium Grid ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera JSON Wire Protocol WebDriver Wire Protocol HTTP HTTP

We bDr i ve r. i o We b Dr i ve r (or Selenium Grid) ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera WebDriver Wire Protocol HTTP HTTP NodeJS

curl – X POST http://localhost:9515/session

d '{" desiredCapabilities ":{" browserName ":"chrome"}} ' { "sessionId" : "your

session

id

here" , "status" : 0 , "value" :{...}} curl http: / /localhost:9515/session/$SESSION_ID/ url \

d '{" url ":"http:// www.cnn.com /"} ' { " sessionId " : "..." , "status" : 0 , "value" : null } curl http: / /localhost:9515/session/$SESSION_ID/element

d '{"using":"tagName","value":"h1"}' { " sessionId " : "..." , "status" : 0 , "value" :{ " ELEMENT" : "element

object

id

here " }} curl http: / /localhost:9515/session/$SESSION_ID/element/$OBJECT_ID/text { " sessionId " : "..." , "status" : 0 , "value" : "Example Domain" } curl – X DELETE http: //localhost:9515/session/your

session

id

here { " sessionId " : "..." , "status" : 0 , "value" : null }

WebDriver.io

To o l c h a i n o v e r v i e w We bDr i ve r. i o Jenkins We b Platform Te s t Allure Chrome Driver Mocha Chrome Axe

Exercise 1: Getting Setup https:// github.com / colinbendell / webdriver

workshop Request https:// duckduckgo.com Enter search term ( eg : "Mr. Dressup ") Navigate to the 4th result and report the

git checkout https:// github.com / colinbendell / webdriver

workshop

npm install chromedriver

npm install webdriverio

const webdriverio

require ( ' webdriverio ' );

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; const webdriverio

require ( ' webdriverio ' );

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

require ( ' webdriverio ' );

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

require ( ' webdriverio ' ); browser.init ();

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

require ( ' webdriverio ' ); browser.init (); browser.url ( 'https:// amazon.com /' ); browser.saveScreenshot ( amazon.png ');

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

require ( ' webdriverio ' ); browser.init ();
browser.url ( 'https:// amazon.com /' ); browser.saveScreenshot ( amazon.png '); browser.end ();

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

require ( ' webdriverio ' ); browser.init ();
browser.url ( 'https:// amazon.com /' ); browser.saveScreenshot ( amazon.png '); browser.end ();

Exercise 1: Getting Setup https:// github.com / colinbendell / webdriver

workshop Request https:// duckduckgo.com Enter search term ( eg : "Mr. Dressup ") Navigate to the 4th result and report the

Te s t i n g with Mocha

https:// mochajs.org /

http:// www.chaijs.com

npm install chai wdio

mocha

framework

npm install wdio

chromedriver

service

npm install allure allure

commandline wdio

allure

reporter

wdio config

========================= WDIO Configuration Helper

? Where do you want to execute your tests? On my local machine ? Which framework do you want to use? mocha ? Shall I install the framework adapter for you? Yes ? Where are your test specs located? ./ src /**/*_ spec.js ? Which reporter do you want to use? dot, allure ? Shall I install the reporter library for you? Yes ? Do you want to add a service to your test setup? Chromedriver ? Shall I install the services for you? Yes ? Level of logging verbosity error ? In which directory should screenshots gets saved if a command fails? ./ errorShots / ? What is the base url ? http://localhost

Exercise 2: Wo r k f l o w Te s t

Adding TODO spec Install mocha, chai, allure create wdio.conf.js Navigate to todomvc.com Using Angular implementation: • add a To D o • marking it complete • clear the list

Exercise 3: POM spec Repeat Exercise 2 Use Page Object Model Simplify Allure output with wdio

allure

ts

Ta ke A c t i o n • Create workflows easily with mocha • Use a Page

Object

Model to abstract selectors for reduced maintenance • Create good & actionable outputs

How Browsers Work

How Browsers Are Supposed to work

Page Load Time: Seconds

2 Second Magic Number Page Load Time: Seconds

! " ! 50ms DNS Lookup 100ms ShoesByColin.com 8.8.8.8

! " ! 50ms DNS Lookup 100ms ShoesByColin.com 8.8.8.8

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms ShoesByColin.com 8.8.8.8

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms ShoesByColin.com 8.8.8.8

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms

• ?? ms ShoesByColin.com 8.8.8.8

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms

• ?? ms ~ 500ms Network
• 500ms? App

HTML DOM JavaScript CSS Render Tr e e Layout Paint CSSOM Images Video JS Exec

Browser APIs

Real User Monitoring: JavaScript Timers Steve Souders Philip Te l l i s https:// github.com / stevesouders /episodes https:// github.com /lognormal/boomerang

https://www.w3.org/wiki/ We b_ Pe r fo r manc e /Publications

Navigation Timing API

window.performance

{ " timeOrigin ": 1528900579651.6672, "timing": { " navigationStart ": 1528900579651, " unloadEventStart ": 0, " unloadEventEnd ": 0, " redirectStart ": 0, " redirectEnd ": 0, " fetchStart ": 1528900583010, " domainLookupStart ": 1528900579730, " domainLookupEnd ": 1528900579730, " connectStart ": 1528900579730, " connectEnd ": 1528900579761, " secureConnectionStart ": 1528900579733, " requestStart ": 1528900579761, " responseStart ": 1528900582922, " responseEnd ": 1528900583010, " domLoading ": 1528900583014, " domInteractive ": 1528900583849, " domContentLoadedEventStart ": 1528900583849, " domContentLoadedEventEnd ": 1528900583876, " domComplete ": 1528900585616, " loadEventStart ": 1528900585617, " loadEventEnd ": 1528900585856 }, "navigation": {"type": 0, " redirectCount ": 0 } }

function getPageLoadTime () { let timing = window.performance.timing ; return timing.loadEventStart

timing.navigationStart }

What about … Javascript | CSS | WebFonts | Images | Videos | XHR/Fetch

window.performance

{ "name": "https:// ajax.googleapis.com /ajax/libs/ jquery /1.11.2/ jquery.min.js ", " entryType ": "resource", " startTime ": 928.70000000039, "duration": 425.8000000008906, " initiatorType ": "script", " nextHopProtocol ": "h2", " workerStart ": 0, " redirectStart ": 0, " redirectEnd ": 0, " fetchStart ": 928.70000000039, " domainLookupStart ": 937.4000000025262, " domainLookupEnd ": 945.9000000024389, " connectStart ": 945.9000000024389, " connectEnd ": 967.5999999999476, " secureConnectionStart ": 946.2000000021362, " requestStart ": 949.5999999999185, " responseStart ": 970.9000000002561, " responseEnd ": 1354.5000000012806, " transferSize ": 33798, " encodedBodySize ": 33495, " decodedBodySize ": 95931, " serverTiming ": [] }

let url = “https:// bendell.ca / logo.png ”; let me = performance.getEntriesByName ( url )[0]; let timings = { loadTime : me. duration , dns : me. domainLookupEnd

me. domainLookupStart , tcp : me. connectEnd

me. connectStart , waiting: me. responseStart

me. requestStart , fetch: me. responseEnd

me. responseStart }

200 OK HTTP/1.1 Timing

Allow

Origin: * Content

Type: application/ javascript Content

Length: 1234

• Slowest resources • Time to first image • Response time by domain • Time a group of assets • Response time by initiator type (element type) • Browser cache

hit ratio for resources • Resource Timing API • Other Uses More: http://www.slideshare.net/bluesmoon/beyond

page

level

metrics

How long does it take to display the main product image on my site?

function loadTemplate () { performance.mark (“ startTask ”); doTemplateStuff () //do stuff (assumes promise()) .then(() => { performance.mark (“ stopTask ”); //Measure the duration between the two marks performance.measure (“ taskDuration ”, “ startTask ”,“ stopTask ”); }); }

< img src ="image1.gif" onload=" performance.mark (' hero_image ')"> https:// stevesouders.com /blog/2015/05/12/hero

image

custom

metrics/

GET /resource HTTP/1.1 Host: example.com HTTP/1.1 200 OK Server

Timing: cdn , dur="130" Server

Timing: cloudinary , dur="10", desc ="new trans"

{ let url = 'https:// example.com / resource.jpg ’; let severEntries [] = window.performance.getEntriesByName ( url ) . filter (( name ) => name === " server ") ; }

let config = { attributes: true, childList : true, subtree: true }; let targetNode

document.querySelector ('body'); let callback = function(mutations) { mutations.forEach (function(mutation) { if ( mutation.type === ' childList ') { let values = []. slice.call ( list.children ) .map(node => node.nodeName ) .filter(n => / img / i.test (n)); console.log ( list_values ); } }); }; let observer = new MutationObserver (callback); observer.observe ( targetNode , config);

Exercise 4: performance budget Navigate to amazon.com Calculate the total bytes downloaded Ensure that the page load time < 2s and bytes < 2MiB Repeat with Disney.com

Ta ke A c t i o n • Add performance.mark () to measure YOUR key metrics • Eg : Hero Image, Product Image, Sport Scores • Keep performance.mark () in production code! • Define Hard and Soft targets that synthetic tests should hit • BHAG: 2s & 1MiB

Mobile Display & Network Throttling Tim Kadlec

chromeOptions : { args :[ "

incognito" , "

headless", "

disable

gpu " ] },

chromeOptions : { args :[ "

incognito" , "

disable

gpu " ], mobileEmulation : { deviceMetrics : { width : 360, height : 720, pixelRatio : 3}, networkThrottle : 'Fast 3G' } }

Ta ke A c t i o n • Run Tests in different viewport configurations to test RWD • Experiment with different Network Conditions • Similarly with different CPU throttling

Web Platform Test

https://web

platform

tests.org

https://platform.html5.org

https:// darobin.github.io /test

harness

tutorial/docs/using

testharness.html

Why Inside v. Outside Browser Testing? • Hawthorne Effect • Platform Widgets & Controls • APIs

Exercise 5: WPT Implement Responsive Images and Test the naturalWidth

A11y with AXE Léonie Watson Marcy Sutton Katie Sylor

Miller

“Websites, tools, and technologies are designed and developed so that people with disabilities can use them .” ACCESSIBILITY (a11y )

W3C Web Accessibility Initiative

THE AMERICANS WITH DISABILITIES ACT (ADA) “No individual shall be discriminated against on the basis of disability in the full and equal enjoyment of the goods, services, facilities, privileges, advantages, or accommodations of any place of public accommodation ”

ADA Sec. 12182 (a)

AMERICANS WITH A DISABILITY UNITED STATES CENSUS REPORT, JULY 2012 57 M 19%

The Basics • We b Conte nt Ac c e s s i bi l i t y Gui de l i ne s ( WCAG 2. 0)
• Developed by the W3C Web Accessibility Three levels of compliance

A, AA, AAA

Exercise 6: A11y Add axe

core npm Load 3 news websites Compare the violations Create a test case to fail on a11y violations

Ta ke A c t i o n • Good Usability === Good Accessibility • Use Semantic HTML (also great for SEO) • Add alt and description tags to media and anchors • Te s t y o u r s i t e ( a x e

core.com ) • Tr y t h e
Funkity

Disability

Simulator Chrome Extension https:// chrome.google.com /webstore/detail/ funkify

disability

simula / ojcijjdchelkddboickefhnbdpeajdjg?hl

en • Be an ally

DNS

Ta ke A c t i o n

Parallization

Ta ke A c t i o n

CI/CD/CB with Jenkins

Ta ke A c t i o n

Real User Monitoring with Lux

Synthetic Real • Consistent • Precise CPU, Network use • Availability • Engagement • Business Metrics (Rev) • Market Segment • Layout • Design

Ta ke A c t i o n • Collect Real

User browser exerpiences • Audit user experience (load time, downloaded content) • Periodically run tests in the wild • Add window.addEventListener (‘error’) to capture console errors • Use Beacon API to ensure metrics are returned

Summary • Browser testing is easy! • Synthetic testing with: • We b Dr i ve r. i o for easy functional testing • We b Pa ge Te st . org to augment performance testing • Use We bPl at for mTe st for direct platform tests • Don’t just test workflows: Design, Performance, and Accessibility • Use Real

User feedback to compare tests in the wild

@ ColinBendell

Further Reading • http:// webdri ver. i o / api.html • https:// github.com / cloudinary / wdi o

allure

ts • https:// github.com / cloudinary / wdi o

allure

ts

example • https:// www.npmjs.com /package/ webdri ver

manager • http:// blog.kevinlamping.com /testing

your

login

an

in

depth

webdri veri o

tutorial/ • https:// www.browserstack.com /automate/ webdri veri o • https:// medi um. com /@ boriscoder /setting

up

selenium

tests

wi th

webdri ver

io

cc7fc3c86629 • https://w3c.github.io/ webdri ver / • https:// www.slideshare.net / LinkMeSrl / webdri veri o?qi d =8cdf045d

9868

4ca8

a8ad

8aefb5c22289&v=&b=& from_search =1 • https:// peter.sh /experiments/chromium

command

line

switches/#load

extension • https:// www.ghacks.net /2013/10/06/list

useful

google

chrome

command

line

switches/ • https:// deliciousbrains.com /how

were

automating

acceptance

testing/ • https:// medi um. com /tech

tajawal /page

object

model

pom

design

pattern

f9588630800b

Agenda Te s t M e t h o d o l o g i e s We bDr i ve r. i o Mocha tests How Browsers Work & Browser APIs We b Pl at for m Te st Mobile & Device Emulation Accessibility Operational: DNS, Paralyzing, CI Real User Testing 9:00 AM Welcome | 10:30 AM Break | 12:00 AM Lunch | 2:30 PM Break | 4:00 PM Wrap

up & Q&A

Break 10:30

10:45 AM 183

Lunch 11:50 AM – 12:30 AM 184 Lunch 12:00 PM – 1:00PM

Break 2:30 – 2:45 PM 185

186