Automating Browser Testing

A presentation at BuzzJS in June 2018 in New York, NY, USA by Colin Bendell

1 Browser Test Automation (in 2018) @ ColinBendell CTO Office, Cloudinary

Agenda Te s t M e t h o d o l o g i e s We bDr i ve r. i o Mocha tests How Browsers Work & Browser APIs We b Pl at for m Te st Mobile & Device Emulation Accessibility Operational: DNS, Paralyzing, CI Real User Testing 9:00 AM Welcome | 10:30 AM Break | 12:00 AM Lunch | 2:30 PM Break | 4:00 PM Wrap
up & Q&A

Cloudinary's Two Probl ems Felix Zilber New Digital
Asset
Management Front
End Automating CDN Changes

10 Te s t i n g : P r e p a r e f o r t h e U n e x p e c t e d

“I just want to buy my Mom’s birthday present.” “I just want to know if it will rain today.” “I just want to share pictures of my vacation”

Te s t i n g = = = M e e t i n g U s e r E x p e c t a t i o n s

Kinds of Testing https:// stackoverflow.com /questions/2741832/unit
tests
vs
functional
tests

Unit Testing • Smallest testable component • Usually simple inputs and outputs • Moch objects v. skeleton dev services

Unit Testing const expect
require( "chai" ).expect; const converter
require( "../app/converter" ); describe( "Color Code Converter" , function () { describe( "RGB to Hex conversion" , function () { it( "converts the basic colors" , function () { var redHex
converter.rgbToHex ( 255 , 0 , 0 ); var greenHex
converter.rgbToHex ( 0 , 255 , 0 ); var blueHex
converter.rgbToHex ( 0 , 0 , 255 ); expect( redHex ). to.equal ( "ff0000" ); expect( greenHex ). to.equal ( "00ff00" ); expect( blueHex ). to.equal ( "0000ff" ); }); }); });

Unit Testing • Blurred line to integration tests • Encourages small component logic v. monoliths • Large volume of Tests

Integration Tests describe( "Color Code Converter API" , function () { describe( "RGB to Hex conversion" , function () { let url
"http://localhost:3000/ rgbToHex?red =255&green=255&blue=255" ; it( "returns status 200" , function () {}); it( "returns the color in hex" , function () {}); }); describe( "Hex to RGB conversion" , function () { let url = "http://localhost:3000/ hexToRgb?hex =00ff00" ; it( "returns status 200" , function () {}); it( "returns the color in RGB" , function () {}); }); });

System Tests • Does it match design? • Meet expectations of design • Often focused on workflows • Vol ume, l oad, stress, securi ty • Usability, Accessibility

Acceptance Tests • Meet business objectives • End
to
End • Can the end user accomplish what we wanted it to do? • Alpha/Beta Feedback cycle

“We develop in the Zoo, But release in the Jungle”
Melanie Cey

Why do we need browser testing • Functional User experience • Wo r k f l o ws • Design & Layout • Preserve Presentation • Above the fold, responsive web • Mobi l e experi ences v. Watch v. ?? • Accessibility • Screen readers • audits • Performance measurement • Performance budgets • Request flows

2007 2008 2009 2010 Present IE7 iPhone Firefox 2 IE7 iPhone 3G Firefox 3 Android Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad ??

2012: 3,977 Different Android Devices Source: OpenSi gnal 2012: 3,977 Different Android Devices

Source: OpenSignal 2014: 18.7k Different Android Devices

Source: OpenSignal 2015: 24.0k Different Android Devices

How many browsers in a typical Thanksgiving weekend?

How many browsers in a typical Thanksgiving weekend? 572

“…25% of new Android phones have only 512MB of RAM. ”
Jen Fitzpatrick VP of product management for Google Maps

How do we test the Browser? • Checklists • Support tickets • Screen capture • Pixel comparison • Programmatic control • RUM Beacons

#1 reason checklists fail to catch browser regression: Humans

open
a Safari https:// cloudinary.com killall Safari screencapture
x testoutput.png compare
identify
metric MAE baseline.png testoutput.png null
same1.png[0] PNG 640x400 640x400+0+0 8
bit DirectClass 1.64KB 0.010u 0:00.019
same2.jpg[0] JPEG 640x400 640x400+0+0 8
bit DirectClass 3.65KB 0.000u 0:00.009
0.196766 (3.00245e

OS API application control • Access the window model directly • Simulate Keyboard strokes and mouse movements • Interact with any application like a human • Screen Readers use this approach to supersede the OS accessibility features

Selenium Evolution • Selenium • Selenium Core • Selenium RC Selenium 1 • Selenium WebDriver Selenium 2 • We b D r i v e r Selenium 3 Gecko Driver (+Marionette) Chrome Driver EdgeDriver Apple Safari Driver Firefox Driver Safari Driver … https:// www.seleniumhq.org /download/

Selenium 3 (W3C WebDriver) Selenium Bindings Java Ruby C# Python JavaScript ? Selenium Grid ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera JSON Wire Protocol WebDriver Wire Protocol HTTP HTTP

We bDr i ve r. i o We b Dr i ve r (or Selenium Grid) ChromeDriver GeckoDriver SafariDriver EdgeDriver OperaDriver Browsers Chrome Firefox Safari Edge Opera WebDriver Wire Protocol HTTP HTTP NodeJS

curl – X POST http://localhost:9515/session
d '{" desiredCapabilities ":{" browserName ":"chrome"}} ' { "sessionId" : "your
session
id
here" , "status" : 0 , "value" :{...}} curl http: / /localhost:9515/session/$SESSION_ID/ url \
d '{" url ":"http:// www.cnn.com /"} ' { " sessionId " : "..." , "status" : 0 , "value" : null } curl http: / /localhost:9515/session/$SESSION_ID/element
d '{"using":"tagName","value":"h1"}' { " sessionId " : "..." , "status" : 0 , "value" :{ " ELEMENT" : "element
object
id
here " }} curl http: / /localhost:9515/session/$SESSION_ID/element/$OBJECT_ID/text { " sessionId " : "..." , "status" : 0 , "value" : "Example Domain" } curl – X DELETE http: //localhost:9515/session/your
session
id
here { " sessionId " : "..." , "status" : 0 , "value" : null }

To o l c h a i n o v e r v i e w We bDr i ve r. i o Jenkins We b Platform Te s t Allure Chrome Driver Mocha Chrome Axe

Exercise 1: Getting Setup https:// github.com / colinbendell / webdriver
workshop Request https:// duckduckgo.com Enter search term ( eg : "Mr. Dressup ") Navigate to the 4th result and report the
<title>

git checkout https:// github.com / colinbendell / webdriver
workshop

const webdriverio
require ( ' webdriverio ' );

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; const webdriverio
require ( ' webdriverio ' );

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio
require ( ' webdriverio ' );

npm install allure allure
commandline wdio
allure
reporter

========================= WDIO Configuration Helper
? Where do you want to execute your tests? On my local machine ? Which framework do you want to use? mocha ? Shall I install the framework adapter for you? Yes ? Where are your test specs located? ./ src /**/*_ spec.js ? Which reporter do you want to use? dot, allure ? Shall I install the reporter library for you? Yes ? Do you want to add a service to your test setup? Chromedriver ? Shall I install the services for you? Yes ? Level of logging verbosity error ? In which directory should screenshots gets saved if a command fails? ./ errorShots / ? What is the base url ? http://localhost

Exercise 2: Wo r k f l o w Te s t
Adding TODO spec Install mocha, chai, allure create wdio.conf.js Navigate to todomvc.com Using Angular implementation: • add a To D o • marking it complete • clear the list

Exercise 3: POM spec Repeat Exercise 2 Use Page Object Model Simplify Allure output with wdio
allure
ts

Ta ke A c t i o n • Create workflows easily with mocha • Use a Page
Object
Model to abstract selectors for reduced maintenance • Create good & actionable outputs

2 Second Magic Number Page Load Time: Seconds

! " ! 50ms DNS Lookup 100ms ShoesByColin.com 8.8.8.8

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms ShoesByColin.com 8.8.8.8

! " ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms ShoesByColin.com 8.8.8.8

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms
?? ms ShoesByColin.com 8.8.8.8

! " ! ! 50ms DNS Lookup 100ms TCP Connect +100ms TLS Negotiation +200ms Application +100ms
?? ms ~ 500ms Network
500ms? App

HTML DOM JavaScript CSS Render Tr e e Layout Paint CSSOM Images Video JS Exec

Real User Monitoring: JavaScript Timers Steve Souders Philip Te l l i s https:// github.com / stevesouders /episodes https:// github.com /lognormal/boomerang

https://www.w3.org/wiki/ We b_ Pe r fo r manc e /Publications

{ " timeOrigin ": 1528900579651.6672, "timing": { " navigationStart ": 1528900579651, " unloadEventStart ": 0, " unloadEventEnd ": 0, " redirectStart ": 0, " redirectEnd ": 0, " fetchStart ": 1528900583010, " domainLookupStart ": 1528900579730, " domainLookupEnd ": 1528900579730, " connectStart ": 1528900579730, " connectEnd ": 1528900579761, " secureConnectionStart ": 1528900579733, " requestStart ": 1528900579761, " responseStart ": 1528900582922, " responseEnd ": 1528900583010, " domLoading ": 1528900583014, " domInteractive ": 1528900583849, " domContentLoadedEventStart ": 1528900583849, " domContentLoadedEventEnd ": 1528900583876, " domComplete ": 1528900585616, " loadEventStart ": 1528900585617, " loadEventEnd ": 1528900585856 }, "navigation": {"type": 0, " redirectCount ": 0 } }

function getPageLoadTime () { let timing = window.performance.timing ; return timing.loadEventStart
timing.navigationStart }

What about … Javascript | CSS | WebFonts | Images | Videos | XHR/Fetch

{ "name": "https:// ajax.googleapis.com /ajax/libs/ jquery /1.11.2/ jquery.min.js ", " entryType ": "resource", " startTime ": 928.70000000039, "duration": 425.8000000008906, " initiatorType ": "script", " nextHopProtocol ": "h2", " workerStart ": 0, " redirectStart ": 0, " redirectEnd ": 0, " fetchStart ": 928.70000000039, " domainLookupStart ": 937.4000000025262, " domainLookupEnd ": 945.9000000024389, " connectStart ": 945.9000000024389, " connectEnd ": 967.5999999999476, " secureConnectionStart ": 946.2000000021362, " requestStart ": 949.5999999999185, " responseStart ": 970.9000000002561, " responseEnd ": 1354.5000000012806, " transferSize ": 33798, " encodedBodySize ": 33495, " decodedBodySize ": 95931, " serverTiming ": [] }

let url = “https:// bendell.ca / logo.png ”; let me = performance.getEntriesByName ( url )[0]; let timings = { loadTime : me. duration , dns : me. domainLookupEnd
me. domainLookupStart , tcp : me. connectEnd
me. connectStart , waiting: me. responseStart
me. requestStart , fetch: me. responseEnd
me. responseStart }

200 OK HTTP/1.1 Timing
Allow
Origin: * Content
Type: application/ javascript Content
Length: 1234

• Slowest resources • Time to first image • Response time by domain • Time a group of assets • Response time by initiator type (element type) • Browser cache
hit ratio for resources • Resource Timing API • Other Uses More: http://www.slideshare.net/bluesmoon/beyond
page
level
metrics

How long does it take to display the main product image on my site?

function loadTemplate () { performance.mark (“ startTask ”); doTemplateStuff () //do stuff (assumes promise()) .then(() => { performance.mark (“ stopTask ”); //Measure the duration between the two marks performance.measure (“ taskDuration ”, “ startTask ”,“ stopTask ”); }); }

< img src ="image1.gif" onload=" performance.mark (' hero_image ')"> https:// stevesouders.com /blog/2015/05/12/hero
image
custom
metrics/

GET /resource HTTP/1.1 Host: example.com HTTP/1.1 200 OK Server
Timing: cdn , dur="130" Server
Timing: cloudinary , dur="10", desc ="new trans"

{ let url = 'https:// example.com / resource.jpg ’; let severEntries [] = window.performance.getEntriesByName ( url ) . filter (( name ) => name === " server ") ; }

let config = { attributes: true, childList : true, subtree: true }; let targetNode
document.querySelector ('body'); let callback = function(mutations) { mutations.forEach (function(mutation) { if ( mutation.type === ' childList ') { let values = []. slice.call ( list.children ) .map(node => node.nodeName ) .filter(n => / img / i.test (n)); console.log ( list_values ); } }); }; let observer = new MutationObserver (callback); observer.observe ( targetNode , config);

Exercise 4: performance budget Navigate to amazon.com Calculate the total bytes downloaded Ensure that the page load time < 2s and bytes < 2MiB Repeat with Disney.com

Ta ke A c t i o n • Add performance.mark () to measure YOUR key metrics • Eg : Hero Image, Product Image, Sport Scores • Keep performance.mark () in production code! • Define Hard and Soft targets that synthetic tests should hit • BHAG: 2s & 1MiB

Mobile Display & Network Throttling Tim Kadlec

chromeOptions : { args :[ "
incognito" , "
headless", "
disable
gpu " ] },

chromeOptions : { args :[ "
incognito" , "
disable
gpu " ], mobileEmulation : { deviceMetrics : { width : 360, height : 720, pixelRatio : 3}, networkThrottle : 'Fast 3G' } }

Ta ke A c t i o n • Run Tests in different viewport configurations to test RWD • Experiment with different Network Conditions • Similarly with different CPU throttling

https:// darobin.github.io /test
harness
tutorial/docs/using
testharness.html

Why Inside v. Outside Browser Testing? • Hawthorne Effect • Platform Widgets & Controls • APIs

Exercise 5: WPT Implement Responsive Images and Test the naturalWidth

A11y with AXE Léonie Watson Marcy Sutton Katie Sylor
Miller

“Websites, tools, and technologies are designed and developed so that people with disabilities can use them .” ACCESSIBILITY (a11y )
W3C Web Accessibility Initiative

THE AMERICANS WITH DISABILITIES ACT (ADA) “No individual shall be discriminated against on the basis of disability in the full and equal enjoyment of the goods, services, facilities, privileges, advantages, or accommodations of any place of public accommodation ”
ADA Sec. 12182 (a)

AMERICANS WITH A DISABILITY UNITED STATES CENSUS REPORT, JULY 2012 57 M 19%

The Basics • We b Conte nt Ac c e s s i bi l i t y Gui de l i ne s ( WCAG 2. 0)
• Developed by the W3C Web Accessibility Three levels of compliance
A, AA, AAA

Exercise 6: A11y Add axe
core npm Load 3 news websites Compare the violations Create a test case to fail on a11y violations

Ta ke A c t i o n • Good Usability === Good Accessibility • Use Semantic HTML (also great for SEO) • Add `alt` and `description` tags to media and anchors • Te s t y o u r s i t e ( a x e
core.com ) • Tr y t h e
Funkity
Disability
Simulator Chrome Extension https:// chrome.google.com /webstore/detail/ funkify
disability
simula / ojcijjdchelkddboickefhnbdpeajdjg?hl
en • Be an ally

Synthetic Real • Consistent • Precise CPU, Network use • Availability • Engagement • Business Metrics (Rev) • Market Segment • Layout • Design

Ta ke A c t i o n • Collect Real
User browser exerpiences • Audit user experience (load time, downloaded content) • Periodically run tests in the wild • Add window.addEventListener (‘error’) to capture console errors • Use Beacon API to ensure metrics are returned

Summary • Browser testing is easy! • Synthetic testing with: • We b Dr i ve r. i o for easy functional testing • We b Pa ge Te st . org to augment performance testing • Use We bPl at for mTe st for direct platform tests • Don’t just test workflows: Design, Performance, and Accessibility • Use Real
User feedback to compare tests in the wild

Further Reading • http:// webdri ver. i o / api.html • https:// github.com / cloudinary / wdi o
allure
ts • https:// github.com / cloudinary / wdi o
allure
ts
example • https:// www.npmjs.com /package/ webdri ver
manager • http:// blog.kevinlamping.com /testing
your
login
an
in
depth
webdri veri o
tutorial/ • https:// www.browserstack.com /automate/ webdri veri o • https:// medi um. com /@ boriscoder /setting
up
selenium
tests
wi th
webdri ver
io
cc7fc3c86629 • https://w3c.github.io/ webdri ver / • https:// www.slideshare.net / LinkMeSrl / webdri veri o?qi d =8cdf045d
9868
4ca8
a8ad
8aefb5c22289&v=&b=& from_search =1 • https:// peter.sh /experiments/chromium
command
line
switches/#load
extension • https:// www.ghacks.net /2013/10/06/list
useful
google
chrome
command
line
switches/ • https:// deliciousbrains.com /how
were
automating
acceptance
testing/ • https:// medi um. com /tech
tajawal /page
object
model
pom
design
pattern
f9588630800b

Lunch 11:50 AM – 12:30 AM 184 Lunch 12:00 PM – 1:00PM

Colin Bendell
@colinbendell

1 / 186

In this training we will put all the pieces you need to adopt browser based testing without the complexity. For you: build more resilient front end apps. For your boss: lower the barrier for test writing and get "support" to write browser tests as bug reports.

Client side testing is mostly a known science, but it is not always easy, boring to build and often very fragile. The challenge is to lower the barrier-to-entry so that anyone can create tests, scale and resilience. We will cover the basics of browser based testing, webdriver standards and infrastructure advances. Different testing paradigms for client side testing with BDD models like mocha to WebPlatformTests from the W3C. Finally the CI automation and scaling with CI pipelines.

Automating Browser Testing

Agenda Te s t M e t h o d o l o g i e s We bDr i ve r. i o Mocha tests How Browsers Work & Browser APIs We b Pl at for m Te st Mobile & Device Emulation Accessibility Operational: DNS, Paralyzing, CI Real User Testing 9:00 AM Welcome | 10:30 AM Break | 12:00 AM Lunch | 2:30 PM Break | 4:00 PM Wrap

Cloudinary's Two Probl ems Felix Zilber New Digital

Asset

Management Front

Kinds of Testing https:// stackoverflow.com /questions/2741832/unit

tests

vs

functional

Functional v. Non

Unit Testing const expect

require( "chai" ).expect; const converter

require( "../app/converter" ); describe( "Color Code Converter" , function () { describe( "RGB to Hex conversion" , function () { it( "converts the basic colors" , function () { var redHex

converter.rgbToHex ( 255 , 0 , 0 ); var greenHex

converter.rgbToHex ( 0 , 255 , 0 ); var blueHex

Integration Tests describe( "Color Code Converter API" , function () { describe( "RGB to Hex conversion" , function () { let url

Acceptance Tests • Meet business objectives • End

to

“We develop in the Zoo, But release in the Jungle”

Kinds of Testing https:// stackoverflow.com /questions/2741832/unit

tests

vs

functional

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

open

a Safari https:// cloudinary.com killall Safari screencapture

x testoutput.png compare

identify

curl – X POST http://localhost:9515/session

d '{" desiredCapabilities ":{" browserName ":"chrome"}} ' { "sessionId" : "your

session

id

here" , "status" : 0 , "value" :{...}} curl http: / /localhost:9515/session/$SESSION_ID/ url \

d '{" url ":"http:// www.cnn.com /"} ' { " sessionId " : "..." , "status" : 0 , "value" : null } curl http: / /localhost:9515/session/$SESSION_ID/element

d '{"using":"tagName","value":"h1"}' { " sessionId " : "..." , "status" : 0 , "value" :{ " ELEMENT" : "element

object

id

here " }} curl http: / /localhost:9515/session/$SESSION_ID/element/$OBJECT_ID/text { " sessionId " : "..." , "status" : 0 , "value" : "Example Domain" } curl – X DELETE http: //localhost:9515/session/your

session

id

Exercise 1: Getting Setup https:// github.com / colinbendell / webdriver

git checkout https:// github.com / colinbendell / webdriver

const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = { host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

Exercise 1: Getting Setup https:// github.com / colinbendell / webdriver

npm install chai wdio

mocha

npm install wdio

chromedriver

npm install allure allure

commandline wdio

allure

========================= WDIO Configuration Helper

Exercise 2: Wo r k f l o w Te s t

Exercise 3: POM spec Repeat Exercise 2 Use Page Object Model Simplify Allure output with wdio

allure

Ta ke A c t i o n • Create workflows easily with mocha • Use a Page

Object

function getPageLoadTime () { let timing = window.performance.timing ; return timing.loadEventStart

let url = “https:// bendell.ca / logo.png ”; let me = performance.getEntriesByName ( url )[0]; let timings = { loadTime : me. duration , dns : me. domainLookupEnd

me. domainLookupStart , tcp : me. connectEnd

me. connectStart , waiting: me. responseStart

me. requestStart , fetch: me. responseEnd

200 OK HTTP/1.1 Timing

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; const webdriverio

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

let options = {
host : "localhost" , port : 9515, path : "/" , desiredCapabilities : { browserName : 'chrome' }; let browser = webdriverio. remote (options); const webdriverio

The Basics • We b Conte nt Ac c e s s i bi l i t y Gui de l i ne s ( WCAG 2. 0)
• Developed by the W3C Web Accessibility Three levels of compliance

Ta ke A c t i o n • Good Usability === Good Accessibility • Use Semantic HTML (also great for SEO) • Add `alt` and `description` tags to media and anchors • Te s t y o u r s i t e ( a x e

core.com ) • Tr y t h e
Funkity