One Number, Multiple Metrics

A presentation at #PerfMatters in March 2020 in by Andrew Scheuermann

Slide 1

Slide 1

One Number, Multiple Metrics A Multidimensional Approach to Performance Andrew Scheuermann / @a15n_

Slide 2

Slide 2

Who Am I? • Management degree + bootcamp • Senior Engineer at Airbnb • Cook, read, board games

Slide 3

Slide 3

⚡

Slide 4

Slide 4

“ Load is not a single moment in time — it’s an experience that no one metric can fully capture. There are multiple moments during the load experience that can affect whether a user perceives it as ‘fast’ or ‘slow’. - developers.google.com

Slide 5

Slide 5

Slide 6

Slide 6

Time To Airbnb Interactive Reached here

Slide 7

Slide 7

Time To Airbnb Interactive Reached here

Slide 8

Slide 8

Time To Airbnb Interactive Reached here Ignored

Slide 9

Slide 9

Slide 10

Slide 10

Slide 11

Slide 11

~Lighthouse, with real user data

Slide 12

Slide 12

New System Requirements Real User Monitoring Measure Multiple Moments Comprehensible

Slide 13

Slide 13

Metric Requirements • User-centric – Is it happening? – Is it useful? – Is it usable? – Is it delightful? • Comprehensive browser coverage • Direct request + Client-side request

Slide 14

Slide 14

🖌 🖼 👇 🕐 Time To First Contentful Paint Time To First Meaningful Paint First Input Delay Total Blocking Time

Slide 15

Slide 15

Metrics in Action

Slide 16

Slide 16

Time To First Contentful Paint 🖌 Is it happening?

Slide 17

Slide 17

Time To First Meaningful Paint 🖼 Is it useful?

Slide 18

Slide 18

Time To First Meaningful Paint 🖼 Is it useful?

Slide 19

Slide 19

First Input Delay 👇 Is it usable?

Slide 20

Slide 20

Total Blocking Time 🕐 Is it delightful?

Slide 21

Slide 21

Multiple Metrics Total Blocking Time 🕐 Time To First Meaningful Paint 🖼 Time To First Contentful Paint 🖌 First Input Delay 👇

Slide 22

Slide 22

Slide 23

Slide 23

airbnb.com?pps=log

Slide 24

Slide 24

Slide 25

Slide 25

New System Requirements Real User Monitoring Measure Multiple Moments Comprehensible

Slide 26

Slide 26

One Number

Slide 27

Slide 27

“ That which is measured improves. That which is measured and reported improves exponentially. Karl Pearson

Slide 28

Slide 28

🖌 🖼 👇 🕐 Time To First Contentful Paint Time To First Meaningful Paint First Input Delay Total Blocking Time

Slide 29

Slide 29

events

5 seconds 3 seconds (67% progress) Fast 10 seconds (28% progress) Slow

Slide 30

Slide 30

Slide 31

Slide 31

New System Requirements Real User Monitoring Measure Multiple Moments Comprehensible

Slide 32

Slide 32

Page Performance Score

Slide 33

Slide 33

PPS Real user data 100% browser coverage Lighthouse 0-100 Score User-centric metrics Correlation Lab data Performance advice

Slide 34

Slide 34

Case Studies

Slide 35

Slide 35

Service Worker

Slide 36

Slide 36

• First time visitors download the service worker • Repeat visitors saw the App Shell • A/B tested

Slide 37

Slide 37

Seconds Default Service Worker 0 3.5 4.5 6.5 9 11

Slide 38

Slide 38

Time To First Contentful Paint 20% faster Time To First Meaningful Paint 67% slower Page Views 1.1% higher Bookings 0.6% higher

Slide 39

Slide 39

New Calendar

Slide 40

Slide 40

Old New

Slide 41

Slide 41

Slide 42

Slide 42

Google Map

Slide 43

Slide 43

Slide 44

Slide 44

Slide 45

Slide 45

Slide 46

Slide 46

Client-Side Route to Wishlists

Slide 47

Slide 47

Direct Request Client Request

Slide 48

Slide 48

Time To First Contentful Paint 20% faster Time To First Meaningful Paint 4% faster First Input Delay 27% faster Total Blocking Time 39% lower Page Views 6% higher

Slide 49

Slide 49

Learnings

Slide 50

Slide 50

Speed Seconds Good <3 Average 3-5 Slow

5

Slide 51

Slide 51

Speed Seconds Score Good <3

70 Average 3-5 60-69 Slow 5 <60

Slide 52

Slide 52

Slide 53

Slide 53

Slide 54

Slide 54

Meta Metrics • First Input Delay – Time To Initial Hydrate – Time To First Input • Total Blocking Time – Number of Blocking Tasks – Longest Blocking Task Duration – Longest Blocking Task Index – Page Duration

Slide 55

Slide 55

Slide 56

Slide 56

KEY TAKEAWAYS Load is not a single moment Measure multiple, user-centric metrics Real User Monitoring Combine into a single number

Slide 57

Slide 57

Resources Blog post: https://web.dev/user-centric-performance-metrics/ @a15n_ Polyfills (open source coming soon) Score Calculator: shorturl.at/lnwBN

Slide 58

Slide 58