Tales after midnight

A presentation at WEBdeLDN in January 2019 in London, UK by Marco Cedaro

Slide 1

Slide 1

Slide 2

Slide 2

TalEs aftEr midnight for whom thE pagEr tolls

Slide 3

Slide 3

TalEs aftEr midnight for whom thE pagEr tolls An anthology of short cautionary horror storiEs

Slide 4

Slide 4

@cEdmax WEbmastEr bEforE it was cool TEch LEad Condé Nast IntErnational TalEs aftEr Midnight

Slide 5

Slide 5

TalEs aftEr midnight for whom thE pagEr tolls “ThErE arE 5 typEs of EnginEErs: ThosE that havE brokEn production and thosE who arE about to brEak production” Matt BoylE

Slide 6

Slide 6

ChaptEr #1 Don’t DEPLOY that CODE

Slide 7

Slide 7

Don’t DEPLOY that CODE

Slide 8

Slide 8

Don’t DEPLOY that CODE

Slide 9

Slide 9

Don’t DEPLOY that CODE

Slide 10

Slide 10

Don’t DEPLOY that CODE

Slide 11

Slide 11

Don’t DEPLOY that CODE

Slide 12

Slide 12

Don’t DEPLOY that CODE

Slide 13

Slide 13

Don’t DEPLOY that CODE

Slide 14

Slide 14

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL

  • thE procEss was pronE to Errors
  • our rollback stratEgy 🤓 What wE lEarnt
  • AUTOMATE ALL THE THINGS Don’t DEPLOY that CODE

Slide 15

Slide 15

ChaptEr #5 THE CSS OF DOOM

Slide 16

Slide 16

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

Slide 17

Slide 17

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

Slide 18

Slide 18

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

Slide 19

Slide 19

img { behavior: url(“pngbehavior.htc”); } THE CSS OF DOOM

Slide 20

Slide 20

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

Slide 21

Slide 21

function fixImage() { var src = element.src; if ( /.png$/.test( src.toLowerCase() ) ) { element.src = blankSrc; element.runtimeStyle.filter = “progid:DXImageTransform.Microsoft.AlphaImageLoader(src=’” + src + “’,sizingMethod=’scale’)”; } } THE CSS OF DOOM

Slide 22

Slide 22

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

Slide 23

Slide 23

THE CSS OF DOOM

Slide 24

Slide 24

THE CSS OF DOOM

Slide 25

Slide 25

☠ WHAT WENT WRONG

🧘 WHAT WENT WELL I don’t EvEn know whErE to start:

  • I havE a fun story to tEll
  • using thE prod db for monitoring - assuming thE CSS CAn’t BREAK anything
  • our rollback stratEgy
  • siloEd & BlamE culturE 🤓 What I lEarnt
  • You nEEd to undErstand thE systEm THE CSS OF DOOM

Slide 26

Slide 26

ChaptEr #3 POLAROID

Slide 27

Slide 27

Replica Replica Polaroid

Slide 28

Slide 28

Replica Replica Polaroid

Slide 29

Slide 29

Polaroid

Slide 30

Slide 30

Polaroid

Slide 31

Slide 31

Polaroid Run books didn’t havE anything on thE issuE ThE support Escalation paths wErE wrong WE FAILED TO COMMUNICATE THE FIX

Slide 32

Slide 32

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL

  • Monitoring wasn’t sEtup for thE rEplicas
  • wE CAUGHT THE PROBLEM BEFORE IT
  • ThE runbooks wErEn’t updatEd BECAME CRITICAL 🤓 What wE lEarnt
  • you can’t control EvErything - communication is KEy Polaroid

Slide 33

Slide 33

AppEndix POST MORTEM

Slide 34

Slide 34

THE DOCUMENT

Summary Post MortEm

Slide 35

Slide 35

THE DOCUMENT

Summary TimElinE Post MortEm

Slide 36

Slide 36

THE DOCUMENT

Summary TimElinE Impact Post MortEm

Slide 37

Slide 37

THE DOCUMENT

Summary TimElinE Impact Diagnosis Post MortEm

Slide 38

Slide 38

THE DOCUMENT

Summary TimElinE Impact Diagnosis Actions Post MortEm

Slide 39

Slide 39

SUGGESTIONS Post MortEm KEEp thE mEEting small KEEp it tEchnical Focus on thE whys? MakE surE thE actions arE rEalistic KEEp it blamElEss

Slide 40

Slide 40

TalEs aftEr midnight for whom thE pagEr tolls TakEaways

Sh*t will go wrong, BE prEparEd

KEEp your docs updatEd Try to undErstand your APP/STACK Don’t BLAME PEOPLE BUT PROCESSES