Tales after midnight

A presentation at WEBdeLDN in January 2019 in London, UK by Marco Cedaro

TalEs aftEr midnight for whom thE pagEr tolls

TalEs aftEr midnight for whom thE pagEr tolls An anthology of short cautionary horror storiEs

@cEdmax WEbmastEr bEforE it was cool TEch LEad Condé Nast IntErnational TalEs aftEr Midnight

TalEs aftEr midnight for whom thE pagEr tolls “ThErE arE 5 typEs of EnginEErs: ThosE that havE brokEn production and thosE who arE about to brEak production” Matt BoylE

ChaptEr #1 Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL
thE procEss was pronE to Errors
our rollback stratEgy 🤓 What wE lEarnt
AUTOMATE ALL THE THINGS Don’t DEPLOY that CODE

ChaptEr #5 THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

img { behavior: url(“pngbehavior.htc”); } THE CSS OF DOOM

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

function fixImage() { var src = element.src; if ( /.png$/.test( src.toLowerCase() ) ) { element.src = blankSrc; element.runtimeStyle.filter = “progid:DXImageTransform.Microsoft.AlphaImageLoader(src=’” + src + “’,sizingMethod=’scale’)”; } } THE CSS OF DOOM

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

THE CSS OF DOOM

THE CSS OF DOOM

☠ WHAT WENT WRONG
🧘 WHAT WENT WELL I don’t EvEn know whErE to start:
I havE a fun story to tEll
using thE prod db for monitoring - assuming thE CSS CAn’t BREAK anything
our rollback stratEgy
siloEd & BlamE culturE 🤓 What I lEarnt
You nEEd to undErstand thE systEm THE CSS OF DOOM

ChaptEr #3 POLAROID

Replica Replica Polaroid

Replica Replica Polaroid

Polaroid

Polaroid

Polaroid Run books didn’t havE anything on thE issuE ThE support Escalation paths wErE wrong WE FAILED TO COMMUNICATE THE FIX

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL
Monitoring wasn’t sEtup for thE rEplicas
wE CAUGHT THE PROBLEM BEFORE IT
ThE runbooks wErEn’t updatEd BECAME CRITICAL 🤓 What wE lEarnt
you can’t control EvErything - communication is KEy Polaroid

AppEndix POST MORTEM

THE DOCUMENT
Summary Post MortEm

THE DOCUMENT
Summary TimElinE Post MortEm

THE DOCUMENT
Summary TimElinE Impact Post MortEm

THE DOCUMENT
Summary TimElinE Impact Diagnosis Post MortEm

THE DOCUMENT
Summary TimElinE Impact Diagnosis Actions Post MortEm

SUGGESTIONS Post MortEm KEEp thE mEEting small KEEp it tEchnical Focus on thE whys? MakE surE thE actions arE rEalistic KEEp it blamElEss

TalEs aftEr midnight for whom thE pagEr tolls TakEaways
Sh*t will go wrong, BE prEparEd
KEEp your docs updatEd Try to undErstand your APP/STACK Don’t BLAME PEOPLE BUT PROCESSES

Marco Cedaro
@cedmax

1 / 40

Tales after midnight is an anthology of modern horror cautionary stories, including “Don’t deploy that code”, “The CSS of Doom”, “Polaroid” et al.

It’s really important to be ready, in case it was you the pager was tolling for, just after midnight.

“Based on true stories” – me

“Not your typical horror stories” – also me

“The tales are varied as they are interesting and it promises to entertain readers from all types of horror genres” – not me, but also not necessarily about this talk