Production testing through monitoring

Slide 1

Troubleshooting with monitoring Testing in production DevOps monitoring [something] testing [something] monitoring [something] in production Leon Fayer @papa_fire

Slide 2

WHO AM I? ๏ engineer for 20+ years ๏ professional cynic ๏ @ OmniTI ๏ build and operate big systems ๏ we are hiring! ๏ omniti.com/is/hiring THAT’S ME ❖ ❖ ❖ ❖ @papa_fire leon@omniti.com fayerplay.com slideshare.net/LeonFayer1

Slide 3

I HATE TESTING @papa_fire

Slide 4

testing is required @papa_fire

Slide 5

testing is not enough @papa_fire

Slide 6

unit testing > functional testing > resilience testing > performance testing >… @papa_fire

Slide 7

testing can give a false sense of security @papa_fire

Slide 8

testing is deterministic @papa_fire

Slide 9

data problem @papa_fire

Slide 10

quantity of data > frequency of data > quality of data @papa_fire

Slide 11

example Wolfe+585 @papa_fire

Slide 12

example Hubert Blaine Wolfeschlegelsteinhausenbergerdorffwelchevoralternwarengewissenhaftschaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbe schutzenvorangreifendurchihrraubgierigfeindewelchevoralternzwolfhundert tausendjahresvorandieerscheinenvonderersteerdemenschderraumschiff genachtmittungsteinundsiebeniridiumelektrischmotorsgebrauchlichtalsseinur sprungvonkraftgestartseinlangefahrthinzwischensternartigraumaufdersuchen nachbarschaftdersternwelchegehabtbewohnbarplanetenkreisedrehensichundwo hinderneuerassevonverstandigmenschlichkeitkonntefortpflanzenundsicher freuenanlebenslanglichfreudeundruhemitnichteinfurchtvorangreifenvor andererintelligentgeschopfsvonhinzwischensternartigraum, Sr. @papa_fire

Slide 13

user problem @papa_fire

Slide 14

“ Users (n) - distributed fault injection test suite for production @papa_fire

Slide 15

example Corrupted Blood bug @papa_fire

Slide 16

example @papa_fire

Slide 17

other factors @papa_fire

Slide 18

lack of foresight (Y2K bug) > too many use-cases (female Tauren bug) > change to assumptions @papa_fire

Slide 19

testing is great for “known knowns” @papa_fire

Slide 20

testing is ok for “known unknowns” @papa_fire

Slide 21

testing is bad for “unknown unknowns” @papa_fire

Slide 22

enter monitoring @papa_fire

Slide 23

why monitor? @papa_fire

Slide 24

because testing isn’t enough @papa_fire

Slide 25

software is never perfect > systems are complex > external dependency worry > proactive is better than reactive >… @papa_fire

Slide 26

because things change @papa_fire

Slide 27

because things change in production @papa_fire

Slide 28

what to monitor? @papa_fire

Slide 29

“ in God we trust all others we monitor @papa_fire

Slide 30

systems > databases > applications > integration points > performance > user behavior >… @papa_fire

Slide 31

is it enough? @papa_fire

Slide 32

is it too much? @papa_fire

Slide 33

what is important? @papa_fire

Slide 34

what is important? (i.e. what to alert on) @papa_fire

Slide 35

example > servers up and running > HTTP checks return 200 > tweets are lost @papa_fire

Slide 36

s/system checks/unit tests/ @papa_fire

Slide 37

“ I don’t give a **** if the datacenter is on fire as long as I am still making money — CEO @papa_fire

Slide 38

we monitor because things change @papa_fire

Slide 39

changes effect business @papa_fire

Slide 40

top-down approach > understand business > define baseline > correlate data @papa_fire

Slide 41

example ๏ online marketing company ๏ major e-commerce component ๏ ~100 million users ๏ 1 billion emails/month ๏ 300,000 lines of code ๏5600+ metrics collected @papa_fire