Troubleshooting en production : quand chaque incident devient une enquête !

A presentation at France DevOps in March 2025 in by Seb Ferrer

Slide 1

Slide 2

Slide 3

git commit git push

Slide 4

❝ ❞ ❝ to join the night/weekend on-call rotation 2 months later ❞

Slide 5

Slide 6

Slide 7

Slide 8

Project A Project B Project C Project D Project E Project F

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

[project-a] creating user: ErrValidator: POST /rest/my-api: 403 … … …

Slide 14

Last 24 hours 28 days ago timestamp message 2024-02-02 08:34:12.548951 +02:00 creating user: ErrValidator: … 2024-02-02 08:34:01.548951 +02:00 creating user: ErrValidator: … 2024-02-02 08:33:36.548951 +02:00 creating user: ErrValidator: … 2024-02-02 08:33:24.548951 +02:00 creating user: ErrValidator: … 2024-02-02 08:33:04.548951 +02:00 creating user: ErrValidator: … 2024-02-02 08:32:44.548951 +02:00 creating user: ErrValidator: …

Slide 15

Slide 16

Root Cause Analysis (RCA)

Slide 17

What problem am I trying to solve?

Slide 18

Load balancer

Slide 19

Load balancer

Slide 20

Load balancer

Slide 21

Load balancer

Slide 22

Slide 23

Slide 24

creating user: creating user: creating user:

Slide 25

creating user: creating user: creating user:

Slide 26

Slide 27

Slide 28

Users can no longer connect to the application The authentication system returns an error 500 The authentication service is unable to query the user database The authn service cannot establish a connection to the database