Architecture & practice·topic 9 of 10

Incidents and what happens after

The vocabulary of a bad afternoon and the review that follows it. Most of it exists to keep two things apart: stopping the bleeding and understanding the cause, which are different jobs done by different people at different times.

Read in order · tick what you already know

01
you declared it before you knew how bad it was, because declaring is cheap and being late is not.
Incident
02
checkout was returning errors for everyone for eleven minutes, and that is the part customers will remember.
Outage
03
you had to pick a number in the first two minutes that decided who got woken up.
Severity level
04
one person stopped debugging, started coordinating, and the channel got quieter and faster.
Incident commander
05
you cut the affected region out of the load balancer before anyone understood the bug.
Containment
06
you failed over to the other region at 02:40 and went back to bed with the bug still there.
Mitigation
07
the actual fix shipped nine days after the incident, once someone had understood the race.
Remediation
08
the errors started at 09:02 and the first alert fired at 09:41, and a customer had emailed at 09:12.
Time to detect
09
you sat down a week later with the timeline and worked out why the wrong dashboard was the obvious one to check.
Incident review
10
the engineer who ran the command explained exactly what they saw, because nothing bad was going to happen to them for it.
Just culture
11
the review produced fourteen actions and by the next quarter two of them were done.
Action item
12
you asked why five times and ended up at a deploy process nobody had questioned in two years.
Five whys
13
the report named one cause, and three of the people who were there could each name a different one.
Root cause
14
the deploy triggered it, and the missing alert, the stale runbook and the one-person rotation all had to be true as well.
Contributing factor
15
there were four safeguards and the change slipped through a hole in each of them on the same afternoon.
Swiss cheese model
16
the backup had been failing silently for five months and you found out when you needed it.
Latent failure
17
you broke the database on purpose during working hours to find out whether the failover worked.
Game day
18
the deploy has warned about that for months and everyone knows to ignore it.
Normalisation of deviance
19
the budget was spent in the first week of the quarter, and feature work stopped until reliability caught up.
Error budget policy