Comparison
ContainmentvsMitigation
Containment
you cut the affected region out of the load balancer before anyone understood the bug.
Limiting the spread and impact of a failure before it is understood. It is the first job in an incident and it is regularly delayed by an instinct to diagnose first, which is the wrong order when impact is ongoing. The design side of it is what makes it possible at all: kill switches, flags, region isolation and rollback are the things you can reach for at 3am without knowing the cause.
Full entry →Mitigation
you failed over to the other region at 02:40 and went back to bed with the bug still there.
An action that reduces or removes the impact without fixing the underlying cause. It is the correct goal during an incident, and confusing it with a fix is how people stay up until dawn chasing a root cause with customers still affected. A mitigation deployed and then forgotten is its own risk, which is why the follow-up work is tracked separately and explicitly.
Full entry →