Comparison
Confirmation biasvsNull hypothesis of an outage
Confirmation bias
you believed it was the cache and read the logs looking for the cache, and found something.
Seeking and weighting evidence that supports the hypothesis you already hold. It is at its most costly during debugging and incidents, where a plausible early theory determines which logs get read and which get skipped. The practical defence is to state what would disprove the theory before looking, and to hand the theory to someone who has not spent an hour in it.
Full entry →Null hypothesis of an outage
the graph turned at 14:07 and someone asked what shipped at 14:05.
The working assumption at the start of an incident that the cause is the most recent change. It is right often enough to be the first thing checked and the first thing ruled out, and it is what makes a deploy timeline the most valuable dashboard in an incident. The failure mode is stopping there: once the recent change is excluded, people tend to reach for exotic explanations before checking the second-most-recent one.
Full entry →