Comparison
Correlation is not causationvsNull hypothesis of an outage
Correlation is not causation
latency and CPU rose together and you spent two hours on the CPU, which was a symptom of the same queue.
Two things moving together does not establish that one moved the other. In incidents this is the most expensive reasoning error available, because dashboards are full of correlated metrics and the compelling one is often downstream of the real cause. The way out is a mechanism: say out loud how A would cause B, and if you cannot, treat the correlation as a clue rather than as a conclusion.
Full entry →Null hypothesis of an outage
the graph turned at 14:07 and someone asked what shipped at 14:05.
The working assumption at the start of an incident that the cause is the most recent change. It is right often enough to be the first thing checked and the first thing ruled out, and it is what makes a deploy timeline the most valuable dashboard in an incident. The failure mode is stopping there: once the recent change is excluded, people tend to reach for exotic explanations before checking the second-most-recent one.
Full entry →