Comparison
Mean time to restorevsTime to detect
Mean time to restore
the outage started at 09:14 and traffic was healthy again at 09:31, and that number is the one that got tracked.
How long it takes to restore service after a failure in production. It matters more than the failure rate for most systems, because failures are inevitable and duration is the part you can engineer: rollback speed, observability, a rehearsed on-call. Optimising for it changes design decisions, since a system that can be restored in minutes can accept risks that one requiring a four-hour restore cannot.
Full entry →Time to detect
the errors started at 09:02 and the first alert fired at 09:41, and a customer had emailed at 09:12.
How long a failure lasted before anyone knew about it. It is the part of an incident timeline most likely to be embarrassing and most likely to be improvable, since it is a monitoring problem rather than a hard engineering one. A review that only examines what happened after the page is skipping the segment where the cheapest minutes are usually available.
Full entry →