Comparison
Manual interventionvsRunbook
Manual intervention
someone marked the failed task as successful at 03:40 so the rest of the pipeline could proceed, and did not write down why.
Overriding a pipeline's state by hand — clearing a task to re-run it, forcing a run past a failure, skipping a step. It is a necessary tool and an undocumented state change to a production system, and the two facts sit badly together. Treating it like a deploy — recorded, attributed, with a reason — is what stops next month's investigation ending at 'the data is odd for one day in March and nobody knows why'.
Full entry →Runbook
the alert links to a page that tells the person on call what to check and what to do, at four in the morning.
Written procedure for diagnosing and mitigating a specific alert. It converts institutional knowledge into something a tired non-expert can execute. Runbooks rot faster than code because nothing fails when they go stale, so linking them from the alert is what keeps them honest.
Full entry →