Comparison
On-call rotationvsYou build it, you run it
On-call rotation
one person is responsible for answering at 3am this week, and next week it is you.
The schedule that says who responds to production alerts outside working hours. Beyond its obvious purpose it is the main feedback loop between operations and design, and its health is measurable: pages per shift, pages that were actionable, sleep lost. A rotation smaller than about six people is a burnout schedule, and one where the same person is always called is an ownership problem wearing a rota.
Full entry →You build it, you run it
the team that wrote the code carried the pager for it, and the retry logic got better within a month.
The team that builds a system also operates it. Its power is as a forcing function rather than as an efficiency: feedback from production reaches the people who can change the design, so unloggable failures and unactionable alerts get fixed instead of being endured by somebody else. It is expensive in a way that must be funded — on-call time is real time — and it fails when teams are given the pager without the authority to change what pages them.
Full entry →