Comparison
Game dayvsLatent failure
Game day
you broke the database on purpose during working hours to find out whether the failover worked.
A scheduled exercise where a failure is deliberately induced to test the system and the people around it. Its main product is usually not the technical finding but the procedural one: the runbook was wrong, the alert went to a former employee, nobody knew who could authorise the failover. Running one is the only reliable way to find latent failures before they find you.
Full entry →Latent failure
the backup had been failing silently for five months and you found out when you needed it.
A fault already present in the system that has not yet produced a visible failure. Untested backups, misconfigured alerts, an autoscaler that would not actually scale, a certificate nobody renewed. They accumulate quietly and are only found by deliberately looking — which is what game days, restore drills and failover exercises are for.
Full entry →