Comparison
Fault injectionvsGame day
Fault injection
you deliberately add two hundred milliseconds to one dependency in production and watch what the rest of the system does.
Introducing controlled failures — latency, errors, dropped packets, killed instances — to observe real behaviour rather than assumed behaviour. It is the mechanism a chaos exercise is built from, and it is most valuable in the small: injecting latency into one dependency reveals missing timeouts far more often than killing instances reveals anything. It requires a way to stop, which is why the blast-radius controls matter more than the experiment.
Full entry →Game day
you broke the database on purpose during working hours to find out whether the failover worked.
A scheduled exercise where a failure is deliberately induced to test the system and the people around it. Its main product is usually not the technical finding but the procedural one: the runbook was wrong, the alert went to a former employee, nobody knew who could authorise the failover. Running one is the only reliable way to find latent failures before they find you.
Full entry →