Comparison
Chaos engineeringvsRestore testing
Chaos engineering
you deliberately kill an instance in production during working hours to find out whether the failover actually works.
Injecting controlled failures into a running system to verify that its resilience mechanisms work before an incident tests them. The value is in disproving assumptions: the retry that was never exercised, the replica that was never promoted. Doing it without a hypothesis, a blast radius limit and a stop button is not chaos engineering, it is an outage.
Full entry →Restore testing
you restore the backup into a scratch environment on a schedule, because a backup nobody has restored is a hypothesis.
Regularly performing a real restore from backups and verifying the result, rather than trusting that the backup job exited zero. It routinely uncovers backups of the wrong database, encrypted backups whose key is gone, restores that take far longer than the objective, and permissions that only exist in the environment being replaced. It is the single highest-value reliability exercise available and the most commonly skipped.
Full entry →