Comparison
Fault injectionvsStatic stability
Fault injection
you deliberately add two hundred milliseconds to one dependency in production and watch what the rest of the system does.
Introducing controlled failures — latency, errors, dropped packets, killed instances — to observe real behaviour rather than assumed behaviour. It is the mechanism a chaos exercise is built from, and it is most valuable in the small: injecting latency into one dependency reveals missing timeouts far more often than killing instances reveals anything. It requires a way to stop, which is why the blast-radius controls matter more than the experiment.
Full entry →Static stability
the control plane was down for an hour and traffic kept flowing, because nothing in the request path needed to ask it anything.
Designing so a system keeps working from its existing state when its dependencies for making changes are unavailable, rather than needing a successful call to stay healthy. The classic application is capacity: pre-provision for the failed state so a zone failure requires no scaling event, since the scaling API is likely to be struggling at exactly that moment. It costs money in idle capacity and buys you not depending on anything working during the worst hour of the year.
Full entry →