Comparison
BulkheadvsContainment
Bulkhead
you give each downstream its own pool of connections so one slow dependency cannot consume every thread you have.
Partitioning resources so that exhaustion in one compartment cannot sink the whole service. In practice: separate connection pools, thread pools or instances per dependency or per tenant. It converts a total outage into a partial one, which is the entire goal of resilience engineering.
Full entry →Containment
you cut the affected region out of the load balancer before anyone understood the bug.
Limiting the spread and impact of a failure before it is understood. It is the first job in an incident and it is regularly delayed by an instinct to diagnose first, which is the wrong order when impact is ongoing. The design side of it is what makes it possible at all: kill switches, flags, region isolation and rollback are the things you can reach for at 3am without knowing the cause.
Full entry →