Comparison
Graceful degradationvsLoad shedding
Graceful degradation
the provider went down and users got a cached answer or an honest error, because you decided that in design rather than at 3am.
A designed answer to 'the model is down or over budget': cached responses, a smaller model, a non-AI code path, or an honest error. Decided in design, not improvised mid-incident.
Full entry →Load shedding
you start rejecting requests immediately with a 503 instead of accepting work you have no chance of finishing in time.
Deliberately dropping a fraction of incoming work when the system is beyond capacity, so the remainder is served correctly. Accepting everything and timing out is worse for everyone: the client already gave up, and you spent the capacity anyway. Shed the cheapest-to-reject and least valuable traffic first, which requires knowing which traffic that is before the incident.
Full entry →