Comparison
Session affinityvsStatelessness
Session affinity
you pin each user to the same instance so the state cached in that process is still there on their next request.
Routing a client's requests consistently to one backend. It makes in-process state usable, which is sometimes the only practical option for WebSockets or large session data. It also breaks even load distribution, prevents clean draining on deploy, and turns one instance's death into a visible failure for a subset of users.
Full entry →Statelessness
it is HTTP. Each request stands alone, exactly like a REST endpoint without sessions.
The API remembers nothing between calls. Any 'memory' a product shows is application code storing and resending history. This is the single most common misconception among engineers new to LLM APIs.
Full entry →