Applied AI·Local and self-hosted inference
your GPU sat mostly idle between single requests until you served many at once, and throughput went up several times over.
Batching / continuous batching
Also calledcontinuous batching
Draft summary, pending review
Serving many requests together to keep the GPU saturated. Continuous batching admits new requests into the batch as others finish, rather than waiting for the slowest, and is the single biggest throughput win in serving engines.