Comparison
Batching / continuous batchingvsMicro-batch
Batching / continuous batching
your GPU sat mostly idle between single requests until you served many at once, and throughput went up several times over.
Serving many requests together to keep the GPU saturated. Continuous batching admits new requests into the batch as others finish, rather than waiting for the slowest, and is the single biggest throughput win in serving engines.
Full entry →Micro-batch
the 'streaming' job actually wakes every thirty seconds, reads whatever arrived, and writes one file.
Running batch machinery on a short interval so it behaves approximately like a stream. You keep batch's restart story and its file-sized writes, and you accept latency equal to the interval. It is the honest middle of the market and it is what a great many production 'real-time' systems are, which matters when someone quotes you a five-second freshness figure that is really an interval plus a run time.
Full entry →