Applied AI·The API layer
the batch job wants tokens per second across the fleet and the chat box wants this one answer now, and you cannot tune for both.
Latency vs throughput
Draft summary, pending review
Latency is how fast one request completes; throughput is how many tokens or requests the system handles per second overall. Batch pipelines optimise throughput, chat optimises latency, and the two trade off against each other.