Comparison
LatencyvsThroughput
Latency
you measure the delay before the work starts moving, separately from how long the work itself takes.
Strictly, the delay before a transfer or operation begins — network propagation, queueing, scheduling. Casually it is used for total request duration, which is why the word causes so many crossed wires in performance conversations. Say which you mean, because latency and service time have completely different fixes.
Full entry →Throughput
you measure how many requests the system finishes per second, which is a different question from how long any one of them takes.
Completed work per unit time. It is the capacity number, and it trades against latency: batching, queueing and parallelism raise throughput while making individual requests slower. A system can have excellent throughput and unusable latency, which is why quoting one without the other means nothing.
Full entry →The difference
Latency is how fast one request completes; throughput is how many tokens or requests the system handles per second overall. Batch pipelines optimise throughput, chat optimises latency, and the two trade off against each other.
Entry: Latency vs throughput →