Comparison
Percentile latency (p50, p95, p99)vsTail latency
Percentile latency (p50, p95, p99)
the average looked fine and one user in twenty was waiting eleven seconds, because output length varies enormously.
Latency distributions, not averages: p95 is what a twentieth of your users feel. LLM latency is high-variance (output length varies), so percentiles and timeouts matter more here than in typical services.
Full entry →Tail latency
the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.
The slow end of the latency distribution, where garbage collection, cache misses, retries and contention live. It matters disproportionately because a request that fans out to N services experiences the tail with probability roughly N times higher. Averages hide it completely, which is why percentiles are the only useful summary.
Full entry →