Applied AI·Production patterns and cost
the average looked fine and one user in twenty was waiting eleven seconds, because output length varies enormously.
Percentile latency (p50, p95, p99)
Draft summary, pending review
Latency distributions, not averages: p95 is what a twentieth of your users feel. LLM latency is high-variance (output length varies), so percentiles and timeouts matter more here than in typical services.