Comparison
Response timevsTail latency
Response time
you measure what the user actually experiences: request sent to last byte received, queueing and all.
The total elapsed time a client observes, comprising network latency, queueing delay and service time. It is the number that matters to users and the one to put in an SLO. Under load it is dominated by queueing rather than processing, which is why optimising the handler does nothing once the system is saturated.
Full entry →Tail latency
the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.
The slow end of the latency distribution, where garbage collection, cache misses, retries and contention live. It matters disproportionately because a request that fans out to N services experiences the tail with probability roughly N times higher. Averages hide it completely, which is why percentiles are the only useful summary.
Full entry →