Comparison
Straggler taskvsTail latency
Straggler task
the stage is at nine hundred and ninety-nine of a thousand tasks complete and has been for six minutes.
A single task holding up a stage while everything else has finished. The cause is either skew, where the task genuinely has more data, or an unhealthy node, where it does not. Distinguishing the two is the first diagnostic step, because the remedies are unrelated: one is a repartition or a salted key, the other is speculative execution or a dead machine.
Full entry →Tail latency
the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.
The slow end of the latency distribution, where garbage collection, cache misses, retries and contention live. It matters disproportionately because a request that fans out to N services experiences the tail with probability roughly N times higher. Averages hide it completely, which is why percentiles are the only useful summary.
Full entry →