Comparison
CPU throttlingvsTail latency
CPU throttling
average CPU use is twelve percent and the p99 latency is terrible, because the container is being stopped for milliseconds at a time.
The kernel pausing a container that has used its CPU quota within the current scheduling period. Because the quota is enforced per short window rather than on average, a workload that is bursty — which most request handlers are — gets throttled while looking idle in any per-minute graph. It is the most under-diagnosed source of tail latency on container platforms, and the metric for it is not CPU usage but throttled time.
Full entry →Tail latency
the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.
The slow end of the latency distribution, where garbage collection, cache misses, retries and contention live. It matters disproportionately because a request that fans out to N services experiences the tail with probability roughly N times higher. Averages hide it completely, which is why percentiles are the only useful summary.
Full entry →