Comparison
Cluster DNSvsTail latency
Cluster DNS
you connect to a name rather than an address, and every workload in the cluster resolves it to whatever is currently healthy.
The in-cluster resolver that maps service names to virtual addresses, making service-to-service calls a hostname rather than a discovery problem. It is a hard dependency of nearly every request path, so a saturated or crash-looping DNS deployment presents as intermittent failures everywhere at once rather than as a DNS incident. Aggressive search-domain expansion also means one lookup can become five queries, which is why DNS is a surprisingly common source of tail latency.
Full entry →Tail latency
the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.
The slow end of the latency distribution, where garbage collection, cache misses, retries and contention live. It matters disproportionately because a request that fans out to N services experiences the tail with probability roughly N times higher. Averages hide it completely, which is why percentiles are the only useful summary.
Full entry →