Comparison
Metric cardinality explosionvsTrace sampling
Metric cardinality explosion
you add user id as a label and your metrics bill and query latency both go up by three orders of magnitude.
The multiplicative growth in stored time series when a label has many distinct values. Every unique combination of label values is a separate series, so one unbounded label — user id, URL with ids in it, error message — can create millions. It is the standard way to take down a metrics backend from inside your own application.
Full entry →Trace sampling
you keep one percent of traces because storing all of them costs more than the service does.
Recording only a subset of traces to bound cost. Head sampling decides at the start and is cheap but blind; tail sampling decides after seeing the whole trace and can keep all the errors and slow ones, at the cost of buffering. Uniform head sampling is why the trace for the incident you care about is usually missing.
Full entry →