Comparison
Metric definitionvsMetric drift
Metric definition
two dashboards show different revenue because one includes refunds and neither of them says so anywhere.
The precise, written statement of how a business measure is computed, including what is excluded. Most metric disagreements are not disputes about a number but discoveries that two people were computing different things, and the definition is what makes that discoverable in seconds. It has to live where it is executed rather than in a wiki, otherwise the wiki becomes the aspirational version.
Full entry →Metric drift
the same metric name means something slightly different than it did in March, and the year-on-year chart is comparing two definitions.
A measure's definition changing over time without the change being versioned or announced. It is uniquely damaging because it corrupts comparisons rather than values: every number is correct on the day it was computed and the trend between them is meaningless. Versioning definitions and recording when each took effect is the only durable fix, since restating history is often impossible.
Full entry →