Comparison
Null-rate checkvsVolume anomaly
Null-rate check
the column has been two percent null for a year, this morning it is sixty percent, and nothing else changed.
Monitoring the proportion of missing values in a column against its normal range rather than against zero. It catches upstream breakages with unusual precision, because a failed join or a renamed source field usually shows up first as a column that has gone empty. Testing for 'not null' misses this entirely, since the real signal is the change in the rate rather than the presence of any nulls at all.
Full entry →Volume anomaly
the load succeeded and inserted eleven rows, where every previous day inserted about four hundred thousand.
A row count far outside the range that period normally produces. It is the highest-value check per line of code in existence, because almost every ingestion failure — a filter matching nothing, a partial extract, a source outage — shows up here first. It needs day-of-week and seasonality awareness to be usable, otherwise every Sunday is an incident and the alert gets muted.
Full entry →