Comparison
Data freshnessvsIngestion lag
Data freshness
the dashboard renders perfectly and every number on it is from Thursday.
How old the data behind an output is, measured from the source event rather than from the last successful run. It is the metric that catches the failure mode nothing else catches: a pipeline that succeeds while producing nothing. Measuring it at the point of consumption rather than at the pipeline is what makes it honest, because every intermediate success is compatible with a stale answer.
Full entry →Ingestion lag
the row was committed in the source at 09:00 and appeared in the warehouse at 09:41, and nobody could say which hop ate the forty minutes.
The delay between a change happening in the source and it being queryable at the destination. It is the number the business actually feels, and it is a sum over every hop, so it can only be diagnosed if each stage stamps its own arrival time. Measuring it end to end rather than per component is the difference between 'the pipeline is slow' and 'the connector is fine, the merge step waits for a lock'.
Full entry →