Comparison
Row-count reconciliationvsSilent failure
Row-count reconciliation
you count the rows in the source and in the warehouse for the same period and prove that the numbers agree.
Comparing aggregates between source and destination to demonstrate that nothing was lost or duplicated in transit. Counts are the cheap version; sums of a monetary column are the version finance will accept. It is unglamorous, and it is the only control that actually catches a silently dropped batch, since every intermediate step reports success on the rows it did see.
Full entry →Silent failure
every task is green, the run took its usual eleven minutes, and it processed zero rows.
A pipeline reporting success while producing nothing or producing nonsense. It is the characteristic data failure, because most pipeline code cannot tell the difference between 'no new data' and 'the filter is broken'. The only defences are assertions about the output rather than about the run — freshness, volume, distribution — which is why quality checks are operational monitoring in this domain rather than a testing nicety.
Full entry →