Comparison
Pipeline backfillvsRestatement
Pipeline backfill
the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.
Re-running a pipeline over historical periods to correct or populate them. Whether it is routine or terrifying is decided entirely by idempotency and by blast radius: a backfill of an idempotent partitioned model is boring, and a backfill of an append-only table is a duplication incident. The other half is resource contention — ninety runs at once will happily starve the scheduled work — which is why concurrency limits and a deliberate order matter more than the code does.
Full entry →Restatement
last quarter's published figures change after a backfill, and somebody has to tell the people who already presented them.
Publishing corrected values for a period that has already been reported. Technically it is a backfill; organisationally it is an announcement, because someone has made a decision or a filing based on the old number. The engineering consequence is that any table feeding published figures needs its versions to be reconstructible, so the change can be explained rather than merely applied.
Full entry →