Data engineering·Orchestration and running it in anger
the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.
Pipeline backfill
Also calledhistorical rerun, reprocessing history
Re-running a pipeline over historical periods to correct or populate them. Whether it is routine or terrifying is decided entirely by idempotency and by blast radius: a backfill of an idempotent partitioned model is boring, and a backfill of an append-only table is a duplication incident. The other half is resource contention — ninety runs at once will happily starve the scheduled work — which is why concurrency limits and a deliberate order matter more than the code does.