Comparison
CatchupvsPipeline backfill
Catchup
you enabled a pipeline with a start date last January and it immediately queued four hundred runs.
The scheduler's behaviour of automatically running every interval between a pipeline's start date and now. It is genuinely useful for a new model over historical data and genuinely dangerous by default, because a paused pipeline resuming will try to do the same thing. It is worth knowing as a named setting because the accident it causes — a source system flattened by four hundred simultaneous extracts — is loud and entirely preventable.
Full entry →Pipeline backfill
the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.
Re-running a pipeline over historical periods to correct or populate them. Whether it is routine or terrifying is decided entirely by idempotency and by blast radius: a backfill of an idempotent partitioned model is boring, and a backfill of an append-only table is a duplication incident. The other half is resource contention — ninety runs at once will happily starve the scheduled work — which is why concurrency limits and a deliberate order matter more than the code does.
Full entry →