Comparison
CatchupvsConcurrency limit
Catchup
you enabled a pipeline with a start date last January and it immediately queued four hundred runs.
The scheduler's behaviour of automatically running every interval between a pipeline's start date and now. It is genuinely useful for a new model over historical data and genuinely dangerous by default, because a paused pipeline resuming will try to do the same thing. It is worth knowing as a named setting because the accident it causes — a source system flattened by four hundred simultaneous extracts — is loud and entirely preventable.
Full entry →Concurrency limit
you cap the number of tasks that may touch the production replica at four, so a backfill cannot take the source down.
A cap on how many tasks may run at once, globally or against a named resource. It is the orchestrator's version of a bulkhead, and it is the mechanism that stops one greedy pipeline from starving everything else or overwhelming a shared dependency. Setting them per resource rather than globally is what makes them useful, because the thing you are protecting is the source database, not the scheduler.
Full entry →