Orchestration and running it in anger

something has to decide that the transform runs only after all four extracts finished, and retry the one that did not.

Orchestrator

the pipeline is drawn as boxes with arrows, nothing points backwards, and a step waits for everything pointing at it.

DAG

the job is set to hourly, takes seventy minutes, and now two copies are running at once.

Schedule interval

the first step of the pipeline does nothing but sit there checking every minute whether the file has landed.

Sensor

the model runs when the table it depends on is actually updated, rather than at half past two because that is usually late enough.

Data-aware scheduling

the run that started at 02:00 on Tuesday is processing Monday's data, and half the confusion in the team is about which date the run is named after.

Logical date

you ran the same job three times after a failure and the output table is identical to running it once.

Idempotent pipeline

the run deletes its own day's partition and writes it again, so re-running never duplicates a row.

Partition overwrite

the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.

Pipeline backfill

you enabled a pipeline with a start date last January and it immediately queued four hundred runs.

Catchup

you cap the number of tasks that may touch the production replica at four, so a backfill cannot take the source down.

Concurrency limit

one malformed row fails the batch, the batch retries, and the same row fails it again every five minutes for two days.

Poison record

rows that fail validation go to a side table with the reason attached instead of failing the load or being silently dropped.

Quarantine table

the scheduler thinks the task is running, the worker that was running it died an hour ago, and nothing has noticed.

Zombie task

someone marked the failed task as successful at 03:40 so the rest of the pipeline could proceed, and did not write down why.

Manual intervention

you need to know whether last Tuesday's run of this model succeeded, and how long it took the last thirty times.

Pipeline run history

the dashboard renders perfectly and every number on it is from Thursday.

Data freshness

the agreement says this table is never more than three hours behind, so a four-hour delay is a breach rather than an opinion.

Freshness SLA

the number is wrong and you need to know every table between it and the source, without reading forty SQL files.

Data lineage

you need to know whether dropping this one column breaks anything, not whether the table is used.

Column-level lineage

before changing the model you list every dashboard, export and reverse-ETL sync that reads it, and it is longer than you expected.

Impact analysis

two pipelines have no declared relationship and one of them only works because the other happens to run first.

Pipeline dependency inversion

the source team changed a status code's meaning, the pipeline kept running, and your funnel report has been wrong since Tuesday.

Upstream drift

the source team's deploy pipeline fails if they remove a field your models depend on, because the expectation is written down and checked.

Data contract

the producer registers the schema and gets an id, and the consumer looks that id up rather than guessing at the payload.

Schema registry

you tried to add a required field and the registry rejected it, because old consumers could not read the new messages.

Schema compatibility mode

before merging the refactor you run the new model over last month and diff its output against what the old one produced.

Golden run