Data engineering·Orchestration and running it in anger
you ran the same job three times after a failure and the output table is identical to running it once.
Idempotent pipeline
Also calledre-runnable pipeline, rerun-safe
A pipeline whose run for a given period can be repeated with the same result, usually by deleting and replacing that period's output rather than appending to it. It is the property that makes retries, backfills and incident recovery routine instead of frightening. Without it every failure becomes a manual investigation into what was half-written, which is the difference between a five-minute morning and a lost day.