Comparison
Idempotent pipelinevsPartition overwrite
Idempotent pipeline
you ran the same job three times after a failure and the output table is identical to running it once.
A pipeline whose run for a given period can be repeated with the same result, usually by deleting and replacing that period's output rather than appending to it. It is the property that makes retries, backfills and incident recovery routine instead of frightening. Without it every failure becomes a manual investigation into what was half-written, which is the difference between a five-minute morning and a lost day.
Full entry →Partition overwrite
the run deletes its own day's partition and writes it again, so re-running never duplicates a row.
The standard implementation of pipeline idempotency: a run owns a partition and replaces it wholesale. It is simple, it needs no keys and no merge logic, and it bounds the damage of a bad run to one partition. Its limits are that late-arriving data belonging to an old partition needs that partition re-run, and that the partition has to actually be the unit a run produces, which is a modelling decision made much earlier.
Full entry →