Comparison
IdempotentvsIdempotent pipeline
Idempotent
you can safely run the same operation five times and the system ends up in exactly the state it would after one.
The property that applying an operation repeatedly has the same effect as applying it once. It is what makes retries safe, and retries are unavoidable in any networked system, because a timeout does not tell you whether the work happened. Idempotence is about the effect on state, not about the return value or the work performed.
Full entry →Idempotent pipeline
you ran the same job three times after a failure and the output table is identical to running it once.
A pipeline whose run for a given period can be repeated with the same result, usually by deleting and replacing that period's output rather than appending to it. It is the property that makes retries, backfills and incident recovery routine instead of frightening. Without it every failure becomes a manual investigation into what was half-written, which is the difference between a five-minute morning and a lost day.
Full entry →