Comparison
DeduplicationvsIdempotency key
Deduplication
the same event is in the table three times because the connector retried, and you keep the latest by version per key.
Removing repeated records that represent the same real event or entity. In a pipeline it is nearly always the sharp end of at-least-once delivery: duplicates are expected, and the deduplication rule is what makes the result correct. The rule needs both a key and a deterministic tiebreak, and picking arrival order as the tiebreak is what makes a re-run produce a different table.
Full entry →Idempotency key
the client sends a unique id with the payment request so that retrying it cannot charge the card twice.
A caller-supplied identifier that lets the server recognise a retry of a request it has already handled and return the original result. It is the standard way to make a non-idempotent operation safe to retry over an unreliable network. The server side needs storage with a retention window and a rule for what happens when the same key arrives with a different body.
Full entry →