Comparison
DeduplicationvsDeduplication window
Deduplication
the same event is in the table three times because the connector retried, and you keep the latest by version per key.
Removing repeated records that represent the same real event or entity. In a pipeline it is nearly always the sharp end of at-least-once delivery: duplicates are expected, and the deduplication rule is what makes the result correct. The rule needs both a key and a deterministic tiebreak, and picking arrival order as the tiebreak is what makes a re-run produce a different table.
Full entry →Deduplication window
you keep the ids you have seen for the last twenty-four hours and drop repeats, and a duplicate that arrives twenty-five hours later gets through.
Suppressing repeated records by remembering their keys for a bounded period. It is the cheap, honest form of exactly-once: correct within the window, and stated as such. The window length is a memory budget, and it should be chosen against the worst realistic retry delay of the upstream system rather than against a round number, because the failure it is protecting against is exactly the one that takes a long time.
Full entry →