Comparison
DeduplicationvsUniqueness test
Deduplication
the same event is in the table three times because the connector retried, and you keep the latest by version per key.
Removing repeated records that represent the same real event or entity. In a pipeline it is nearly always the sharp end of at-least-once delivery: duplicates are expected, and the deduplication rule is what makes the result correct. The rule needs both a key and a deterministic tiebreak, and picking arrival order as the tiebreak is what makes a re-run produce a different table.
Full entry →Uniqueness test
the key you have been treating as unique turns out to have four hundred duplicates, which is why the join inflated the total.
Asserting that a column or combination is unique at the table's stated grain. It is the test that most directly protects against fan traps, because a duplicated key is how a correct query returns an inflated number. It is also the cheapest way to detect an at-least-once pipeline that has stopped deduplicating, which is a failure with no other symptom.
Full entry →