Data engineering·Quality, tests and knowing it is wrong
the same event is in the table three times because the connector retried, and you keep the latest by version per key.
Deduplication
Also calleddedupe, duplicate removal
Removing repeated records that represent the same real event or entity. In a pipeline it is nearly always the sharp end of at-least-once delivery: duplicates are expected, and the deduplication rule is what makes the result correct. The rule needs both a key and a deterministic tiebreak, and picking arrival order as the tiebreak is what makes a re-run produce a different table.