Comparison
Deduplication windowvsEnd-to-end exactly-once
Deduplication window
you keep the ids you have seen for the last twenty-four hours and drop repeats, and a duplicate that arrives twenty-five hours later gets through.
Suppressing repeated records by remembering their keys for a bounded period. It is the cheap, honest form of exactly-once: correct within the window, and stated as such. The window length is a memory budget, and it should be chosen against the worst realistic retry delay of the upstream system rather than against a round number, because the failure it is protecting against is exactly the one that takes a long time.
Full entry →End-to-end exactly-once
the job crashed mid-window, replayed the last two minutes, and the output table still shows each event counted exactly one time.
The guarantee that each input record affects the output exactly once, despite retries and restarts. It is never achieved by delivering once; it is achieved by replaying at-least-once and making the effect idempotent or transactional, which means it is a property of the sink as much as of the engine. A pipeline claiming it while writing to a plain HTTP endpoint does not have it, and the place to check is always the last hop.
Full entry →