Data engineering·Time and correctness in streams
you keep the ids you have seen for the last twenty-four hours and drop repeats, and a duplicate that arrives twenty-five hours later gets through.
Deduplication window
Also calleddedup window, duplicate suppression
Suppressing repeated records by remembering their keys for a bounded period. It is the cheap, honest form of exactly-once: correct within the window, and stated as such. The window length is a memory budget, and it should be chosen against the worst realistic retry delay of the upstream system rather than against a round number, because the failure it is protecting against is exactly the one that takes a long time.