Comparison
End-to-end exactly-oncevsExactly-once processing
End-to-end exactly-once
the job crashed mid-window, replayed the last two minutes, and the output table still shows each event counted exactly one time.
The guarantee that each input record affects the output exactly once, despite retries and restarts. It is never achieved by delivering once; it is achieved by replaying at-least-once and making the effect idempotent or transactional, which means it is a property of the sink as much as of the engine. A pipeline claiming it while writing to a plain HTTP endpoint does not have it, and the place to check is always the last hop.
Full entry →Exactly-once processing
you make the duplicate delivery harmless by deduplicating on a key or committing the offset in the same transaction as the result.
The property that each message affects the result exactly once. Exactly-once *delivery* over an unreliable network is impossible; exactly-once *effect* is achievable, by pairing at-least-once delivery with idempotent processing or a transaction spanning the output and the offset commit. When a vendor says exactly-once, this is what they built, and it only holds inside their system boundary.
Full entry →