Comparison
At-least-once deliveryvsEnd-to-end exactly-once
At-least-once delivery
the worker processed the message, crashed before acknowledging it, and the queue handed the same message to somebody else.
A guarantee that every message is delivered, possibly more than once. It is the default in nearly every broker, because acknowledging after processing is the only way to survive consumer crashes. It makes duplicate handling the consumer's problem, which is why an at-least-once consumer that is not idempotent is a bug waiting for a redeploy.
Full entry →End-to-end exactly-once
the job crashed mid-window, replayed the last two minutes, and the output table still shows each event counted exactly one time.
The guarantee that each input record affects the output exactly once, despite retries and restarts. It is never achieved by delivering once; it is achieved by replaying at-least-once and making the effect idempotent or transactional, which means it is a property of the sink as much as of the engine. A pipeline claiming it while writing to a plain HTTP endpoint does not have it, and the place to check is always the last hop.
Full entry →