Comparison
End-to-end exactly-oncevsPipeline checkpoint
End-to-end exactly-once
the job crashed mid-window, replayed the last two minutes, and the output table still shows each event counted exactly one time.
The guarantee that each input record affects the output exactly once, despite retries and restarts. It is never achieved by delivering once; it is achieved by replaying at-least-once and making the effect idempotent or transactional, which means it is a property of the sink as much as of the engine. A pipeline claiming it while writing to a plain HTTP endpoint does not have it, and the place to check is always the last hop.
Full entry →Pipeline checkpoint
the job died and came back at the position it had saved ninety seconds ago, replaying everything since without duplicating output.
A periodic consistent snapshot of a streaming job's state and input positions, taken so a failure can resume from it rather than from the beginning. Everything about a streaming job's recovery time and its delivery guarantee is set here: the interval bounds how much is re-read, and the atomicity of the snapshot with respect to output is what makes end-to-end exactly-once possible. Frequent checkpoints cost throughput; rare ones cost a long, visible catch-up after every restart.
Full entry →