Data engineering·Time and correctness in streams
the job died and came back at the position it had saved ninety seconds ago, replaying everything since without duplicating output.
Pipeline checkpoint
Also calledstreaming checkpoint, consistent snapshot
A periodic consistent snapshot of a streaming job's state and input positions, taken so a failure can resume from it rather than from the beginning. Everything about a streaming job's recovery time and its delivery guarantee is set here: the interval bounds how much is re-read, and the atomicity of the snapshot with respect to output is what makes end-to-end exactly-once possible. Frequent checkpoints cost throughput; rare ones cost a long, visible catch-up after every restart.