Comparison
Allowed latenessvsWatermark
Allowed lateness
the window emitted a result at 15:00 and keeps updating it for another two hours before it finally throws the state away.
How long after a window closes the job will still accept and re-emit for records belonging to it. It buys correctness with memory and with downstream churn — every update is a rewrite somebody has to handle — and it is bounded because state cannot be kept forever. Setting it is a business conversation dressed as a config value: how wrong may this number be for how long, and can the consumer cope with it changing.
Full entry →Watermark
the job announces that it believes nothing older than 14:30 is still coming, and closes every window that ended before then.
The stream's assertion about how far event time has advanced — a claim that records older than this are no longer expected. It is what lets an unbounded stream ever produce a finished answer, because a window cannot close until something declares the input for it complete. It is a heuristic, not a fact: set it aggressively and you drop real data, set it conservatively and every result waits, which is the central tuning decision in any streaming job.
Full entry →