Comparison
Idle partition stallvsWatermark
Idle partition stall
one quiet partition sent nothing for an hour and the whole job's watermark froze, so no window closed anywhere.
A watermark that stops advancing because it is the minimum across all inputs and one input has gone silent. It is a signature streaming failure: nothing is broken, nothing errors, and output simply stops. The fixes are all forms of admitting the silence — an idleness timeout that lets the quiet source be excluded, or the source emitting periodic heartbeats — and knowing the shape is what turns a three-hour investigation into a five-minute one.
Full entry →Watermark
the job announces that it believes nothing older than 14:30 is still coming, and closes every window that ended before then.
The stream's assertion about how far event time has advanced — a claim that records older than this are no longer expected. It is what lets an unbounded stream ever produce a finished answer, because a window cannot close until something declares the input for it complete. It is a heuristic, not a fact: set it aggressively and you drop real data, set it conservatively and every result waits, which is the central tuning decision in any streaming job.
Full entry →