Comparison
Late-arriving datavsWatermark
Late-arriving data
yesterday's total went up this morning, because eight hundred events from a phone that was offline finally arrived.
Records whose event time falls into a window that has already been computed. Every distributed source produces them — offline devices, retried deliveries, a partition that fell behind — so the question is never whether they exist but what you do with them: drop, count separately, or restate the window. Whichever you choose is visible to whoever reads the number, which is why it belongs in a conversation with them rather than in a config file.
Full entry →Watermark
the job announces that it believes nothing older than 14:30 is still coming, and closes every window that ended before then.
The stream's assertion about how far event time has advanced — a claim that records older than this are no longer expected. It is what lets an unbounded stream ever produce a finished answer, because a window cannot close until something declares the input for it complete. It is a heuristic, not a fact: set it aggressively and you drop real data, set it conservatively and every result waits, which is the central tuning decision in any streaming job.
Full entry →