Time and correctness in streams

the phone was in a tunnel, so the tap happened at 14:02 and your job saw it at 16:40, and the hourly chart has to put it at 14:00.

Event time

you restarted the job to fix a bug and every bucket moved, because the buckets were made from when the job read the record.

Processing time

the record has no usable timestamp of its own, so you stamp it as it enters the log and everything downstream agrees on that.

Ingestion time

the update arrived before the insert, and your naive consumer wrote a row it then couldn't find a parent for.

Out-of-order events

the job announces that it believes nothing older than 14:30 is still coming, and closes every window that ended before then.

Watermark

yesterday's total went up this morning, because eight hundred events from a phone that was offline finally arrived.

Late-arriving data

the window emitted a result at 15:00 and keeps updating it for another two hours before it finally throws the state away.

Allowed lateness

the dashboard shows a partial count that keeps rising, because the window emits early every ten seconds before it finally closes.

Window trigger

every event lands in exactly one five-minute bucket and the buckets sit end to end with no gaps.

Tumbling window

you want 'five errors in any sixty seconds', so every event is counted in twelve overlapping windows at once.

Sliding window

the window has no fixed length; it just stays open until the user does nothing for thirty minutes, and then it closes.

Session window

one quiet partition sent nothing for an hour and the whole job's watermark froze, so no window closed anywhere.

Idle partition stall

the job is holding four hundred gigabytes of per-user counters, and 'just restart it' stopped being free some time ago.

Stateful stream processing

the running total is kept per customer id, so scaling out just moves whole customers between workers.

Keyed state

the operator's state is an embedded key-value store on local disk, and losing that disk means restoring from remote storage before anything can run.

State backend

the job died and came back at the position it had saved ninety seconds ago, replaying everything since without duplicating output.

Pipeline checkpoint

you needed to deploy a new version of the job, so you took a snapshot deliberately, stopped it, and started the new code from that snapshot.

Savepoint

the job crashed mid-window, replayed the last two minutes, and the output table still shows each event counted exactly one time.

End-to-end exactly-once

the writes are staged and only made visible when the checkpoint commits, so a crash halfway leaves nothing half-written.

Transactional sink

you keep the ids you have seen for the last twenty-four hours and drop repeats, and a duplicate that arrives twenty-five hours later gets through.

Deduplication window

you need the click matched to the impression, and one of them is always five minutes behind the other.

Streaming join

you priced the order using the exchange rate as it was at the moment of the order, not the rate as it is now.

Temporal join

the same topic can be read as a log of every change or collapsed into a table of current values, and which one you want depends on the question.

Stream-table duality

there are two implementations of the same metric, one fast and approximate and one slow and correct, and they disagree by two percent.

Lambda architecture

there is one streaming job, and correcting history means replaying the log through it rather than running a different batch job.

Kappa architecture