Comparison
Ingestion timevsProcessing time
Ingestion time
the record has no usable timestamp of its own, so you stamp it as it enters the log and everything downstream agrees on that.
The moment a record entered your system, assigned once at the boundary. It sits between the other two clocks: unlike processing time it does not move when you re-run, and unlike event time it is under your control and monotonic within a partition. It is the pragmatic fallback for sources that send no timestamp or send a dishonest one, and the honest thing to do is name the column so nobody later mistakes it for when the thing happened.
Full entry →Processing time
you restarted the job to fix a bug and every bucket moved, because the buckets were made from when the job read the record.
The clock on the machine doing the work, at the moment it handles the record. It is always available, always in order and never late, which makes it a tempting default and makes every result computed on it irreproducible. Use it for operational questions about the pipeline itself — throughput, lag, how busy a consumer is — and never for a number a person will compare week to week.
Full entry →