Comparison
Batch processingvsStream processing
Batch processing
nothing at all happened for twenty-three hours and then a job read the whole day at once.
Processing a bounded chunk of data on a schedule, rather than each record as it arrives. Batch is the boring, correct default: the input stops moving while you work on it, so a run is reproducible, cheap to reason about and trivial to re-run. Its whole cost is latency — the answer is as old as the gap between runs — and the usual mistake is paying for streaming to remove a delay nobody downstream actually minded.
Full entry →Stream processing
records are handled one at a time as they land, and the job has been running without finishing for four months.
Processing records continuously as they arrive, over an input that never ends. The job is a long-lived process holding state rather than something that starts and stops, which is what makes every hard streaming question — restarts, ordering, exactly-once, late data — a question about that state. Choose it when the value of an answer actually decays in minutes; choose batch when someone just said 'real time' in a meeting and meant 'not tomorrow'.
Full entry →