Comparison
Micro-batchvsStream processing
Micro-batch
the 'streaming' job actually wakes every thirty seconds, reads whatever arrived, and writes one file.
Running batch machinery on a short interval so it behaves approximately like a stream. You keep batch's restart story and its file-sized writes, and you accept latency equal to the interval. It is the honest middle of the market and it is what a great many production 'real-time' systems are, which matters when someone quotes you a five-second freshness figure that is really an interval plus a run time.
Full entry →Stream processing
records are handled one at a time as they land, and the job has been running without finishing for four months.
Processing records continuously as they arrive, over an input that never ends. The job is a long-lived process holding state rather than something that starts and stops, which is what makes every hard streaming question — restarts, ordering, exactly-once, late data — a question about that state. Choose it when the value of an answer actually decays in minutes; choose batch when someone just said 'real time' in a meeting and meant 'not tomorrow'.
Full entry →