Comparison
Batch processingvsMicro-batch
Batch processing
nothing at all happened for twenty-three hours and then a job read the whole day at once.
Processing a bounded chunk of data on a schedule, rather than each record as it arrives. Batch is the boring, correct default: the input stops moving while you work on it, so a run is reproducible, cheap to reason about and trivial to re-run. Its whole cost is latency — the answer is as old as the gap between runs — and the usual mistake is paying for streaming to remove a delay nobody downstream actually minded.
Full entry →Micro-batch
the 'streaming' job actually wakes every thirty seconds, reads whatever arrived, and writes one file.
Running batch machinery on a short interval so it behaves approximately like a stream. You keep batch's restart story and its file-sized writes, and you accept latency equal to the interval. It is the honest middle of the market and it is what a great many production 'real-time' systems are, which matters when someone quotes you a five-second freshness figure that is really an interval plus a run time.
Full entry →