Data engineering·Moving data out of the source
the 'streaming' job actually wakes every thirty seconds, reads whatever arrived, and writes one file.
Micro-batch
Also calledmini-batch, micro-batching
Running batch machinery on a short interval so it behaves approximately like a stream. You keep batch's restart story and its file-sized writes, and you accept latency equal to the interval. It is the honest middle of the market and it is what a great many production 'real-time' systems are, which matters when someone quotes you a five-second freshness figure that is really an interval plus a run time.