Comparison
Keyed statevsState backend
Keyed state
the running total is kept per customer id, so scaling out just moves whole customers between workers.
State scoped to a partition key, so each worker owns a disjoint slice of it. Making state keyed is what makes a streaming job parallelisable at all, because two workers never need to see the same counter. The consequence is that your key choice is now a partitioning decision, an ordering decision and a memory-hotspot decision at once — one enormous customer becomes one enormous worker.
Full entry →State backend
the operator's state is an embedded key-value store on local disk, and losing that disk means restoring from remote storage before anything can run.
Where a streaming job's state actually lives — on the JVM heap for speed, or in an embedded on-disk store for size, backed up to object storage for durability. The choice sets your ceiling on state size and your recovery time, and it is nearly always the thing behind 'why is this job slow' once the state exceeds memory. It is worth knowing because state operations are invisible in the job's logic and dominant in its performance profile.
Full entry →