Comparison
Competing consumersvsConsumer group
Competing consumers
you run ten copies of the worker against the same queue and whichever one is free grabs the next message.
Multiple consumers pulling from one queue so throughput scales with worker count and slow messages do not block fast ones. It is the default scaling model for work queues. It gives up ordering: two related messages can be processed simultaneously by different workers in either order.
Full entry →Consumer group
you add a second worker and the partitions get split between them so each message is still only processed once.
A set of consumers that jointly consume a stream, with each partition assigned to exactly one member. It is how a log-based system gives you both queue semantics and horizontal scaling. Parallelism is capped by partition count: adding consumers past that number gives you idle processes, not throughput.
Full entry →