Queues, streams and delivery semantics

you push the slow job onto a queue and return to the user immediately, and a worker picks it up whenever it can.

Message queue

you publish the event once and three different services each get their own copy without the publisher knowing they exist.

Publish-subscribe

consumers read from a retained, ordered log at their own position, so a new consumer can start from the beginning of history.

Event stream

you record the position you have processed up to, so restarting the consumer does not start it from the beginning again.

Offset

you add a second worker and the partitions get split between them so each message is still only processed once.

Consumer group

you run ten copies of the worker against the same queue and whichever one is free grabs the next message.

Competing consumers

the producers are writing faster than the consumers are reading and the backlog has been growing all afternoon.

Consumer lag

the update arrived before the create because the two messages went to different partitions and were processed in parallel.

Message ordering

you acknowledge the message first and then process it, so a crash mid-processing means it is simply gone.

At-most-once delivery

the worker processed the message, crashed before acknowledging it, and the queue handed the same message to somebody else.

At-least-once delivery

the client sends a unique id with the payment request so that retrying it cannot charge the card twice.

Idempotency key

you make the duplicate delivery harmless by deduplicating on a key or committing the offset in the same transaction as the result.

Exactly-once processing

the message is hidden from other workers while you process it, and reappears if you do not finish in time.

Visibility timeout

one malformed payload keeps crashing the consumer, gets redelivered, crashes it again, and the whole queue stops moving.

Poison message

the message failed five times so it gets moved somewhere else, instead of blocking the queue forever by being retried in a loop.

Dead letter queue

you fixed the bug and reset the consumer back to last Tuesday so it processes the last week of events again correctly.

Replay

you write the row and the event to publish into the same database transaction, so you can never commit one without the other.

Transactional outbox

instead of asking the service to publish events, you read the database's own replication log and turn every row change into one.

Change data capture