Comparison
Message queuevsQueueing and async processing
Message queue
you push the slow job onto a queue and return to the user immediately, and a worker picks it up whenever it can.
A buffer of work items consumed by workers, where each message is delivered to one consumer and removed once acknowledged. It decouples request latency from work duration and absorbs bursts. It also converts synchronous errors into invisible ones, so a queue without a dead letter path and depth alerting is a place for work to go and quietly die.
Full entry →Queueing and async processing
nobody is waiting on this call, so it goes on a queue with bounded concurrency and the 429 storm stops.
Decoupling LLM work from request threads for anything a user is not actively awaiting: enqueue, process with controlled concurrency under the rate limit, deliver results. Smooths bursts and stops 429 storms.
Full entry →