Queues & Async Workflows

Not everything should happen inside the user’s request. When a user uploads a video, they shouldn’t wait for it to transcode; when they sign up, they shouldn’t wait for the welcome email to send. A message queue lets you accept the work now and do it later — decoupling the fast path from the slow path.

The core idea: decoupling producers and consumers

A queue sits between a producer (which puts messages in) and one or more consumers (which take them out and process them). The producer doesn’t know or care who processes the message, or when. This buys you three big properties:

• Responsiveness — return to the user immediately after enqueuing.
• Load smoothing — a traffic spike fills the queue instead of overwhelming your workers; they drain it at their own pace.
• Resilience — if a consumer crashes, the message is still in the queue and gets retried, rather than lost.

# Producer — inside the web request, returns instantly
queue.publish("emails", {"to": user.email, "template": "welcome"})
return {"status": "accepted"}         # HTTP 202

# Consumer — a separate worker process
for msg in queue.consume("emails"):
    send_email(msg["to"], msg["template"])
    msg.ack()                          # acknowledge only after success

Queues vs. streams

Two related shapes you should distinguish:

• Task queues (RabbitMQ, SQS, Celery) — a message is a unit of work handed to one consumer, then removed. Think “process this job.”
• Event streams / logs (Kafka, Kinesis, Pulsar) — an append-only log that many consumers read independently, each tracking its own position. Think “here is a firehose of events; subscribe to it.” Great for analytics, audit logs, and fanning one event out to several systems.

Delivery guarantees and idempotency

Distributed messaging can’t perfectly promise “exactly once.” Most systems give you at-least-once delivery — a message may be delivered more than once (e.g. a consumer processed it but crashed before acknowledging). The practical consequence is a rule you must internalize:

Design consumers to be idempotent — processing the same message twice has the same effect as processing it once.

You achieve idempotency by keying on a unique message ID and skipping work you’ve already done, or by making operations naturally repeatable (“set status = paid” rather than “add $10”).

Backpressure and dead letters

• Backpressure — when producers outrun consumers, the queue grows. You need a plan: scale up consumers, shed load, or cap the queue. A silently unbounded queue is a latent outage.
• Dead-letter queue (DLQ) — a message that keeps failing shouldn’t retry forever. After N attempts, route it to a DLQ for inspection so one poison message can’t block the pipeline.

When to introduce a queue

Reach for async processing when work is slow (media processing, LLM calls), spiky (bursts you want to absorb), failure-prone (third-party APIs you’ll want to retry), or simply not needed for the response (emails, analytics, webhooks). If the user genuinely needs the result now, keep it synchronous — adding a queue where you need an immediate answer just adds latency and complexity.

Pattern recognition

Variations

Worked problems

Queue designs are capacity designs. State arrival rate, drain rate, retry policy, and idempotency before naming a queue product.

2,000,000 images / 20 minutes
20 minutes = 1,200 seconds
arrival ~= 1,667 images/sec

100 workers × 5 images/sec = 500 images/sec
backlog growth ~= 1,167 images/sec

1,667 images/sec / 5 per worker ~= 334 workers
Add headroom: ~400 workers

receive event payment_succeeded:evt_123
create fulfillment row
crash before ack
queue redelivers evt_123
consumer creates fulfillment again unless guarded

begin transaction
  insert processed_event(evt_123)  # unique key
  if insert conflicts:
      ack message; return
  mark order paid
  create fulfillment if not exists for order_id
commit
ack message

attempt 1: immediate retry for obvious transient failure
attempt 2-5: exponential backoff with jitter
attempt >5: move to DLQ with error type, stack, payload pointer, first_seen_at