On this page
Description
Not everything a request triggers has to happen before the user gets an answer. Sign-up, for example, does five things: create the account, send a welcome email, resize the avatar, add the user to the CRM, and log an analytics event. Done one after another inside the request, sign-up takes 3 seconds - and if the email provider has a bad day, sign-up fails. Yet only the first step has to happen before the user sees "Welcome!".
A message queue lets the request do only what must happen now (create the account), publish a message - "user 42 signed up" - and return. Consumers do the rest at their own pace, retrying as needed. The request is fast, a slow email provider no longer breaks sign-up, and a traffic spike becomes a backlog instead of an outage. In this lesson:
- Work queues versus logs (SQS and RabbitMQ versus Kafka).
- Delivery guarantees - and why "exactly once" really means idempotent consumers.
- Keeping events in order.
- Retries, poison messages and dead-letter queues.
- Back-pressure when consumers can't keep up.
- The outbox: saving data and publishing an event without losing either.
Work queues and logs
| Work queue (SQS, RabbitMQ) | Log (Kafka, Kinesis) | |
|---|---|---|
| A message goes to | One consumer | Every consumer group (each reads all messages) |
| After processing | Deleted | Kept for the retention period |
| Replay | No | Yes - rewind a group's offset |
| Ordering | Best-effort (FIFO queues exist) | Strict within a partition |
| Typical use | Jobs: send this email, resize this image | Event streams many services react to |
If search indexing, analytics and notifications all need "order placed", a log lets each consume the same stream independently:
Within a group, each partition is read by exactly one consumer; each group reads every partition and keeps its own position (offset).
Delivery guarantees
A consumer acknowledges a message to say "done". When it acknowledges decides what can go wrong:
- At-most-once - acknowledge, then process. A crash mid-processing loses the message.
- At-least-once - process, then acknowledge. A crash (or a timeout) before the acknowledgment means the message is delivered again - duplicates.
- Exactly-once as an end-to-end effect comes from at-least-once delivery plus an idempotent consumer: processing the same message twice has the same effect as once. Record processed message IDs (or an idempotency key such as the payment ID) and skip repeats, or make the operation naturally idempotent ("set status = SHIPPED", not "add 1").
A broker's "exactly-once" setting covers its own reads and writes; it can't un-send an email or un-charge a card your consumer already did.
Ordering
Logs are split into partitions for throughput, and order is guaranteed only within a partition. Events that must stay in order - "order created", "order paid", "order shipped" - use the same partition key (the order ID) so they land in one partition, while different orders spread across all partitions. One partition for everything preserves global order but caps throughput at one consumer.
Retries, poison messages and dead letters
Consumers fail - a dependency is down, a bug hits one record. Retry with exponential backoff (1 s, 2 s, 4 s ...) so a struggling dependency isn't hammered. But a poison message - malformed, or hitting a bug - fails every time; retried forever, it blocks everything behind it. After a fixed number of attempts, move it to a dead-letter queue (DLQ): processing continues, an alert fires, and someone inspects and replays it after a fix.
Back-pressure
If producers publish 5,000 messages a second and consumers handle 3,000, the backlog grows by 7.2 million an hour. Eventually messages outlive the retention period and are lost, or the broker's disk fills. Handle it:
- Watch consumer lag (how far behind the consumers are) and alert on it.
- Scale consumers on lag - up to the number of partitions, the maximum parallelism of a log.
- Push back when you can't keep up: slow producers down, reject non-essential work, or shed load at the edge.
The dual-write problem and the outbox
A service saves an order and publishes "order placed". Written as two steps, a crash between them leaves an order with no event (or, reversed, an event for an order that was never saved). The transactional outbox fixes it: in the same database transaction as the order, insert a row into an outbox table. A separate relay reads the outbox and publishes each row, marking it sent - at-least-once, so consumers stay idempotent.
Which tool?
| Need | Tool |
|---|---|
| Background jobs, each done once (emails, thumbnails) | A work queue |
| One event, many independent consumers; replay | A log (Kafka, Kinesis, Pulsar) |
| Per-entity ordering | A log, partitioned by the entity ID |
| A job at a future time (reminders, retries later) | A delay queue or a scheduler table |
| Write to the database and publish, reliably | Transactional outbox (or change data capture) |
| A multi-step business process with compensations | A workflow engine or saga orchestrator |
Going further. The deep dive (next tab) covers what Kafka's durability settings really guarantee, sizing partitions and draining a backlog, why retries break ordering, fairness between tenants sharing a queue, evolving event schemas safely, and orchestration versus choreography.
Check yourself
Try to answer before opening each one - recalling beats re-reading.
1. A consumer charges a card and crashes before acknowledging. What happens next, and what prevents a double charge?
Show answer
The message is redelivered (at-least-once), and a naive consumer charges again. Pass an idempotency key (the payment ID) to the payment provider, or record processed message IDs and skip repeats.
2. Why can't you just use one Kafka partition to keep all events in order?
Show answer
One partition is consumed by one consumer at a time, so it caps throughput. Usually only events for the same entity need ordering - key by the entity and spread the rest.
3. What is a dead-letter queue for?
Show answer
Messages that keep failing after the retry limit. Moving them aside lets processing continue, and keeps them for inspection and replay once the cause is fixed.
Key takeaways
- Queue work that can wait: faster requests, isolation from slow dependencies, spikes become backlogs.
- Work queues hand each message to one consumer; logs keep messages for many groups and replay.
- Delivery is at-least-once: make consumers idempotent.
- Order holds within a partition - key related events together.
- Retry with backoff and a cap; send poison messages to a DLQ.
- Watch consumer lag; scale consumers or push back on producers.
- Use a transactional outbox so a write and its event can't half-happen.