Message queue

What is a message queue?

A message queue sits between two systems and holds work in transit. One system, the producer, writes a message into the queue. Another system, the consumer, reads it and processes it when it has capacity. Neither side has to be online at the same moment.

That gap is the whole point. A webshop can accept an order the instant a customer clicks buy, drop it in the queue, and let the accounting system pull it out a few seconds later. If accounting is restarting or running behind, the order waits in line instead of failing.

Think of it as the ticket counter at a busy bakery. You take a number and leave your request, and the staff work through the tickets at their own pace. Products like RabbitMQ, Amazon SQS and Azure Service Bus are message queues built around exactly this idea.

How producers, consumers and acknowledgement work

A producer publishes a message and moves on. It does not wait for the consumer. On the other end, a consumer receives the message, does its work, and then acknowledges it, which tells the queue it can drop the message.

The acknowledgement step is what makes a queue reliable. Until a message is acknowledged, the queue assumes the work might still fail. Amazon SQS models this with a visibility timeout: when a consumer receives a message it stays in the queue but turns invisible to everyone else for a set period, 30 seconds by default. The consumer processes the message and deletes it. If the consumer crashes and never deletes it, the message becomes visible again after the timeout and another consumer picks it up.

RabbitMQ uses the same principle with manual acknowledgements: an unacknowledged message whose consumer connection drops is automatically requeued and delivered again. Azure Service Bus calls its version a peek-lock. The mechanism differs, the promise is the same. A message is not lost just because a consumer fell over halfway through.

Delivery guarantees and why idempotence matters

Most queues give you at-least-once delivery in practice. A message will be delivered, and under retries or network trouble it may be delivered more than once. Amazon SQS states this plainly for standard queues: they provide at-least-once delivery, but more than one copy of a message might be delivered and messages can arrive out of order.

The consequence lands on the consumer. If the same order message can arrive twice, processing it twice must not create two invoices. This is why queue consumers should be idempotent: running the same message again leaves the system in the same state as running it once. Idempotence and at-least-once delivery are a pair you design for together. Some brokers advertise exactly-once processing, but the guarantee usually stops at the broker boundary, so the moment your consumer writes to an outside database or API you are back to needing idempotent writes.

Queue versus log

Two different structures both get called message queues, and they behave differently once a message is read.

A classic queue, such as RabbitMQ or Amazon SQS, is built for work distribution. A message is delivered to one consumer out of a competing pool, and once that consumer acknowledges it, the queue removes it. Ten workers can share one queue and each order is handled once by exactly one of them.

A log, such as Apache Kafka, does not remove a message when it is read. It appends events to a partition and keeps them for a configured retention period, whether or not anyone consumed them. Each consumer tracks its own position with an offset that it controls, so it reads at its own speed and can rewind that offset to reprocess older events from scratch. That is not possible in a classic queue, where a consumed message is already gone.

The rule of thumb: reach for a queue when the job is "do this one task once", and for a log when the job is "let several independent readers replay this stream of events". Confusing the two is a common design mistake.

When to use a message queue

1. Talking to an unreliable downstream. When a partner API or an internal system goes down for maintenance, messages wait in the queue and drain once it comes back.

2. Moving slow work off the request. Generating a PDF, sending email, or resizing an upload runs in the background, so the user is not left staring at a spinner.

3. Wiring up an event-driven architecture. Services publish events and react to them without calling each other directly, which is the backbone of event-driven architecture.

What to watch out for with message queues

Design consumers to tolerate duplicates.
Because delivery is at-least-once, the same message can land twice. Use an idempotency key or an upsert so a repeat does not double-charge or double-insert.

Watch queue depth, not just error rates.
A queue can quietly fill faster than it drains while every component still reports healthy. Track the number of waiting messages and the age of the oldest one, because that age is your real latency.

Have a plan for messages that never succeed.
A single poison message that fails every attempt should not block the line. Most brokers move it to a dead-letter queue after a set number of tries; Azure Service Bus does this after a default of ten delivery attempts. Pair that with a retry policy for transient failures, and a circuit breaker so a failing downstream is not hammered.

Decide what travels inside the message.
Sending the full payload keeps the consumer self-contained but can go stale; sending only an ID that the consumer looks up stays fresh but adds a dependency. Choose deliberately per message type.