At-least-once delivery

What is at-least-once delivery?

At-least-once delivery is a messaging guarantee: every message should be delivered one or more times. The system tries hard not to lose the message. The trade-off is that the same message can arrive twice.

The duplicate usually appears around acknowledgements. A sender publishes a message and waits for confirmation that it was received or processed. If the acknowledgement does not arrive, the sender cannot tell whether the message was lost or only the acknowledgement was lost. The safe move is to send the message again. If the first message did arrive, the receiver now sees it twice.

That is the whole bargain. At-least-once protects against loss by accepting the possibility of duplicates.

Why at-least-once is common

For most business data, a duplicate is easier to repair than a missing event. If an order event arrives twice, a careful consumer can recognise the order ID and ignore the second copy. If the order event never arrives, the downstream system may never know it missed anything.

That is why at-least-once delivery shows up across queues, pub/sub systems, and streaming platforms. Amazon SQS standard queues warn that a message may occasionally be delivered more than once and tell you to make processing idempotent. Google Pub/Sub offers at-least-once delivery by default. RabbitMQ gives at-least-once semantics when acknowledgements are used correctly. Kafka-based pipelines commonly end up in at-least-once territory unless the whole producer, processing, and sink path is designed for stricter guarantees.

At-least-once is not sloppy engineering. It is a deliberate distributed-systems trade-off: do not lose data, and make the consumer safe against repeats.

The three delivery guarantees

• At-most-once: zero or one delivery. A message is not duplicated, but it can be lost.

• At-least-once: one or more deliveries. A message should not be lost, but it can be duplicated.

• Exactly-once: one effective processing result. This needs coordination and usually works only within a carefully bounded system.

The important word in real projects is often effective. A pipeline can receive the same event twice and still produce exactly one order row if the consumer is idempotent.

What this means for the consumer

If a consumer reads from an at-least-once stream, it must assume that a message may arrive again. Two patterns make that safe.

Idempotent processing
Running the same message twice leaves the target in the same state as running it once. An upsert by order ID is idempotent. A blind insert is not.

Deduplication key
Each message carries a stable unique key, such as an event ID, transaction ID, order ID, or source log position. The consumer records which keys it has already handled and skips repeats.

Example: a source sends order 8842 was placed. The consumer writes the order to a warehouse, but the acknowledgement is lost. The source retries. If the consumer blindly inserts rows, the order appears twice. If it merges on order ID or tracks the event ID, the second copy changes nothing.

Where at-least-once shows up

At-least-once is normal in streaming data, Change Data Capture, webhooks, queue-based background jobs, and ingestion pipelines. A webhook provider may retry until your endpoint returns a success code. A CDC tool may resend after a restart. A queue may redeliver a message if a worker crashes before acknowledging it.

The practical design question is not can duplicates happen? The answer is yes. The question is whether duplicates are harmless because the target logic was designed for them.

What to watch out for with at-least-once delivery

Do not acknowledge too early
If you acknowledge before the target write is durable, a crash can lose the message. Acknowledge after the work is safely recorded.

Do not rely on timestamps for deduplication
Timestamps are useful for ordering and windows, but they are rarely unique. Use a real event key.

Side effects are harder than table writes
Sending an email, charging a card, or calling an external API twice can hurt. These steps need their own idempotency key or outbox pattern.

Exactly-once claims have boundaries
A platform may provide exactly-once processing inside its own components. The moment you call an external system, you need to check whether that system participates in the guarantee.