Exactly-once processing

What is exactly-once processing?

Exactly-once processing means every event has exactly one effect in the final result, even when systems fail and messages are retried. If an order event arrives, it should increase revenue once: not zero times because the event was lost, and not twice because a retry was processed twice.

The important word is effect. A message may travel over the network more than once. It may be read again after a restart. A consumer may retry the same batch. Exactly-once processing is about making sure those repeats do not create duplicate rows, duplicate payments, duplicate stock movements, or duplicate alerts.

This matters most when events change money, inventory, customer state, access rights, or official counts. For low-value telemetry, at-least-once with harmless duplicates may be enough. For payments and stock, duplicate effects are not harmless.

Why exactly-once is hard

The classic problem is a lost acknowledgement.

A producer sends a message to a broker. The broker writes it and sends an acknowledgement back. If the acknowledgement disappears, the producer cannot tell which thing failed: did the message fail to arrive, or did only the acknowledgement fail to return?

If the producer does not retry, it may lose the message. If it retries, it may create a duplicate. A network cannot remove that uncertainty by trying harder. The system needs additional mechanisms: stable identifiers, deduplication, transactions, checkpoints, and sinks that participate in the guarantee.

How systems achieve it in practice

Most practical designs start from at-least-once delivery and then make duplicate effects impossible. That is sometimes called effectively-once processing.

Idempotent writes
Running the same event twice leaves the target in the same state as running it once. An upsert by order ID is idempotent. A blind insert into a sales table is not.

Deduplication keys
Each event carries a stable key: event ID, transaction ID, source log position, order number, or another unique value. The consumer records processed keys and ignores repeats.

Producer idempotence
Kafka's idempotent producer design uses producer IDs and sequence numbers so the broker can detect retry duplicates from the same producer session and avoid appending them twice.

Transactions
Kafka transactions can bind a read-process-write loop together: output records and consumed offsets are committed as one unit. Consumers that read only committed transactional data avoid seeing aborted output.

Checkpoints
Flink uses checkpoints to restore state and input position after failure. With compatible sinks, that lets a job recover as if the failed attempt had not partially happened.

The sink decides the boundary

This is the trap: exactly-once inside a broker or stream processor is not the same as exactly-once all the way into your warehouse, database, API, or SaaS system.

The destination, often called the sink, must participate. It needs transactions, idempotent upserts, a deduplication table, a two-phase commit protocol, or another way to make repeated writes safe. If the stream processor is transactional but the sink blindly inserts rows, duplicates can still appear at the end.

That is why the architecture matters more than the product claim. Ask: exactly once between which components, under which failure modes, and with which sink?

At-most-once, at-least-once, and exactly-once

At-most-once
A message is delivered zero or one time. It should not duplicate, but it can be lost. Useful for low-value telemetry where the next reading arrives soon.

At-least-once
A message is delivered one or more times. Loss is avoided, but duplicates are possible. This is common in queues, webhooks, CDC, and streaming platforms.

Exactly-once
Each event creates one effective result. It needs coordination across producer, broker, processor, state, and sink. It costs more engineering and usually more runtime overhead.

In real projects, exactly-once is rarely a single switch. It is a chain of design decisions.

A payment example

A payment provider sends payment 981 settled. Your consumer writes a paid invoice, updates customer balance, and sends a receipt.

If the consumer crashes after writing the invoice but before acknowledging the message, the message will be delivered again. With blind inserts, you might mark the invoice twice or send two receipts. With an idempotency key based on payment ID, the second attempt sees that payment 981 is already handled and changes nothing.

The message arrived twice. The business effect happened once. That is the result you want.

What to watch out for

Exactly-once claims have scope
Read the small print. The guarantee may apply only within Kafka Streams, within a Flink job, or between specific connectors.

External APIs are hard
Calling a payment API, email service, or CRM twice can create real side effects. Use idempotency keys or an outbox pattern.

Offsets are not enough
Committing a read position does not prove the target write is safe. The offset and output need to be coordinated.

Dedup tables need retention
If duplicates can arrive days later, do not expire processed event IDs after one hour.

Reconciliation still matters
Even strong guarantees deserve checks against a source of truth. Financial and inventory systems should reconcile totals, not just trust the stream.