Transactional outbox

What is a transactional outbox?

A transactional outbox writes an outgoing event into a plain database table in the same transaction as the business change that produced it. The order row and the event row commit together or not at all. A separate process reads that table afterwards and publishes the event to a message queue or broker, so other services find out what happened.

It exists to close one specific gap. You cannot atomically write to your database and publish to a broker in a single step, because they are two independent systems with no shared transaction. That is the dual-write problem. If you save the order first and publish second, a crash in between leaves the order committed with no event, and the rest of the system never hears about it. If you publish first and the transaction then rolls back, you have announced an order that does not exist.

The outbox moves the publish off that fragile path. The event is committed inside the same ACID transaction as the business data, so the two can never disagree: either both are saved or neither is. Publishing happens afterwards, and that step can be retried. The pattern turns up wherever services talk to each other through events, in event-driven architecture and in the saga pattern, where each local transaction publishes an event that triggers the next step.

How the pattern works

The write side is one transaction. Inside it you change the business data and insert a single row into an outbox table describing what happened. When the transaction commits, both rows are durable. The example below is a Postgres write using the outbox columns Debezium expects.

BEGIN;

INSERT INTO orders (id, customer_id, status)
VALUES (1001, 42, 'created');

INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (gen_random_uuid(), 'order', '1001', 'OrderCreated',
        '{"order_id": 1001, "customer_id": 42}');

COMMIT;

After the commit, a relay takes over. It finds outbox rows that have not been published yet, sends each one to the broker, and then marks the row as sent or deletes it. The relay can fail and retry without touching the business data, because its only job is to drain the table.

Polling relay versus log tailing

There are two common ways to run the relay, and they differ in how it spots a new outbox row.

A polling publisher is a worker that queries the outbox on a short interval, say every second, asking for rows that are not yet marked as published. It sends them, marks them done, and waits for the next tick. It runs on any SQL database and is easy to reason about. The costs are the extra query load on the database and the trouble of keeping strict order when several workers drain the table at once.

Log tailing avoids the polling. Instead of querying the table, a tool reads the database transaction log, the write-ahead log in PostgreSQL or the binlog in MySQL, and streams each inserted outbox row straight to the broker. This is change data capture applied to the outbox. Debezium is the usual implementation: its outbox event router treats the table as an append-only queue, routes each row to a topic using the aggregatetype column, and uses aggregateid as the message key so related events stay in order. There is no polling load and latency is lower, but you take on a CDC tool and a setup tied to your database's log format.

At-least-once delivery, not exactly-once

The outbox gives you a firm guarantee on the write side: an event is published if, and only if, its transaction committed. No event is lost, and no event escapes for work that rolled back. What it does not give you is exactly-once delivery.

The weak spot is the relay. It can send an event to the broker and then crash before it records that the row was published. On restart it sees an unpublished row and sends the same event again. So the outbox provides at-least-once delivery: every event arrives, and some arrive more than once.

That is why consumers have to be idempotent. Idempotence means that handling OrderCreated twice leaves the same result as handling it once, reached through an idempotency key, an upsert on the order id, or a table of already-seen event ids. Ordering is not automatic either. If the sequence of events matters for one customer or one order, use that id as the message key so related events stay in order rather than trusting the broker to sort them.

What to watch out for with a transactional outbox

• The relay is a separate moving part. If it stalls, business writes keep succeeding but events pile up in the outbox and consumers fall behind. Watch the table's backlog and alert on growing lag, because a silent relay looks the same as a healthy one until someone notices missing events.

• Old rows need a cleanup plan. A polling relay that marks rows as sent leaves history behind, so pick a retention window and delete or archive on a schedule. A log-tailing setup can delete a row almost immediately, since the transaction log already captured the insert.

• Not every write earns an outbox. Publishing straight to the broker is fine when a lost or duplicated event does no harm. The outbox is worth its extra parts, a table, a relay, retries and idempotent consumers, when the event carries a business fact that must not vanish, such as a paid invoice or a placed order.

• It is not event sourcing. Event sourcing is a separate pattern some teams reach for instead. An outbox only has to hand events off reliably; it does not change where your source of truth lives.