00:00

Help Us Improve

We're sorry this tutorial wasn't helpful. Please tell us what we can do better.

Transactional Outbox Pattern: Save Data and Publish Events Reliably

Last updated:

The transactional outbox pattern makes a service save its data and record the event it wants to publish in the same local database transaction, by writing the event into an "outbox" table. A separate relay process then reads the outbox and publishes the events to a message broker such as Kafka. Either both the data and the event are stored, or neither is, so other services never miss an event and never hear about data that was rolled back.

It solves one of the most common hidden bugs in event-driven microservices: the dual write.

1. The Dual Write Problem

Imagine a theatre booking service. When a customer books seat C7, the service must do two things: save the booking in its own database, and publish a BookingPlaced event so the payment and notification services can react.

The obvious code does both, one after the other:

  1. Insert the booking and commit.
  2. Publish the event to Kafka.

This is called a dual write: one action writes to two different systems. They can't share a transaction, so there is always a moment between the two steps where something can go wrong:

  • The broker is down or slow when step 2 runs. The booking exists, but no event is ever sent, so the customer is never charged and never gets a ticket.
  • The service crashes between the steps, for example because the pod is restarted during a deployment. Same result.
  • You swap the order, publishing first and committing second. Now a failed commit leaves an event about a booking that doesn't exist, and the payment service charges for a ghost booking.

Retrying doesn't fully help either, because the service that should retry may be the one that crashed. Distributed transactions (two-phase commit) could in theory cover both systems, but Kafka doesn't support XA transactions, and two-phase commit is slow and fragile anyway.

2. Key Terms

  • Local transaction: an ordinary database transaction in one database. It is atomic: all its changes are saved, or none are.
  • Outbox table: a table in the service's own database where events wait to be published. Each row holds an id, the event type, the related entity id (aggregate id) and the payload.
  • Relay (message relay): the process that reads new outbox rows and publishes them to the broker.
  • Polling publisher: a relay that queries the outbox table every few hundred milliseconds.
  • Change data capture (CDC): reading committed changes from the database's transaction log instead of querying tables. Debezium is the most common CDC tool for Java teams.
  • At-least-once delivery: every event is delivered, but some may be delivered more than once.
  • Idempotent consumer: a consumer that gives the same result whether it handles an event once or several times.

3. How the Transactional Outbox Pattern Works

Transactional outbox flow1: the booking service writes the booking and an outbox row in one local transaction in its own database. 2: a relay, either a poller or change data capture, reads new outbox rows. 3: it publishes them to a Kafka topic using the booking id as the key. 4: it marks the rows as sent. 5: the payment and notification services consume the events and skip duplicates. Booking serviceplaceBooking() bookings + outbox service's own database Relaypoller or CDC Kafka topickey = booking id Payment service Notification 1. one localtransaction 2. read new rows 4. mark sent 3. publish 5. consume,skip duplicates
The service only ever writes to its own database; the relay turns committed outbox rows into broker messages, and consumers must tolerate the occasional duplicate.
  1. The service writes the booking and the outbox row in one transaction. It never talks to Kafka directly.
  2. The relay finds new rows, either by polling the table or by reading the transaction log.
  3. It publishes them, using the booking id as the message key so all events for one booking go to the same Kafka partition and stay in order.
  4. It records progress: a polling relay marks rows as sent or deletes them; a CDC relay stores its position in the log.
  5. Consumers process the events, ignoring any they have already handled.
What happens internally: the relay's own work is also a dual write: it publishes to Kafka and then updates the outbox. If it crashes between the two, it will publish the same row again after a restart. So the outbox pattern guarantees at-least-once delivery, never exactly-once. Kafka's idempotent producer, enabled by default since Kafka 3.0, removes duplicates caused by the producer's own network retries, but it can't know that a restarted relay is resending an old row. That's why every consumer of outbox events must be idempotent.

4. Code Examples

These JDK-only Java 21 programs model the database, broker and relay in memory, so each failure can be shown step by step.

4.1 Example 1: Dual Write vs Outbox

The database class gives real transaction behaviour: changes become visible only on commit().

DualWriteDemo.java 
import java.util.ArrayList;
import java.util.List;

public class DualWriteDemo {

    // A tiny database with real transaction semantics: changes become visible only on commit
    static class Database {
        final List<String> orders = new ArrayList<>(), outbox = new ArrayList<>();
        class Tx {
            final List<String> newOrders = new ArrayList<>(), newOutbox = new ArrayList<>();
            void insertOrder(String o) { newOrders.add(o); }
            void insertOutbox(String e) { newOutbox.add(e); }
            void commit() { orders.addAll(newOrders); outbox.addAll(newOutbox); } // all or nothing
        }
        Tx begin() { return new Tx(); }
    }

    static class Broker {
        final List<String> topic = new ArrayList<>();
        boolean down;
        void publish(String event) {
            if (down) throw new IllegalStateException("broker unreachable");
            topic.add(event);
        }
    }

    // Dual write: two separate systems, two separate steps
    static void placeOrderDualWrite(Database db, Broker broker, String id, boolean crashBetween) {
        Database.Tx tx = db.begin();
        tx.insertOrder(id);
        tx.commit();                                         // step 1 succeeds
        if (crashBetween) throw new IllegalStateException("pod killed");
        broker.publish("OrderPlaced " + id);                 // step 2 may never happen
    }

    // Outbox: the event is just another row in the same local transaction
    static void placeOrderWithOutbox(Database db, String id, boolean crashBeforeCommit) {
        Database.Tx tx = db.begin();
        tx.insertOrder(id);
        tx.insertOutbox("OrderPlaced " + id);
        if (crashBeforeCommit) throw new IllegalStateException("pod killed"); // nothing is committed
        tx.commit();
    }

    static void report(String title, Database db, Broker broker) {
        if (broker != null) System.out.printf("%-34s orders=%s  broker=%s%n", title, db.orders, broker.topic);
        else System.out.printf("%-34s orders=%s  outbox=%s%n", title, db.orders, db.outbox);
    }

    public static void main(String[] args) {
        Database db1 = new Database(); Broker b1 = new Broker(); b1.down = true;
        try { placeOrderDualWrite(db1, b1, "BK-101", false); } catch (IllegalStateException e) { /* logged */ }
        report("Dual write, broker down:", db1, b1);

        Database db2 = new Database(); Broker b2 = new Broker();
        try { placeOrderDualWrite(db2, b2, "BK-102", true); } catch (IllegalStateException e) { /* logged */ }
        report("Dual write, crash between steps:", db2, b2);

        Database db3 = new Database();
        try { placeOrderWithOutbox(db3, "BK-103", true); } catch (IllegalStateException e) { /* logged */ }
        report("Outbox, crash before commit:", db3, null);

        Database db4 = new Database();
        placeOrderWithOutbox(db4, "BK-104", false);
        report("Outbox, normal:", db4, null);
    }
}
Output 
Dual write, broker down:           orders=[BK-101]  broker=[]
Dual write, crash between steps:   orders=[BK-102]  broker=[]
Outbox, crash before commit:       orders=[]  outbox=[]
Outbox, normal:                    orders=[BK-104]  outbox=[OrderPlaced BK-104]

With dual writes, both failures leave a booking with no event, and nothing will ever notice. With the outbox, a crash before commit leaves nothing behind, so the customer simply sees an error and can try again. A successful commit always contains both the booking and its event.

4.2 Example 2: The Relay Crashes, and the Consumer Handles the Duplicate

The relay publishes rows in order and marks each one as sent. Here it crashes right after publishing row 2.

OutboxRelayDemo.java 
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class OutboxRelayDemo {

    // One row in the outbox table
    static class OutboxRow {
        final long id; final String payload; boolean sent;
        OutboxRow(long id, String payload) { this.id = id; this.payload = payload; }
    }

    static final List<OutboxRow> outbox = new ArrayList<>();
    static final List<String> topic = new ArrayList<>();   // what the broker received

    // The relay: publish unsent rows in id order, then mark them sent
    static void relayRun(boolean crashAfterPublishingRow2) {
        for (OutboxRow row : outbox) {
            if (row.sent) continue;
            topic.add(row.id + ":" + row.payload);          // publish (the broker acknowledged it)
            if (crashAfterPublishingRow2 && row.id == 2) {
                System.out.println("  relay crashed after publishing row 2, before marking it sent");
                return;
            }
            row.sent = true;                                // UPDATE outbox SET sent = true WHERE id = ?
        }
    }

    // An idempotent consumer remembers which event ids it has already handled
    static int applyAll(List<String> messages, Set<Long> processed) {
        int applied = 0;
        for (String m : messages) {
            long eventId = Long.parseLong(m.substring(0, m.indexOf(':')));
            if (processed.add(eventId)) applied++;           // add() returns false for a duplicate
        }
        return applied;
    }

    public static void main(String[] args) {
        outbox.add(new OutboxRow(1, "TicketReserved seat=C7"));
        outbox.add(new OutboxRow(2, "TicketReserved seat=C8"));
        outbox.add(new OutboxRow(3, "PaymentRequested 64.00"));

        System.out.println("First relay run:");
        relayRun(true);
        System.out.println("Second relay run (after restart):");
        relayRun(false);

        System.out.println("Messages on the topic: " + topic);
        System.out.println("Consumer without dedup applies: " + topic.size() + " events");
        System.out.println("Idempotent consumer applies:    " + applyAll(topic, new HashSet<>()) + " events");
    }
}
Output 
First relay run:
  relay crashed after publishing row 2, before marking it sent
Second relay run (after restart):
Messages on the topic: [1:TicketReserved seat=C7, 2:TicketReserved seat=C8, 2:TicketReserved seat=C8, 3:PaymentRequested 64.00]
Consumer without dedup applies: 4 events
Idempotent consumer applies:    3 events

After the restart, row 2 is still unsent, so it is published a second time. A consumer without deduplication would reserve seat C8 twice. The idempotent consumer stores the outbox row id, which travels with the event as a header or field, and skips the repeat. In a real service, store processed ids in the consumer's own database, in the same transaction as the work the event causes.

4.3 Example 3: Scaling the Relay Without Breaking Order

One relay may not keep up with a busy service, but running several needs care.

OutboxOrderingDemo.java 
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class OutboxOrderingDemo {

    record Row(long id, String orderId, String event) {}

    static List<Row> outbox() {
        return List.of(new Row(1, "BK-7", "Created"), new Row(2, "BK-8", "Created"),
                new Row(3, "BK-7", "Paid"), new Row(4, "BK-8", "Cancelled"),
                new Row(5, "BK-7", "Shipped"), new Row(6, "BK-8", "Refunded"));
    }

    // Is every order's history in the right order on the topic?
    static boolean inOrder(List<Row> published) {
        Map<String, Long> lastId = new HashMap<>();
        for (Row r : published) {
            Long prev = lastId.put(r.orderId(), r.id());
            if (prev != null && prev > r.id()) return false;
        }
        return true;
    }

    public static void main(String[] args) {
        // Naive: two relay instances poll the same unsent rows at the same moment
        List<Row> naive = new ArrayList<>();
        List<Row> seenByA = outbox(), seenByB = outbox();    // neither has marked anything sent yet
        naive.addAll(seenByA);
        naive.addAll(seenByB);
        System.out.println("Naive, 2 relays:      published " + naive.size() + " messages for 6 rows");

        // Round-robin batches across 2 relays: no duplicates, but relay B happens to finish first
        List<Row> rows = outbox();
        List<Row> batchA = rows.subList(0, 3), batchB = rows.subList(3, 6);
        List<Row> roundRobin = new ArrayList<>(batchB);
        roundRobin.addAll(batchA);
        System.out.println("Split by batch:       published " + roundRobin.size() + ", per-order order kept: " + inOrder(roundRobin));

        // Partition by order id: each relay owns some orders, so one order's rows never race
        List<Row> relay0 = new ArrayList<>(), relay1 = new ArrayList<>();
        for (Row r : outbox()) (Math.floorMod(r.orderId().hashCode(), 2) == 0 ? relay0 : relay1).add(r);
        List<Row> partitioned = new ArrayList<>(relay1);     // even if relay 1 finishes first
        partitioned.addAll(relay0);
        System.out.println("Partitioned by order: published " + partitioned.size() + ", per-order order kept: " + inOrder(partitioned));
        System.out.println("  relay 0 owns " + relay0.stream().map(Row::orderId).distinct().toList()
                + ", relay 1 owns " + relay1.stream().map(Row::orderId).distinct().toList());
    }
}
Output 
Naive, 2 relays:      published 12 messages for 6 rows
Split by batch:       published 6, per-order order kept: false
Partitioned by order: published 6, per-order order kept: true
  relay 0 owns [BK-8], relay 1 owns [BK-7]

Two relays reading the same unsent rows publish everything twice. Splitting rows into batches removes the duplicates, but "Cancelled" can now arrive before "Created" for the same booking. Assigning each booking to one relay keeps each booking's history in order, whichever relay finishes first. In SQL, relays usually claim rows with SELECT ... FOR UPDATE SKIP LOCKED (PostgreSQL 9.5+, MySQL 8.0+), which lets each relay lock different rows without waiting for the others.

5. Polling Relay vs Change Data Capture

Polling relayChange data capture (CDC)
How it finds new eventsQueries the outbox table repeatedlyReads the database's transaction log (PostgreSQL WAL, MySQL binlog)
Extra infrastructureNone, runs inside your serviceDebezium, usually with Kafka Connect
LatencyPolling interval, often 100 ms to 1 sUsually well under a second
Load on the databaseConstant queries, even when idleLow, reads the log
OrderingOrdered by row id; needs care with several relaysFollows commit order in the log
CleanupMark rows sent, delete them laterRows can be deleted right after insert; the log already has them
Best forSmall to medium volume, simple setupsHigh volume, many services, teams already running Kafka Connect

Debezium ships an Outbox Event Router, a transformation that reads outbox rows from the log and routes each one to a topic based on its aggregate type, using the aggregate id as the message key.

6. Common Mistakes and Pitfalls

1. Writing the outbox row in a separate transaction. If the outbox insert happens in a new transaction, for example in a different Spring bean with REQUIRES_NEW, you're back to a dual write. The booking and the outbox row must commit together.

2. Assuming exactly-once delivery. As Example 2 shows, duplicates will happen. Consumers need deduplication by event id, or operations that are naturally safe to repeat, such as "set status to PAID".

3. Letting the outbox grow forever. A table with millions of sent rows makes polling slower every day. Delete or archive sent rows on a schedule, or partition the table by date.

4. Using a broker-generated id for deduplication. Kafka assigns a new offset to a re-published message, so consumers can't use offsets to spot repeats. Use the outbox row's own id or a UUID created when the row was written.

5. Ignoring a stuck relay. If the relay stops, bookings still succeed but no events go out, and other services quietly fall behind. Monitor the age of the oldest unsent row and alert when it grows.

6. Putting huge payloads in the outbox. Copying a 2 MB document into every event bloats both the table and the topic. Publish the essential facts and an id, and let consumers fetch details if they really need them.

7. Best Practices

  • Write the business change and the outbox row in one local transaction, in the same method.
  • Give every event a unique id created when the row is written, and make consumers idempotent.
  • Use the aggregate id as the message key so each entity's events stay in order.
  • Design events for consumers: stable field names, a version number and only the data they need.
  • Monitor relay lag with the age of the oldest unsent row.
  • Clean up sent rows regularly.
  • Prefer CDC at high volume, and a simple polling relay when you are starting out.
  • Consider ready-made support: Spring Modulith's event publication registry, Axon or Eventuate Tram provide outbox-style publishing out of the box.

8. Summary

  • A dual write to a database and a broker can't be made reliable with retries alone.
  • The transactional outbox pattern stores the event as a row in the same local transaction as the data.
  • A relay publishes outbox rows to the broker, by polling or with change data capture.
  • Delivery is at-least-once, so consumers must handle duplicates.
  • Use the aggregate id as the key, and partition relays by it, to keep events in order.
  • Clean up the outbox and monitor how far the relay is behind.

If your services save data and then call kafkaTemplate.send(), you have a dual write today. Adding the transactional outbox pattern is a small change that removes a whole class of lost and phantom events.