Outbox Pattern: How to Stop Losing Messages in Distributed Systems
Every backend engineer eventually hits the same wall. A user places an order. You write to the database. You publish an event to Kafka, RabbitMQ, or SQS. The network blip. The order is in your database, but the event is gone — and downstream services never knew the order happened.
This is the classic dual-write problem: two systems, one logical operation, no atomicity. The Outbox pattern is the most widely adopted solution. It works in any stack, on any database, with any message broker. Let me walk you through how to implement it properly, the trade-offs nobody tells you about, and the failure modes I have seen break production systems.
The Problem: Why Dual-Writes Always Fail
Consider this PHP code, which looks correct:
<?php
$db->beginTransaction();
$orderId = $db->insert('orders', $payload);
$db->commit();
$broker->publish('orders.created', ['order_id' => $orderId]);
Between the commit() and the publish() call, anything can go wrong. The process crashes. The network partitions. The broker is unreachable. The result: a committed order with no event in the queue. Your warehouse never ships. Your email never sends. Your analytics are wrong.
Wrapping both in a single transaction does not help — message brokers are not part of your database transaction manager. The two systems are fundamentally decoupled.
The Outbox Pattern in 30 Seconds
The Outbox pattern solves this by writing the event into the same database as your business data, inside the same transaction. A separate process (the relay) reads these outbox rows and publishes them to the broker, marking them as processed once acknowledged.
<?php
$db->beginTransaction();
$orderId = $db->insert('orders', $payload);
$db->insert('outbox', [
'event_type' => 'orders.created',
'payload' => json_encode(['order_id' => $orderId]),
'created_at' => now(),
'processed_at' => null,
]);
$db->commit();
Now your database transaction guarantees that the order and the event are persisted atomically. The publish to the broker becomes a derived, eventually-consistent step. If the relay fails, the event is still in your database and will be retried.
Three Ways to Run the Relay
Once you have the outbox table, you need a process to drain it. The three main approaches have very different operational characteristics.
1. Polling Worker
A cron job or background worker runs every few seconds, selects unprocessed rows with SELECT ... FOR UPDATE SKIP LOCKED, publishes them, and marks them as done.
<?php
$rows = $db->query("
SELECT id, event_type, payload
FROM outbox
WHERE processed_at IS NULL
ORDER BY id ASC
LIMIT 100
FOR UPDATE SKIP LOCKED
");
foreach ($rows as $row) {
$broker->publish($row['event_type'], json_decode($row['payload'], true));
$db->update('outbox', ['processed_at' => now()], ['id' => $row['id']]);
}
Pros: Simple to implement, works on any database, easy to monitor, no extra infrastructure.
Cons: Latency is bounded by the polling interval, polling load on the database at high volumes.
The SKIP LOCKED clause is critical — it lets you run multiple polling workers in parallel without double-publishing. PostgreSQL, MySQL 8+, and most modern databases support it.
2. Transaction Log Tailing (CDC)
Tools like Debezium read the database's transaction log (WAL in PostgreSQL, binlog in MySQL) and stream changes to Kafka. Your outbox table is just another table — Debezium watches for new rows and publishes them.
Pros: Sub-second latency, no polling load, exactly-once delivery guarantees when paired with Kafka transactions.
Cons: Requires running Kafka Connect (or Debezium Server), operational complexity, harder to debug.
3. LISTEN/NOTIFY (PostgreSQL only)
PostgreSQL has a built-in pub/sub mechanism. Your application issues NOTIFY outbox_channel after commit, and a long-running worker uses LISTEN to wake up and process new rows.
Pros: Low latency without CDC infrastructure, simple primitives.
Cons: Notifications are not durable — if the listener is down when you NOTIFY, the notification is lost. You still need the polling worker as a fallback.
In practice, I default to polling for most systems. The latency is acceptable, the code is auditable, and the failure modes are obvious. I reach for CDC only when latency requirements are below one second and the team has the operational maturity to run Kafka Connect reliably.
Idempotency Is Not Optional
The Outbox pattern delivers at-least-once semantics. Your consumers must be idempotent — they will see the same event more than once, eventually. Network retries, worker crashes, and broker redeliveries all cause duplicates.
The standard approach: every event gets a unique event_id (a UUID generated when the row is inserted). Consumers track the set of processed event IDs in their own database, or use the event ID as a deduplication key in their write operations.
CREATE UNIQUE INDEX outbox_event_id_idx ON outbox (event_id);
Then on the consumer side:
<?php
$db->beginTransaction();
$eventId = $message['event_id'];
$exists = $db->query("SELECT 1 FROM processed_events WHERE event_id = ?", [$eventId]);
if ($exists) {
$db->commit();
return; // Already processed, skip.
}
applySideEffect($message);
$db->insert('processed_events', ['event_id' => $eventId]);
$db->commit();
This is the boring, correct way. Do not try to be clever with two-phase commit, distributed transactions, or XA. They do not survive contact with real networks.
Cleaning Up the Outbox
Your outbox table grows forever unless you clean it. After a row is successfully published and acknowledged by the broker, you can delete it (or move it to an archive table if you need an audit trail).
Run a periodic job that deletes rows older than 7–30 days, well after the longest possible retry window has closed. Some teams keep a separate cold-storage archive for compliance.
DELETE FROM outbox
WHERE processed_at < NOW() - INTERVAL '7 days';
For high-volume systems, partition the outbox table by date and drop old partitions instead of running DELETE. DELETE on large tables in PostgreSQL generates dead tuples that need to be vacuumed, which hurts performance.
Common Failure Modes
After deploying Outbox in production across several systems, here are the failure modes I have actually seen.
Outbox table becomes the bottleneck
At very high write rates, the outbox table becomes a hot table — every business write also writes to the outbox. Index it carefully (the processed_at index is the one that matters for the polling worker) and partition it by date if you are above ~10k events per second.
Poison messages block the queue
If a single event consistently fails to publish (a schema mismatch, a permanently broken downstream), the relay will retry it forever, blocking newer events. Always have a dead-letter strategy: after N retries, move the row to a outbox_failed table for human review.
Clock skew on processed_at
Do not rely on application timestamps for ordering or deduplication. Use the database's id column (a monotonically increasing sequence) for ordering. The relay processes rows in id order, not by timestamp.
Testing the failure path
Most teams test the happy path and stop there. To actually verify your Outbox implementation, kill the broker mid-test. Stop the relay. Roll back the database. You need to confirm that the system recovers gracefully and that no events are lost or duplicated beyond what idempotent consumers can handle.
When Not to Use the Outbox
The Outbox is not free. It adds a database write, a polling process (or CDC infrastructure), and operational complexity. Skip it when:
- You do not need reliable event delivery — the event is purely informational and can be lost
- You are publishing fewer than a few events per second and a single, well-tested publish call is enough
- Your broker supports exactly-once semantics natively and you can use it (most do not)
For everything else, especially in systems that handle money, inventory, or anything where a missed event causes user-visible harm, the Outbox pattern is the right default. It is one of those rare patterns that is both conceptually simple and operationally robust — and it has been battle-tested in production at every large company you have ever used.
Build it once, get it right, and sleep well knowing your events will arrive exactly when they should.