Inbox Pattern: Stop Double-Processing Messages in Distributed Systems
Every distributed system has a consumer. A Lambda function reads from SQS. A PHP worker pulls from RabbitMQ. A Go routine subscribes to Kafka. The consumer receives a message, processes it, and acknowledges it. But between the processing and the acknowledgement, something goes wrong. The process crashes. The network partitions. The broker times out.
The result: your system processed the message, but the broker does not know. The message is redelivered. Your consumer processes it again. This time, the database has a duplicate. The email is sent twice. The credit card is charged twice.
This is the dual-write problem on the consumer side. The Inbox pattern solves it by inverting the order of operations: write to your database first, then process the message. The database write becomes the source of truth. The broker acknowledgement becomes a secondary concern.
In this article, I will walk you through the Inbox pattern in detail — how to implement it in PHP 8.5, how Amazon SQS implements the same pattern natively, and the failure modes you must design for from day one.
The Consumer-Side Problem
Here is the classic failure pattern on the consumer side. Consider this PHP code that looks correct:
<?php
$message = $broker->consume('orders.created');
$orderId = $message['order_id'];
$db->beginTransaction();
$db->insert('orders', ['id' => $orderId, 'status' => 'received']);
$db->commit();
$broker->acknowledge($message);
Between commit() and acknowledge(), the consumer crashes. The broker does not know the message was processed. It waits for the visibility timeout, then redelivers the message. The consumer processes the same order twice.
The problem is ordering: you process the message first (which succeeds), then acknowledge it (which fails). The broker never sees the acknowledgement, so it redelivers. Your database has the order, but your consumer does not know it was already processed.
The solution is the Inbox pattern: write to a deduplication table before you process the message.
The Inbox Pattern in 30 Seconds
The Inbox pattern works by maintaining a deduplication log in your database. Every message that arrives is recorded in an inbox table before any business logic runs. The event_id (a unique identifier for each message) is the deduplication key.
<?php
$message = $broker->consume('orders.created');
$db->beginTransaction();
// Check the inbox for this event_id
$exists = $db->query(
'SELECT 1 FROM inbox WHERE event_id = ?',
[$message['event_id']]
);
if ($exists) {
// Already processed — skip.
$db->commit();
$broker->acknowledge($message);
return;
}
// Write to inbox first
$db->insert('inbox', [
'event_id' => $message['event_id'],
'event_type' => 'orders.created',
'payload' => json_encode($message['payload']),
'status' => 'received',
'received_at' => now(),
]);
// Then apply business logic
$orderId = $db->insert('orders', $message['payload']);
// Mark as processed
$db->update('inbox', ['status' => 'processed'], ['event_id' => $message['event_id']]);
$db->commit();
$broker->acknowledge($message);
This is the Inbox pattern. The inbox table acts as a deduplication log. If the consumer crashes after commit() but before acknowledge(), the broker redelivers the message, and the consumer finds the event_id already in the inbox and skips it.
The key insight: write to your database first, then talk to the broker. This inverts the order of operations so that the database write is always successful, even if the broker acknowledgement fails.
The Inbox Table Schema
The inbox table is simple but requires careful indexing:
CREATE TABLE inbox (
id BIGSERIAL PRIMARY KEY,
event_id UUID NOT NULL UNIQUE,
event_type VARCHAR(255) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'received',
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed_at TIMESTAMPTZ,
error TEXT
);
CREATE INDEX inbox_status_idx ON inbox (status);
CREATE INDEX inbox_received_at_idx ON inbox (received_at);
The UNIQUE constraint on event_id is the deduplication mechanism. If the same message arrives twice, the INSERT fails with a unique constraint violation, and the consumer knows it was already processed.
The status column tracks the lifecycle of each message: received (the message arrived), processing (the message is being processed), processed (the message was successfully processed), failed (the message could not be processed).
In PHP 8.5, you can model this as a readonly class:
<?php
readonly class InboxMessage
{
public function __construct(
public string $eventId,
public string $eventType,
public array $payload,
public string $status,
public \DateTimeImmutable $receivedAt,
) {}
public static function fromRow(array $row): self
{
return new self(
eventId: $row['event_id'],
eventType: $row['event_type'],
payload: json_decode($row['payload'], true),
status: $row['status'],
receivedAt: new \DateTimeImmutable($row['received_at']),
);
}
}
Using readonly classes (available since PHP 8.1, fully integrated in 8.5) ensures that once an inbox message is loaded, it cannot be mutated. This is a strong contract — your consumer code cannot accidentally modify a message it has already processed.
The Inbox in PHP 8.5
Let me show you a complete Inbox implementation using PHP 8.5 features. This is the production code I use today.
1. Event Types as an Enum
PHP 8.1 introduced enum types. In 8.5, they are fully integrated into the type system and work seamlessly with match expressions. Use them for event types — they are safer than strings and provide IDE autocompletion:
<?php
enum EventType: string
{
case OrderCreated = 'orders.created';
case OrderShipped = 'orders.shipped';
case PaymentReceived = 'payments.received';
case InvoiceGenerated = 'invoices.generated';
public function queueName(): string
{
return match ($this) {
self::OrderCreated => 'orders',
self::OrderShipped => 'shipping',
self::PaymentReceived => 'payments',
self::InvoiceGenerated => 'invoicing',
};
}
}
2. The Inbox Message as a Readonly Value Object
<?php
readonly class InboxMessage
{
public function __construct(
public string $eventId,
public string $eventType,
public array $payload,
public string $status,
public \DateTimeImmutable $receivedAt,
) {}
public static function fromRow(array $row): self
{
return new self(
eventId: $row['event_id'],
eventType: $row['event_type'],
payload: json_decode($row['payload'], true),
status: $row['status'],
receivedAt: new \DateTimeImmutable($row['received_at']),
);
}
}
3. The Inbox Repository
<?php
class InboxRepository
{
public function __construct(
private readonly \PDO $db,
) {}
public function exists(string $eventId): bool
{
$stmt = $this->db->prepare('SELECT 1 FROM inbox WHERE event_id = ?');
$stmt->execute([$eventId]);
return (bool) $stmt->fetchColumn();
}
public function insert(EventType $eventType, array $payload, string $eventId): void
{
$this->db->beginTransaction();
$stmt = $this->db->prepare('
INSERT INTO inbox (event_id, event_type, payload, status, received_at)
VALUES (?, ?, ?, ?, NOW())
');
$stmt->execute([
$eventId,
$eventType->value,
json_encode($payload),
]);
$this->db->commit();
}
public function findUnprocessed(int $limit = 100): array
{
$rows = $this->db->query("
SELECT id, event_id, event_type, payload, status, received_at
FROM inbox
WHERE status = 'received'
ORDER BY id ASC
LIMIT {$limit}
FOR UPDATE SKIP LOCKED
");
return array_map(
fn (array $row) => new InboxMessage(
eventId: $row['event_id'],
eventType: $row['event_type'],
payload: json_decode($row['payload'], true),
status: $row['status'],
receivedAt: new \DateTimeImmutable($row['received_at']),
),
$rows->fetchAll(\PDO::FETCH_ASSOC),
);
}
public function markProcessed(string $eventId): void
{
$this->db->prepare('
UPDATE inbox
SET status = 'processed', processed_at = NOW()
WHERE event_id = ?
')->execute([$eventId]);
}
}
Notice the array_map with an arrow function — PHP 8.5 makes this pattern clean and type-safe. The readonly class guarantees that once an inbox message is loaded, the consumer cannot accidentally modify it.
4. The Inbox Consumer Using Fibers
PHP 8.5 supports Fibers (cooperative multitasking). For an inbox consumer, you can use Fibers to process multiple messages concurrently without the overhead of full process forking:
<?php
class InboxConsumer
{
public function __construct(
private readonly InboxRepository $inbox,
private readonly MessageBroker $broker,
) {}
public function run(): void
{
while (true) {
$message = $this->broker->consume('orders.created');
if (!$message) {
usleep(100_000); // 100ms
continue;
}
$fiber = new Fiber(function () use ($message): void {
$this->process($message);
});
$fiber->start();
$fiber->resume();
}
}
private function process(array $message): void
{
try {
$eventId = $message['event_id'];
if ($this->inbox->exists($eventId)) {
// Already processed — skip
$this->broker->acknowledge($message);
return;
}
$this->inbox->insert(
EventType::from($message['event_type']),
$message['payload'],
$eventId,
);
applyBusinessLogic($message);
$this->inbox->markProcessed($eventId);
$this->broker->acknowledge($message);
} catch (\Throwable $e) {
// Log and continue — the inbox ensures at-most-once
error_log("Failed to process: {$e->getMessage()}");
}
}
}
Fibers are lightweight — they do not create new OS threads or processes. They are ideal for I/O-bound work like consuming from a message broker. Each fiber handles one message, and if it yields (e.g., waiting for a broker acknowledgement), the main loop continues.
In practice, I use Fibers for the inbox consumer when I need concurrent processing without the complexity of pcntl_fork or process managers. For most systems, a simple foreach loop is sufficient — Fibers are for when you need to process multiple messages in parallel within a single process.
Amazon's Inbox Architecture
Amazon has built its entire messaging infrastructure around the Inbox pattern. Every AWS service that delivers messages — SQS, SNS, EventBridge — implements the inbox pattern at its core.
SQS as an Inbox
Amazon SQS is the most widely used implementation of the Inbox pattern. When you send a message to SQS, the message is persisted to SQS's own internal database before it is made available to consumers. SQS guarantees at-least-once delivery — your consumer will see the same message more than once if the consumer fails to acknowledge it.
SQS implements the inbox pattern through its visibility timeout mechanism:
- A consumer receives a message from the queue
- SQS marks the message as invisible for the duration of the visibility timeout
- The consumer processes the message
- If the consumer succeeds, it deletes the message from the queue
- If the consumer fails (or the timeout expires), SQS makes the message visible again for another consumer
This is the inbox pattern in action. The message is written to SQS's storage first, and the consumer's acknowledgement is a DELETE operation. If the consumer crashes, the message returns to the queue — exactly like the inbox table's UNIQUE constraint.
Amazon takes this further with SQS FIFO queues, which provide exactly-once delivery within a message group through a deduplication ID. This is the same event_id deduplication I showed you in the inbox pattern, but implemented at the broker level:
{
"MessageDeduplicationId": "evt_abc123",
"MessageGroupId": "orders",
"Body": "{\"order_id\": 42}"
}
SQS FIFO queues are the only broker that provides native exactly-once semantics. Every other broker (Kafka, RabbitMQ, Pulsar) delivers at-least-once and requires the consumer to implement the inbox pattern.
Lambda as an Idempotent Consumer
AWS Lambda functions that consume from SQS are inherently idempotent — Lambda processes each message in the batch, and if the function fails, SQS makes the messages visible again for retry. But Lambda does not provide idempotency out of the box — you must implement the inbox pattern in your Lambda function.
The standard approach for Lambda + SQS:
<?php
function handler(array $event): void
{
foreach ($event['Records'] as $record) {
$eventId = $record['messageId'];
$body = json_decode($record['body'], true);
$db->beginTransaction();
$exists = $db->query(
'SELECT 1 FROM inbox WHERE event_id = ?',
[$eventId]
);
if ($exists) {
$db->commit();
continue;
}
$db->insert('inbox', [
'event_id' => $eventId,
'event_type' => $body['event_type'],
'payload' => json_encode($body),
]);
$db->commit();
}
}
Lambda + SQS is the most operationally simple form of the inbox pattern. You do not need to manage the visibility timeout, the deduplication logic, or the acknowledgement — SQS handles it all.
DynamoDB Streams as an Inbox
DynamoDB Streams is another implementation of the inbox pattern. Every write to a DynamoDB table is captured in the stream, which is then consumed by Lambda functions or Kinesis consumers. The stream is the inbox — it records every change, and the consumer processes them in order.
{
"Records": [{
"eventID": "1",
"eventName": "INSERT",
"eventSource": "aws:dynamodb",
"dynamodb": {
"Keys": {"order_id": "42"},
"NewImage": {"order_id": "42", "status": "created"}
}
}]
}
Lambda functions consume these stream records and process them. If the function fails, the stream retries the record. This is the inbox pattern built into the database — the most operationally simple form.
The Complete Amazon Pipeline
Putting it all together, Amazon's recommended architecture for reliable message consumption looks like this:
SQS → Lambda → DynamoDB (Inbox) → Consumer
- SQS receives the message and stores it (the inbox)
- Lambda reads from SQS and processes the message
- Lambda writes to DynamoDB (the inbox table)
- The consumer reads from DynamoDB Streams and processes the message
Every component is a managed service. You do not write any deduplication logic, any SELECT 1 FROM inbox checks, or any retry logic. AWS handles it all.
The trade-off is cost. DynamoDB Streams + Lambda + SQS is more expensive than a single PostgreSQL table with a cron job. But for systems where reliability is more important than cost, this is the gold standard.
Idempotency Is Not Optional
The Inbox pattern delivers at-least-once semantics. Your consumers must be idempotent — they will see the same message more than once, eventually. Network retries, worker crashes, and broker redeliveries all cause duplicates.
The standard approach: every message gets a unique event_id (a UUID generated when the message is received). Consumers check the set of processed event IDs in their own database before applying business logic.
CREATE UNIQUE INDEX inbox_event_id_idx ON inbox (event_id);
The UNIQUE constraint is the deduplication mechanism. If the same message arrives twice, the INSERT fails with a unique constraint violation, and the consumer knows it was already processed.
For operations that are not inherently idempotent (like sending an email or charging a credit card), you must add explicit idempotency checks. The inbox table provides the audit trail for this — you can see exactly which messages were processed and which were skipped.
The boring, correct way to handle this:
<?php
$db->beginTransaction();
$exists = $db->query("SELECT 1 FROM inbox WHERE event_id = ?", [$eventId]);
if ($exists) {
$db->commit();
return; // Already processed, skip.
}
applyBusinessLogic($message);
$db->insert('inbox', ['event_id' => $eventId]);
$db->commit();
Do not try to be clever with two-phase commit, distributed transactions, or XA. They do not survive contact with real networks.
Dead-Letter Queues
The inbox pattern needs dead-letter handling. Messages that cannot be processed — because the business logic fails, the database is unreachable, or the message is malformed — must go somewhere.
In SQS, this is the Redrive Policy. After N receives, the message is moved to a dead-letter queue. In the inbox pattern, this is the same concept but implemented in your database:
<?php
$db->beginTransaction();
$db->insert('inbox_failed', [
'event_id' => $eventId,
'event_type' => 'orders.created',
'payload' => json_encode($payload),
'error' => $error,
'failed_at' => now(),
]);
$db->update('inbox', ['status' => 'failed', 'error' => $error], ['event_id' => $eventId]);
$db->commit();
The dead-letter table preserves the message for debugging. A separate process can re-apply failed messages after a fix is deployed.
For high-volume systems, partition the inbox table by date and drop old partitions instead of running DELETE. DELETE on large tables in PostgreSQL generates dead tuples that need to be vacuumed, which hurts performance.
Ordering Guarantees
The inbox pattern preserves message order by default. The consumer processes messages in id order (ascending). This means messages are processed in the order they were received.
But there is a catch: if you have multiple consumers, the order is not guaranteed across processes. Each consumer processes its own batch of messages, and the order between batches depends on which consumer runs first.
For strict ordering, use a single-threaded consumer (one process, one loop) or use partitioned tables where each partition has its own consumer. SQS FIFO queues solve this with message groups — messages in the same group are delivered in order to a single consumer.
In the database, you can achieve the same by partitioning the inbox by event_type and running one consumer per partition:
CREATE TABLE inbox_orders (
CHECK (event_type = 'orders.created')
) INHERITS (inbox);
CREATE TABLE inbox_payments (
CHECK (event_type = 'payments.received')
) INHERITS (inbox);
Each partition has its own consumer, and the consumers process messages in isolation. This gives you per-event-type ordering without a single bottleneck.
When Not to Use the Inbox
The Inbox is not free. It adds a database write, a deduplication check, and operational complexity. Skip it when:
- You do not need reliable message consumption — the message is purely informational and can be lost
- You are processing fewer than a few messages per second and a single, well-tested consumer is enough
- Your broker supports exactly-once semantics natively and you can use it (most do not)
For everything else, especially in systems that handle money, inventory, or anything where a duplicated message causes user-visible harm, the Inbox pattern is the right default. It is one of those rare patterns that is both conceptually simple and operationally robust — and it has been battle-tested in production at every large company you have ever used.
Build it once, get it right, and sleep well knowing your messages will be processed exactly once.