Why Message Delivery Still Fails
You've just deployed a feature that triggers an email confirmation after a user places an order.
The user gets charged.
The order shows up in the database.
But the email never arrives.
The logs show that the service meant to publish an event... but never did.
You sigh, knowing this isn't your first lost message and won't be the last unless you fix your architecture.
The problem? We often focus only on sending messages reliably (Outbox), but forget to handle receiving messages reliably (Inbox).
Both sides matter. And that's why every Outbox needs an Inbox.
Want to master real-world software engineering techniques? CodeCrafters offers hands-on challenges that help you build production-ready systems from the ground up.
Write your own Redis, Git, or Docker to understand how the best tools work under the hood.
Develop resilience and scalability by building real-world software components.
Level up your system design skills with practical exercises inspired by industry challenges.
Sign up and get 40% off if you upgrade.
Thank you to our sponsors who keep this newsletter free!
Understanding Delivery Guarantees
Before diving deeper, it's worth clarifying the three major delivery guarantees in distributed systems:
At-most-once: Messages are delivered at most once. Some may be lost. No duplicates.
At-least-once: Messages are retried until acknowledged. No loss, but duplicates are possible.
Exactly-once: Messages are delivered once and only once. No loss, no duplicates. Very hard to achieve reliably across distributed components.
Most production systems settle on at-least-once + idempotency, as it balances complexity and reliability.
The Outbox and Inbox patterns are what make this delivery guarantee practical.
The Outbox Pattern: Reliable Send
The Outbox pattern ensures that domain events are sent only if the related business operation is committed.
How it works:
1. Begin DB Transaction
2. Save order to `orders` table
3. Insert event to `outbox` table
4. Commit transaction
A background process then polls the outbox
table and sends messages to the message broker.
This ensures:
No message is sent if the DB write fails
No DB write is done without queuing the message
But here's the catch: It only covers your half of the contract.
The Inbox Pattern: Reliable Receive
Now, imagine your order service emits an event, but the inventory service crashes just before processing it. Or processes it twice due to retries.
That's where the Inbox pattern comes in.
How it works:
1. Receive message from broker
2. Start DB transaction
3. Check if `message_id` exists in `inbox` table
4. If not, insert into `inbox`, then process
5. Commit
This ensures:
Messages are processed only once (idempotency)
Messages are never lost due to crashes
The combination of Outbox + Inbox gives you at-least-once delivery with idempotent processing, the practical sweet spot.
Putting All Together
The following sequence diagram ties everything together. It shows the entire lifecycle of a message from the moment a user places an order, through reliable event emission via the Outbox pattern, to safe and idempotent processing on the receiver's side via the Inbox pattern.
After the event is saved to the inbox table, the same service that inserted it (e.g., InventoryService
) proceeds to execute the corresponding business logic, like updating inventory stock or initiating follow-up actions. This execution is part of the same transaction that saves the inbox entry to ensure consistency.
Step-by-step breakdown:
The user places an order.
OrderService
wraps the operation in a transaction, writing to bothorders
andoutbox_events
tables.A background dispatcher picks up new events from the Outbox and publishes them to a message broker.
InventoryService
receives the event, checks its inbox table to avoid duplicates, and processes it if not alr’t already been handled.
This flow ensures that no message is lost and each message is processed exactly once, achieving at-least-once semantics with idempotency.
Here's how the full flow looks as a sequence diagram:
Real-World Example
Stripe uses a variation of this pattern. Events are stored and replayable. Every event includes an idempotency_key
and event delivery is tracked for success.
Many teams roll their own simpler version using:
Postgres tables for outbox/inbox
Kafka or RabbitMQ for transport
A cron job or queue-based worker for dispatch
Trade-offs and Failure Modes
Delayed processing: Outbox dispatcher or Inbox consumer can lag
Duplication: At-least-once delivery means processing logic must be idempotent
Storage growth: Outbox/Inbox tables can grow; need purging/archiving strategy
Delivery not immediate: Adds latency due to polling or background workers
Despite these, it's the most robust design for mission-critical messaging between services.
Anti-Patterns and Pitfalls
While the Outbox and Inbox patterns are powerful for reliable messaging, several pitfalls can undermine their effectiveness if not addressed. These patterns should not be treated as fire-and-forget solutions; they require thoughtful integration into your system’s lifecycle, operations, and failure management strategies:
Unbounded Table Growth: Without a strategy to purge or archive old entries, outbox and inbox tables can grow indefinitely, impacting performance and storage costs. You need to implement regular cleanup or archival processes.
Database as Bottleneck: Relying heavily on the database for both business and messaging operations can reduce system elasticity. If high throughput is a priority, you must consider alternatives or optimizations.
Immediate Delivery Expectations: These patterns introduce latency, as events are processed asynchronously. Don’t assume immediate delivery; design your system to tolerate some delay.
Wrapping up
Sending reliably isn't enough. You must also receive reliably.
Outbox ensures events are emitted only after successful business ops
Inbox ensures events are processed once, even if duplicated
Exactly-once delivery is hard. Aim for at-least-once + idempotency
Both patterns rely on transactional writes and retryable processing
To understand why the Outbox and Inbox patterns matter, consider this well-defined comparison of delivery guarantees:
Build like the network is unreliable—because it is.
Every outbox needs an inbox.
I'm building a community around System Design. Want in?
Join for free, or go paid to unlock private discussions and chat support.
Articles I enjoyed this week
Patterns for Monolith to Microservice Migration by
Coordination Crisis in Modern Tech Work by
How to Better Organize Your React Component Files? by
Hashing in Coding Interviews by
Designing Social Media News Feed System (PAID) by
Thank you for reading System Design Classroom. If you like this post, share it with your friends!
You have to also mention about the acknowledgement event after consumption of the message. This ack will update a flag in the outbox table to make sure, the operation is complete.
nicely explained + visuals make it easier to understand. Good work, Raul.