They start with Kubernetes. Then microservices. Then queues. Then service meshes. Then they memorize diagrams full of boxes, arrows, and cloud logos.

But when a real system gets slow, unstable, or expensive, they struggle to explain what is actually going wrong.

• Why did latency spike?

• Why did retries make the outage worse?

• Why did the database collapse even after adding more app servers?

• Why does everything look healthy on dashboards while users still get the wrong behavior?

That happens because system design is not really about diagrams or tools.

It is about constraints.

If you do not understand the forces underneath the system, the architecture picture gives you confidence without understanding. It helps you describe the system, but not reason about it.

The tax you pay to run multiple agents

If you’ve spent any time with coding agents, you know the feeling. You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director. When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

• Interoperable: Claude Code and Codex compatible, with more coming soon.

• Full Visibility: Confidently run multiple agents and work through your backlog faster.

• Smart Triage: See which agents are blocked or in review and jump in to unblock them.

• Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.

• Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

The real issue with how system design gets studied

The problem is not a lack of resources. Engineers have more books, videos, posts, and courses than ever. The real issue is that many people learn solutions before they understand the problems those solutions were built to solve.

• They learn microservices before they understand network cost.

• They learn queues before they understand duplication and ordering.

• They learn caching before they understand staleness and invalidation.

• They learn replication before they understand read and write access patterns.

That sequence creates a tool-first mental model. And it sounds sophisticated, but it usually breaks under pressure.

In practice, systems often fail for much simpler reasons. A service calls too many dependencies. A query is inefficient. An index is missing. A partition key is skewed. A retry policy multiplies traffic. A cache serves stale data. A queue grows without control. A hot path lacks ownership.

None of those problems looks glamorous, but they shape whether a system survives production. Yes, sophisticated-looking architectures can still fail in predictable ways. The team chose modern components but never built a clear understanding of the forces underlying them.

That is why the order of learning matters so much.

The order that actually builds intuition

If your goal is real system design judgment, the most useful sequence is much more grounded:

• Start with networks

• Then move into databases

• Then study caching

• Then queues and streams

• Then load balancing

• Then build a few classic systems

• Then read postmortems and failure reports

This order works because each layer depends on the one below it. If you skip the lower layers, everything above turns into memorization. If you understand the lower layers, the upper layers naturally begin to make sense.

That order matters, but there is one thing that comes even before it: being clear on what problem you are actually solving.

But before all of that, learn to ask for requirements

Even the best technical foundation will not help much if you skip the first step: understanding the problem.

You need to know what matters most, what can bend, and what absolutely cannot fail.

Good system design starts with questions.

Is this system read-heavy or write-heavy?
Do we care more about latency or consistency?
How fresh does the data need to be?
Is traffic steady, or does it arrive in bursts?
What is the expected scale?
What is the budget?
What is the recovery expectation when something goes wrong?

Those questions shape everything that comes after.

And this is the part many engineers miss: every requirement is a trade-off in disguise.

“The system must be fast” sounds simple until you realize what it usually means. Speed often competes with consistency. A read that returns stale data is fast. A read that coordinates across multiple replicas is safer, but slower. In real systems, you rarely get everything at once.

The first skill in system design is deciding what you are willing to sacrifice.

A read-heavy system pushes you toward caching.
A write-heavy system pushes you toward coordination.
A bursty workload makes queues and backpressure more important.
A low-latency system makes every network hop more expensive.
A system with strict consistency requirements changes how safely you can use replicas, async flows, and stale reads.

This is where many system designs go wrong.

Engineers jump from a vague problem statement to an architecture too quickly. They start suggesting tools before they understand the actual constraints. But requirements are what tell you which constraints matter most.

That is why asking good questions is part of system design, not something that happens before it.

Requirements tell you what matters.
Fundamentals tell you why it matters.
Architecture is what you choose because of both.

And once you understand that, the next thing to study is the layer every distributed system depends on first: communication.

Networks Explain Everything Else

An engineer sets a 30-second timeout on an HTTP call to a payment service.

What they do not realize is that the load balancer upstream times out after 15 seconds.

At second 15, the load balancer cuts the connection and retries the request.

The original call is still running.

There are now two requests in flight.

The payment goes through twice.

No obvious application error.
No crash.
No red alert on the dashboard.
Just a clean wall of 200s and a customer who got charged twice.

That’s a networking bug wearing a business logic disguise, and it’s exactly why networking matters so much in system design.

This is what an innocent‑looking timeout mismatch really does under the hood.

It is also one of the most neglected topics in how engineers learn distributed systems.

Every request that leaves one machine and reaches another carries costs and risks. The moment your system crosses a network boundary, latency becomes real, partial failure becomes possible, retries become dangerous, and duplication becomes something you have to design for.

If you do not understand what happens when bytes leave one machine and travel to another, a lot of your architecture decisions end up resting on assumptions you cannot verify.

That is why networking should come first.

A local function call is fast and predictable. A network call is slower, less predictable, and full of edge cases that only show up under pressure. That difference changes everything.

It changes how you think about service boundaries. It changes how you think about retries. It changes how you think about failure. And it changes how expensive “just one more network hop” really is.

Once you understand the cost of communication, the next question becomes just as important: how the system stores and retrieves data.

Then learn databases

If networks explain how systems communicate, databases explain how systems remember.

A huge amount of system design is really about how data is modeled, accessed, coordinated, and maintained under load. The consequence is that shallow database advice creates so much damage. It is easy to say “use NoSQL for scale” or “add replicas,” but without understanding the fundamentals, those ideas turn into expensive guesses.

You need to understand relational and non-relational models, indexes, query plans, transactions, isolation, replication, and partitioning. Not because every system needs every one of those concepts, but because these are the things that explain why systems behave the way they do.

Many systems do not really have a scale problem. They have a data access problem. The query is inefficient. The index does not match the filter pattern. The primary handles too many reads. The write path needs too much coordination. The partition key creates a hotspot. The model fights the access pattern.

This is also where one of the most important habits in system design begins: thinking separately about reads and writes.

Reads are often easier to scale. You can cache, replicate, precompute, and sometimes serve them from the edge. Writes are harder. Writes need coordination. Writes need durability. Writes need consistency rules.

This is also where ideas like the CAP theorem stop feeling abstract.

A write lands on one node, but replication has not finished yet. A read goes to another node and gets stale data. At that point, the trade-off is no longer theoretical. You can wait for replication and pay the latency cost, or you can serve the stale read and accept temporary inconsistency.

There is no magical third option.

That trade-off explains a huge number of system design decisions. It explains why some systems choose replicas and eventual consistency. It explains why others keep reads close to the primary. It explains why low latency and perfect freshness often pull in opposite directions.

If you cannot reason about the write path, you cannot really reason about the architecture.

And once you understand how costly repeated reads can become, the next step becomes obvious: avoiding unnecessary work.

Then learn caching

Caching is probably the clearest example of a system design trade-off.

You are trading correctness for speed.

Every cached value is a bet that the underlying data will not change before the cache expires. Sometimes that bet pays off, and the system gets much faster at a very low cost. Sometimes it does not, and a user sees a price that no longer exists, clicks buy, and your support team spends the afternoon apologizing.

This is the reason caching matters so much in system design. It forces you to answer a harder question than “should we cache?”

The real question is: what are we comfortable being wrong about, and for how long?

That is what makes caching more than a performance trick.

It is a judgment call.

A lot of engineers first meet caching as an easy win. Put Redis in front of the database. Add a CDN. Set a TTL. Watch latency drop. And sometimes that really is the right move. Caching can remove repeated work, reduce pressure on the database, and give you some of the cheapest performance improvements in the system.

But the moment you cache something, you also accept a new kind of risk.

You accept that the fast answer may not be the latest answer.
You accept that invalidation may be harder than storage.
You accept that two users may see different realities for a short period of time.

Caching only works well when you understand the shape of the data and the cost of being wrong.

Some data is ideal for caching because it changes so little.
Some data can tolerate staleness because the business impact is low.
Some data looks cacheable until the first bad customer experience makes the trade-off painfully real.

That is the real lesson.

Caching is not just about speed. It is about deciding where freshness matters, where it does not, and how much inconsistency the system can tolerate without breaking trust.

Once you understand that, caching starts to look like what it really is: a trade-off among latency, freshness, and complexity.

And even after you reduce repeated reads, one big problem remains: what happens when too much work arrives at once?

Then learn queues and streams

Once you understand request-response systems, the next major shift is learning how to decouple time.

Queues and streams matter because they let you move work off the critical path. Instead of forcing every piece of work to complete during the user request, they let the system defer non-critical tasks, smooth out spikes, and scale consumers independently.

This is one of the cleanest ways to protect the application’s hot path.

This is why asynchronous systems are so powerful. They help absorb traffic bursts, isolate failures, and prevent slow downstream dependencies from dragging down the user-facing experience. But they do not eliminate complexity. They relocate it.

That part often gets learned too late. As soon as you introduce asynchronous processing, you also introduce duplicate delivery, out-of-order events, replay behavior, tracing complexity, idempotency requirements, and eventual consistency.

The system may become more resilient under load, but it also becomes harder to reason about.

That does not make queues a bad choice. It simply means they are a trade-off. They buy resilience and throughput at the cost of immediacy and simplicity. Once you understand that trade clearly, you can choose async patterns for the right reasons instead of treating them like a default upgrade.

As traffic spreads across multiple services, consumers, and workers, another challenge becomes unavoidable: deciding how to distribute work across infrastructure without creating new bottlenecks.

Then learn load balancing

A lot of engineers hear “the system is overloaded” and immediately think, “add more servers.”

Sometimes that works. A lot of the time, it does not.

If the database is already saturated, more app servers will just create more traffic against the same bottleneck. If your workers are slow because downstream writes are struggling, adding more workers may just increase contention faster. If the cache hit rate is poor, spreading requests across more instances does not fix the real problem.

That is why load balancing is easy to oversimplify.

Yes, it distributes traffic. But the real lesson is not “send requests to many servers.” The real lesson is understanding what kind of load you are distributing, how it should be distributed, and whether the rest of the system can absorb it.

That is where details start to matter. Round robin, least connections, sticky sessions, health checks, failover behavior, consistent hashing. These are not just implementation details. They shape whether traffic spreads intelligently or whether the system keeps pushing work into the wrong place.

Load balancing helps you scale horizontally, but only when you understand where the bottleneck actually lives. Otherwise, you are just moving pressure around and hoping it looks like progress.

That is what makes load balancing a useful learning topic. It teaches you that scaling is not about blindly adding machines. It is about controlling where work goes and knowing which layer actually needs help.

And once these building blocks start making sense, the best way to make them stick is to build with them yourself.

Build classic systems yourself

Reading helps. Courses help. Videos help. But building is what forces the ideas to stick. The moment you try to design even a “simple” system yourself, you stop thinking in labels and start thinking in constraints.

Take a rate limiter.

At first, it sounds easy: allow 100 requests per minute and block the rest. But the moment you try to build it, the real questions show up.

1. Where do you store the counters?

2. What happens if multiple servers increment the same key at the same time?

3. Do you reset counts with a fixed window, or use a sliding window that is more accurate but more expensive?

4. Do you optimize for speed, or for fairness?

5. What happens if the limiter itself becomes a bottleneck?

Now you are not just building a small feature. You are dealing with atomicity, concurrency, latency, memory growth, and correctness under load. A local in-memory limiter is fast, but breaks across multiple instances. A centralized Redis-based limiter is more consistent, but adds a network hop and a dependency on another system. Even here, the design is still about trade-offs.

Building these kinds of systems matters. They force you to apply the exact lessons that look abstract on paper.

A few classic systems are especially useful:

• A URL shortener teaches you how to think about key generation, read-heavy traffic, redirects, and cache-friendly design.

• A chat application teaches you how to reason about persistent connections, fan-out, ordering, and message delivery.

• A feed system teaches you how to think through fan-out on read vs fan-out on write, ranking, and precomputation trade-offs.

• A notification system teaches you how to design for retries, idempotency, provider failures, and backpressure.

Practice builds judgment.

Still, even practice systems often remain too clean compared with production. That is why reading about real failures matters so much.

Read real-world failures

Tutorials usually teach the happy path. Postmortems teach reality.

A clean architecture never shows the retry storm that multiplied traffic, the migration that locked writes, the dead-letter queue nobody monitored, the stale cache serving wrong results, or the timeout mismatch that triggered cascading failure. Those are the details that make real systems painful and instructive.

This is where your thinking starts to mature. You stop asking only how the system should work and start asking how it fails. That is the more useful question, because distributed systems are judged by behavior under pressure.

A system that works perfectly under ideal conditions tells you very little. The real design shows up when traffic spikes, replicas lag, workers crash in the middle of a task, messages get duplicated, or downstream services slow to a crawl. Postmortems sharpen your instincts for actual system behavior rather than idealized architecture.

Here are 5 strong postmortem examples worth reading:

1. Postmortem of database outage of January 31 (A great example of transparency, detailed timeline, and clear lessons learned)

2. Slack’s Incident on 2-22-22 (A strong example of how cascading failures happen in complex systems)

3. Meta - More details about the October 4 outage (A useful postmortem for understanding network failures at scale)

4. Post mortem on the Cloudflare Control Plane and Analytics outage (A good example of connecting outage cause, recovery, and follow-up fixes)

5. AWS Post-Event Summaries (A solid resource for studying how large platforms document incidents consistently)

[This deserves its own full article, and there is more coming on that soon.]

After you read enough of them, one idea becomes impossible to ignore: the diagram was never the whole design.

The diagram is not the design

This is one of the biggest mindset shifts in system design.

People often mistake the picture for the architecture.

Client. Load balancer. App layer. Cache. Database. Queue. Workers.

That picture is useful.

But it is not the real design. The real design lives in the rules.

What is the source of truth?
What can be stale?
What retries?
What is idempotent?
What times out first?
What happens under overload?
What fails open?
What fails closed?
What happens if a replica lags?
What happens if a message gets processed twice?

Those questions define the system far more than the visible boxes do.

You can memorize common architecture diagrams and still struggle in a real design conversation.

They know the shapes. They do not understand the behavior.

And that gap between shape and behavior is exactly what separates surface-level familiarity from actual engineering judgment.

What actually makes someone good at system design

What matters is understanding how systems behave under pressure.

That means understanding where latency comes from, why writes deserve more respect than reads, how retries can multiply outages, how caching can improve speed while hurting correctness, and why asynchronous systems need idempotency and ordering strategies.

Once those forces become clear, your design decisions change.

You stop chasing complexity because it looks impressive.
You start protecting the hot path.
You reduce blast radius.
You respect the database.
You design recovery paths instead of focusing only on success paths.

That is when system design stops being an interview performance and starts becoming engineering judgment.

Once you see it that way, the path to studying it becomes much simpler.

Final thought

Many engineers do not need more buzzwords. They need stronger fundamentals.

The best system designers are those who understand what the system is optimizing for, where pressure builds, and which trade-offs actually matter.

They know latency is real.
They know writes are expensive.
They know retries can make things worse.
They know scale is often more about controlling work than adding more machines.

They know good system design starts with constraints, not diagrams.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

An IoT system publishes two events from the same device into a queue:

• “door open”

• “lock door”

They land on a shared topic and get picked up by different consumers running in parallel.

Each event is valid. Each handler executes successfully. Each service does exactly what it was designed to do.

But because processing is async, they’re handled out of order:

Lock door → Door open

And so, the final state is wrong.

The door ends up open, even though a lock command was issued.

Nothing failed.
No alerts fired.
The system behaved exactly as designed.

Each component is locally correct, but no one owns the sequence over time. The result violates the expected invariant: “lock means locked.”

From the user’s perspective, the system is broken.

This is the real danger with async systems: failures that don’t look like failures. To understand why, you need to look at what async really changes.

AI coding gets practical when Agent B knows it must wait for Agent A.

Cline Kanban brings that workflow into one UI: you can run multiple agents, monitor progress at a glance, and define dependencies so work unlocks in the right order. It’s a cleaner way to manage parallel agent execution without losing track of what’s happening.

See how it works

What async actually changes

Async systems unlock scale by removing blocking operations and allowing work to be processed in parallel. Services decouple, throughput increases, and systems can handle spikes without cascading failures.

That’s why queues, streams, and event-driven architectures are everywhere.

But there’s a fundamental trade-off:

You lose control over execution order.

You’re trading simpler consistency for availability and throughput. Order is no longer implicit; it must instead be designed.

Once ordering is not guaranteed, consistency becomes timing-dependent. State updates can arrive in different sequences, retries can reorder operations, and parallel consumers can apply conflicting changes.

The system still “works,” but the result depends on when things happen, not just what happens.

This isn’t a tooling issue. It’s a direct consequence of async processing. And it starts with how messages flow through your system.

Why this happens

At a high level, async pipelines look simple: producers send messages to a queue, and consumers process them.

In reality, several factors break ordering.

Consumers run in parallel, so related messages may be processed at the same time. Retries can push older messages ahead of newer ones. Network delays vary, so delivery order isn’t guaranteed. Batch processing changes execution timing.

All of these introduce reordering.

A message sent first might be processed last. Two related events might be handled by different consumers simultaneously.

The system is behaving correctly from an infrastructure perspective, but incorrectly from a business perspective, because no component owns the invariant over time.

It’s like multiple operators controlling the same device without coordination.

So the question becomes: where do you reintroduce order?

The simplest fix most teams miss

You don’t need ordering across the entire system. That would destroy scalability.

You need ordering where state matters.

Rule of thumb: enforce ordering at the level where you store state (device, user, order), not globally.

This is where grouping comes in.

By grouping related messages and processing them sequentially within that group, you preserve the correct order for a specific entity while still allowing parallelism across unrelated entities.

In practice, this means assigning all events for a device, user, or order to the same processing lane. For Kafka, this is the partition key; for SQS FIFO, this is the message group ID; for Service Bus, this is the session ID.

Within that lane, events are handled in sequence. Across lanes, the system remains fully parallel.

You’re not eliminating concurrency; you’re shaping it.

This pattern is everywhere

Modern messaging systems implement this concept in different ways, but the underlying idea is the same.

SQS FIFO uses message groups to enforce ordered processing. Azure Service Bus uses sessions to ensure sequential handling. Kafka uses partition keys to guarantee order within a partition.

None of these systems attempt to enforce global ordering. They all enforce ordering within a defined boundary.

That boundary is what makes scale possible.

Global ordering would force all messages through a single pipeline, creating a bottleneck. Local ordering allows independent groups to progress without blocking each other.

Which brings you to the most important design decision.

The real decision isn’t grouping

It’s the grouping key.

The grouping key determines how messages are partitioned and how ordering is enforced. A good key aligns with real-world entities (like a device, user, or order) so that related events are processed in sequence.

A poor key introduces problems.

If the key is too broad, unrelated work is forced into the same group, reducing parallelism. If it’s too granular or random, related events may not stay together, breaking ordering guarantees.

And if the key is skewed, you create hotspots. For example, using a “tenant id” as the key when one tenant owns millions of devices collapses parallelism into a single hot group.

You’re not just deciding how to route messages.

You’re defining how your system handles concurrency.

Where things break again

Grouping solves ordering, but it introduces new constraints.

Within a group, processing is sequential. A single slow message can delay everything behind it.

Under uneven traffic, some groups receive far more events than others. A single high-traffic entity can dominate a partition, turning it into a bottleneck while others remain underutilized.

This is the hot partition problem.

It rarely shows up in development environments. It appears under real traffic, when distribution is uneven and unpredictable.

At that point, scaling requires rethinking your grouping strategy, not just adding more consumers.

What grouping actually buys you

Grouping doesn’t eliminate concurrency issues. It scopes them.

Instead of system-wide inconsistency, you limit coordination to smaller, well-defined boundaries. This makes behavior more predictable and easier to reason about.

You still have concurrency, but it’s controlled. You still have trade-offs, but they’re explicit.

That’s what makes async systems workable in practice.

My take

Most teams adopt async for scale and treat ordering as an afterthought. They assume inconsistencies will be rare or manageable.

In reality, those inconsistencies surface under load, when they’re hardest to debug and fix.

A better approach is to design for ordering from the start. Define how your system handles sequence and state transitions before introducing parallelism.

If you don’t design for order, async will decide it for you.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

You rename a class, split a function, or move logic into a different module. If something breaks, you fix it or roll back the commit. The blast radius is usually limited to the application you’re working on.

Refactoring a database is different.

Once data is involved, things get complicated quickly. It’s very rare to find a database only used by a single system

• Mobile apps rely on it.

• Admin portals query it.

• Background jobs write to it.

• And analytics pipelines read from it.

As a result, a simple schema change can become a production risk.

Drop the wrong column, and suddenly, multiple systems start failing at once. A harmless change in a migration script turns into a massive disaster.

Database refactoring requires a different mindset.

The safest way to evolve a schema without breaking running systems is the Expand / Contract pattern.

Building an AI demo is easy. Running AI reliably in production is a different story.

Data pipelines, governance, cost control, observability; that’s where most of the work actually is.

AWS put together a simple guide that breaks this down into 6 architectural stages, from data foundations to operating AI systems in production.

If you’re building AI systems, it’s a useful way to see where your team actually stands.

Download the guide here

A Very Common Production Failure

Imagine a team that wants to improve a table design. They currently store the user’s name like this:

At some point, they realize this structure is limiting. Searching, sorting, and validation all become harder. Splitting the column into two fields makes much more sense:

So someone writes a migration:

The change works perfectly in development. The migration runs without errors. Tests pass.

But when the change reaches production, things start failing.

The mobile application still writes to the FullName column. The admin dashboard still reads from it. A background worker updates the field during profile imports. None of those systems deployed at the same time as the database migration.

The result is predictable: production breaks.

The schema change itself was correct. The problem was that the change was not backwards compatible.

What’s Actually at Stake

Databases usually sit at the center of the architecture.

Every one of these systems depends on the same schema. That means a schema change is rarely isolated, and it becomes a cross-system change.

In theory, you could coordinate deployments across every service that depends on the database. In practice, that rarely works. Teams deploy at different speeds; some systems update frequently, while others change only occasionally.

A safer approach is to design migrations that allow old and new versions of the system to coexist temporarily.

This is exactly what the Expand / Contract pattern provides.

The Expand / Contract Pattern

The idea behind Expand / Contract is simple:

Never introduce a breaking schema change.

Instead, evolve the schema in stages so that both the old and the new structure can function at the same time.

The migration typically follows three phases:

Each phase moves the system forward without breaking existing consumers.

Step 1 — Expand

The first step is to introduce the new structure while keeping the old one.

At this point, the system still works exactly as before. Existing applications continue reading and writing FullName.

Next, migrate existing data into the new structure.

Example migration logic:

Note: split_part here represents a simple string-splitting utility (either built-in depending on the database, or implemented as a helper function). In real systems, migrations often need additional logic to handle edge cases such as single-word names, extra spaces, or inconsistent formatting in the original data.

Now the table supports both formats.

Old systems keep functioning, and new systems can begin using the improved schema.

Step 2 — Transition

During the transition phase, both schemas must stay consistent.

You cannot allow data like this:

If different services read different columns, inconsistent data will appear quickly.

To prevent this, writes must keep both representations synchronized.

A simple way to do this is to use dual writes at the application layer.

Whenever a service updates user data, it writes to both the old column and the new columns.

Example:

Because both values are written inside a single database transaction, the update remains atomic, and both representations stay consistent.

Note: dual writes become much harder when they involve multiple systems (for example writing to two databases or publishing an event). In those cases, patterns like transactional outbox or change data capture (CDC) are typically used to guarantee consistency.

During this phase:

• Older services continue reading FullName

• Newer services read FirstName and LastName

• The application keeps both representations aligned

  

The key requirement is simple: both schemas must stay consistent until the migration finishes.

This transition period allows services to migrate at their own pace. Once every system reads from the new columns, the old column can be safely removed.

Where Should Synchronization Live?

During the transition phase, the system must keep both schema versions synchronized.

There are two common ways to enforce this: application-level dual writes or database-level mechanisms such as triggers or generated columns.

Application-level dual writes keep the logic explicit in the codebase. You can see exactly how data flows and reason about it during reviews and deployments. This approach also avoids adding extra work to the database write path.

However, dual writes rely on every writer being updated correctly. If multiple services or scripts write to the same table, it’s easy for one of them to forget the new columns during the transition.

Database-level mechanisms solve that problem by centralizing the logic. A trigger or generated column guarantees that any write to the table keeps both representations consistent, regardless of which service performs the write.

The trade-off is that triggers can add latency and make behavior less visible to application developers.

A useful rule of thumb is:

If many heterogeneous clients write to the same table, go for database-level mechanisms during the transition period.

If a single service owns the write path, application-level dual writes are usually simpler.

Both approaches follow the same principle: keep the old and new schema consistent until every consumer has migrated.

Step 3 — Contract

You can remove the old structure once all applications have migrated to the new schema.

This is known as the contract phase. The final schema becomes:

At this point, the refactoring is complete, and the compatibility layer is no longer needed.

Expand / Contract Is Not Just About Columns

Most examples focus on columns, but real schemas include more than that.

Indexes, constraints, and foreign keys often depend on the fields being migrated, so they must evolve through the same Expand / Contract lifecycle.

For example, imagine FullName has an index used for search queries. When introducing FirstName and LastName. The migration might look like this.

Expand phase

The old index on FullName remains in place while applications migrate.

Once all systems use the new columns, the old index can be removed.

Contract phase

The same principle applies to:

• unique constraints

• foreign keys

• check constraints

• partial indexes

However, creating indexes or constraints on large tables can be disruptive in itself.

Some databases support online or concurrent index creation (for example, PostgreSQL’s CREATE INDEX CONCURRENTLY), which avoids blocking writes during the operation.

You can also introduce constraints in stages. A common pattern creates the constraint first without validating existing rows, then validates the data afterward.

Example:

This allows the system to keep running while the database verifies existing data.

Realistic Considerations

The pattern itself is straightforward. Real systems introduce complications.

Long transition periods

Some consumers migrate slowly.

Internal tools, reporting jobs, and ETL pipelines often depend on the same tables but deploy on very different schedules. It’s not unusual for the transition phase to last weeks or even months.

Temporary schema complexity is part of the cost of safe evolution.

Partial deployments

It is common for migrations and application changes to deploy at different times.

For example:

Migration Checklist

Before running a schema migration, it helps to answer a few operational questions.

1. Do we have a backfill plan and progress tracking?
   Large tables often require batch migrations. Make sure the process can run incrementally and resume if interrupted.

2. How will we detect when the last consumer of the old schema is gone?
   Base schema removal on evidence. Monitor queries, application logs, or feature flags to confirm that no services still depend on the old fields.

3. What’s the rollback strategy if the new schema misbehaves?
   Even backward-compatible migrations can introduce unexpected issues. Make sure the system can safely revert to the previous behavior while both schemas still exist.

Thinking through these questions early helps you avoid the most common failure mode of database migrations: discovering operational problems only after the change reaches production.

During this entire window, the system must function correctly with both schemas.

High-volume tables

On tables with heavy write traffic, synchronization mechanisms require extra care.

Triggers can add latency because they execute on every write. For very busy tables, many teams prefer dual writes combined with feature flags in the application layer.

Services gradually migrate their reads while the application maintains compatibility during the transition.

Backfilling Large Tables Without Locking Them

Backfilling Large Tables Without Locking Them

The examples above use a simple migration:

This works well on small tables.

On large production tables, it can cause serious problems.

A single UPDATE touching millions of rows can:

• lock the table

• generate massive transaction logs

• block writes

• slow down the database for other workloads

Instead, most teams backfill data in batches.

The idea is simple: update small chunks of rows at a time.

Example:

A background job repeatedly runs until it migrates all rows.

Pseudo workflow:

This approach keeps transactions small and prevents long table locks.

It also allows the migration to run safely while the system continues serving traffic.

Many production systems combine this with progress tracking, so the job can resume if it stops.

The goal is simple:

Migrate the data without impacting the rest of the system.

The Hardest Problem Is Not Technical

The Expand / Contract pattern itself is technically simple.

The hardest problem is organizational.

Shared databases mean multiple teams depend on the same schema. That means migrations require coordination, communication, and clear ownership.

Without that discipline, temporary compatibility layers remain indefinitely. Old columns stay around because nobody is sure which system still depends on them.

Over time, schemas accumulate artifacts like this:

Successful teams treat schema changes like any other cross-team change:

• they track migration progress

• define schema ownership

• schedule the contract phase explicitly

The technical pattern is straightforward.

The real challenge is making sure the organization follows through.

Takeaways

• Database refactoring is harder than code refactoring because many systems depend on the same schema.

• Direct schema changes often break running services.

• The Expand / Contract pattern allows schemas to evolve safely by introducing changes in stages.

• Dual writes keep old and new structures consistent while systems migrate.

• A schema migration completes only after you remove the old structures.

The safest database migration is the one that allows old and new systems to coexist.

Compatibility buys you time.
Time keeps production running.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

Can a system be accurate but inconsistent?

Many candidates pause here because the phrase sounds like an oxymoron; something contradictory that shouldn’t even be possible.

It is not.

Understanding this possibility is important because distributed systems constantly separate accuracy & consistency.
And once systems scale across replicas and regions, the gap between accuracy and consistency becomes impossible to ignore.

To see why, we first need to define what those terms actually mean.

The best way to build any app

Most “AI app builders” aren’t actually app builders. They are infrastructure middlemen. You live inside their box, choose from their secret stack, and force you to start over the moment you want to use a different database, payment processor, or tool. With Orchids:

• You’re not locked-in to Supabase or Stripe.

• Not forced to spend credits, bring your own AI subscriptions with you.

• One-click deployment straight to Vercel

Build anything. From Web, mobile, internal tools, browser extensions, scripts, and bots. Orchids.app is capable of building anything that you can put your mind to.

Use code MARCH15 for 15% off (one-time discount).

Get Started Today

Accuracy vs Consistency

Let’s start with the definitions.

Accuracy means the data reflects the real-world value.

Consistency can mean different things depending on context.
In casual conversations, engineers sometimes mean “replicas return the same value.”
In the CAP theorem, consistency specifically refers to linearizability, where reads always see the most recent committed write.

In a single database running on one machine, these usually appear together.
The correct value is written once, and every read returns that same value.

But distributed systems introduce replication.

Multiple nodes store copies of the same data.
Those copies must stay synchronized.

That synchronization delay is exactly where accuracy and consistency begin to diverge.

The confusion happens because small systems usually give you both properties at the same time.

And the easiest way to see that is through a very common scenario: a system that is accurate but temporarily inconsistent.

Accurate but inconsistent

This situation happens constantly in distributed systems.

Imagine a user updating their email address.

Now imagine a read request arrives during that replication window.

The primary database returns the new email.

A replica still returns the old one.

Some users see the new value.
Others see the old one.

The system already contains the correct value, so it is accurate.

But the nodes do not agree yet, so it is inconsistent.

This behavior appears in systems using:

• read replicas

• multi-region databases

• asynchronous replication

• eventual consistency models

The inconsistency usually resolves itself once replication finishes.

But not all inconsistencies are temporary. In fact, the more dangerous scenario is when a system becomes perfectly consistent while being completely wrong.

Consistent but inaccurate

Now consider a different situation.

A bug writes the wrong tax rate.

Every node returns the same number.

The system is now perfectly consistent.

But the number itself is incorrect.

Consistency guarantees agreement about the relative order of reads and writes as observed by clients, in a way that behaves like a single global log.

It does not guarantee correctness.

A distributed system can synchronize bad data just as efficiently as good data.

This is why strong consistency does not protect a system from logical errors.

And that leads to one of the most common misunderstandings engineers have about the CAP theorem.

The CAP misconception

Many engineers assume strong consistency prevents incorrect data.

But that’s not what the CAP theorem promises.

In the CAP theorem, consistency refers to linearizability: every read appears to see the most recent completed write as if all operations happened in a single global order.

This is different from the everyday meaning of “replicas having the same value” or SQL-style consistency constraints.

If the write itself is wrong, the system will consistently return the wrong value everywhere.

In other words:

Consistency protects agreement about the order of writes, not the truth of the data itself.

Truth depends on things like:

• application validation

• business rules

• transactional safeguards

• domain logic

Distributed systems coordinate during and after writes so nodes agree on the result.

Many production systems relax this guarantee and instead provide eventual consistency, where replicas may temporarily diverge but converge over time.

But that agreement comes at a cost, because coordinating nodes across machines or regions is never free.

The latency tax of coordination

Strong consistency requires nodes to coordinate before confirming a write.

A typical strongly consistent write looks like this:

Each additional replica adds more communication.

Across regions, this becomes expensive.

A write that takes 5 ms locally might take 80–200 ms globally.

At scale, that latency affects user experience.

To avoid that cost, many distributed systems allow writes to succeed without waiting for every replica to confirm.

Updates propagate asynchronously.

This reduces latency but introduces temporary inconsistency.

Strong consistency requires coordination between nodes before a write is considered committed.
Most distributed databases implement this using quorums or consensus protocols, where a majority of replicas must acknowledge the write.

And different products choose different approaches depending on what matters most for the system.

Why systems choose different trade-offs

Architecture decisions always depend on the domain.

Some systems must guarantee correctness immediately.

Others must prioritize speed and availability.

Systems that require strong consistency

These domains cannot tolerate incorrect values:

• banking balances

• payment processing

• financial ledgers

• inventory reservations

If two nodes approve the same payment simultaneously, the system could double-charge a user.

To prevent that, these systems accept the latency cost of coordination.

Correctness with respect to business invariants must be guaranteed before a transaction commits.

But not every system faces that constraint.

Systems that accept eventual consistency

Other domains prioritize responsiveness and global scale:

• social feeds

• analytics dashboards

• recommendation engines

• caching layers

If a post appears a few seconds later in another region, the user experience barely changes.

But forcing every write to coordinate globally would slow the system dramatically.

These architectures allow temporary inconsistency while updates propagate across replicas.

Eventually, all nodes converge to the same value.

And that design decision reveals the real lesson behind CAP.

The real lesson of CAP

The CAP theorem often appears in textbooks as a theoretical concept.

In practice, it represents a constant architectural choice.

Every distributed system must balance:

• coordination cost

• latency

• availability

• correctness guarantees

At a small scale, these trade-offs are easy to ignore.

But once systems span multiple replicas, regions, and networks, they become unavoidable.

Which is why the real challenge in system design isn’t eliminating trade-offs.

It’s deciding which trade-off hurts the least.

Final thought

Distributed systems will always contain imperfections.

Nodes fail.
Networks partition.
Replication lags.

You cannot design those problems away.

But you can decide how the system behaves when they happen.

A system can be:

• accurate but temporarily inconsistent (replication lag)

• consistent but permanently wrong (a bad write replicated everywhere)

• both inaccurate and inconsistent (bugs combined with replication delay)

And the architecture determines which one occurs.

Good engineers don’t try to build perfect systems.

They design systems where the inevitable failure mode causes the least damage.

Because at scale, system design isn’t about perfection.

It’s about choosing the right compromise.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

User requests a report → API queues a job → worker generates the PDF.
Is this pub/sub or a work queue?

The team was split.
Some argued it was pub/sub because the producer is decoupled from the consumer.
Others said it was a queue because only one worker processes the job.

It looked like a terminology debate.

It wasn’t.

The answer changes how the system behaves under load, failure, and scale, which means we need to step back and look beyond the tools.

The best way to build any app

Most “AI app builders” aren’t actually app builders. They are infrastructure middlemen. You live inside their box, choose from their secret stack, and force you to start over the moment you want to use a different database, payment processor, or tool. With Orchids:

• You’re not locked-in to Supabase or Stripe.

• Not forced to spend credits, bring your own AI subscriptions with you.

• One-click deployment straight to Vercel

Build anything. From Web, mobile, internal tools, browser extensions, scripts, and bots. Orchids.app is capable of building anything that you can put your mind to.

Use code MARCH15 for 15% off (one-time discount).

Get Started Today

Start with intent, not tooling

When engineers discuss async systems, the conversation often jumps straight to Kafka, SQS, RabbitMQ, or Redis.

That’s like choosing a truck before deciding whether you’re moving furniture or delivering packages.

Transport moves messages.
Patterns define meaning.

The real question is: what is the system trying to accomplish?

Understanding the intent becomes easier when we look at the actual flow that triggered the debate.

The report generator scenario

1. User requests a report

2. API enqueues a job

3. Worker generates the PDF

4. Result is stored, and the client is notified the report is ready

The system expects the job to run once. If the worker crashes, the job must retry. If demand increases, more workers can be added.

Nothing here suggests broadcasting information.

The system is executing work.

This is a work queue.

To see why, we need to look at the pattern designed specifically for executing tasks.

Work queue: built to execute work

A work queue says:

“Do this task.”

Each job is handled by a single worker. If you add more workers, the workload spreads across them. If no worker successfully processes the job, the task remains incomplete, and the system retries it or moves it to a dead-letter queue, but this is a story for another day.

Think about generating a PDF, sending an email receipt, or resizing images. You don’t want five services doing the same work. You want the work done once, reliably.

This model optimizes for throughput, retries, and completion guarantees.

To understand the contrast, let’s look at the pattern built for distributing information instead of executing work.

Pub/sub: built to distribute facts

Pub/sub says:

“This happened.”

When an event occurs, multiple systems may react independently. Each subscriber receives its own copy. One subscriber failing does not block the others.

Imagine a user signs up. That single event might trigger analytics tracking, a welcome email, CRM updates, and fraud checks.

No single service owns the event. They react to it.

This model optimizes for independence and extensibility.

At this point, the confusion usually appears: both patterns decouple components, so why aren’t they the same?

Why decoupling alone isn’t the definition

Decoupling is a side effect, not the defining trait.

If decoupling were the definition, every async system would be pub/sub, and the term would stop being useful.

What matters is consumption behavior.

If another consumer joins:

• should the work split?

• or should the work duplicate?

That answer reveals the pattern.

Your PDF generation should not run twice.

Another way to clarify the difference is to look at what the message represents.

Commands vs events: a useful shortcut

A command tells the system to do something.
An event tells the system something already happened.

“GenerateReport” is a command.
“ReportGenerated” is an event.

Commands belong in work queues.
Events belong in pub/sub systems.

By the way, most of the time, mature architectures use both.

This distinction may sound subtle, but it becomes very real once systems reach production scale.

Why the distinction becomes painful in production

The difference may sound subtle during design discussions, but it becomes very real once systems are under load and failures begin to surface.

Teams that blur the boundary often discover problems the hard way.

Duplicate billing because the same job ran twice.
Customers receiving the same email multiple times.
Retries triggering unexpected side effects.
Work completed but reported as failed due to acknowledgment timing.

These issues don’t come from bad intentions. They come from how messaging systems behave in real environments.

Most queues and streaming platforms provide at-least-once delivery. If a worker crashes, times out, or fails to acknowledge completion, the job becomes visible again and is retried. This improves reliability, but it also means the same job can execute more than once.

Because of this, consumers must be idempotent; the job may run multiple times, but the outcome must remain correct.

Event systems face a related challenge. Pub/sub consumers may replay historical events, reprocess messages after failures, or start from earlier offsets. Idempotency ensures these replays do not create duplicate side effects.

Exactly-once delivery is often discussed but rarely absolute. Some platforms provide transactional guarantees within the messaging system, yet external side effects (sending emails, charging payments, calling APIs) still require safeguards.

Understanding delivery guarantees explains why duplicates happen:

• Retries improve reliability

• Reliability introduces duplicates

• Idempotency preserves correctness

Queues prioritize reliable execution of work.
Pub/sub prioritizes reliable distribution of events.

Getting the pattern right and designing for its guarantees is what keeps systems predictable when real failures occur.

The clean evolution path

Start by enqueueing a command to generate the report.

After completion, emit a ReportGenerated event.

Now other systems can react, notifications, analytics, auditing, and caching, without coupling themselves to the report service.

The work remains controlled.
The ecosystem stays extensible.

If the distinction still feels abstract, a real-world analogy makes the difference obvious.

A simple analogy

Think of an airport baggage system.

When you check a suitcase, it is routed to one handler who loads it onto your flight. That’s a work queue. The task must be completed once, and if it fails, your bag doesn’t arrive.

When the plane lands, an arrival event is triggered. Baggage claim displays update, ground crew prepare the gate, and notifications are sent to connecting flights. That’s pub/sub. Multiple systems react independently to the same event.

Different intent. Different behavior.

Understanding the difference is what allows systems to scale without surprises.

If you remember one thing

Most async flows are not pub/sub.
They are work queues that may emit events after completion.

Understanding that difference prevents duplicate work, simplifies retries, and keeps system responsibilities clear.

And clarity is what allows systems to scale without surprises.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

You inherit a production monolith that starts failing during peak traffic. You have 90 days to improve scalability. A full rewrite is not an option. What do you do?

This is one of the best system design interview questions I’ve heard.

Not because it tests tools. Not because it tests architecture diagrams.

But because it tests something much harder:

Can you reason under real constraints?

Most candidates hear this question and immediately say:

“I’d break it into microservices.”

That answer usually fails.

Not because microservices are bad. But because rewriting a system that’s already failing is the fastest way to make things worse.

This question is really about judgment.

And judgment is the core skill that System Design interviews aim to measure.

If you want to train this kind of judgment intentionally, this is exactly what we focus on at joinenginuity.com.

Not memorizing architectures, but learning how to reason under constraints, pressure, and incomplete information.

To make it easier to get started, here is a 30% off for the first 25 engineers using the coupon code ENGCLASSROOM30.

Join Us

Before jumping into timelines and tactics, there is a step many candidates skip entirely.

It is the most important one.

If you do not get this right, every decision that follows will be flawed, even if your architecture skills are strong.

First: Reframe the problem

Before timelines, phases, or technical steps, this question tests something more basic: can you restate the problem correctly?

Most system design failures don’t start with bad code. They start with a bad understanding of the problem being solved.

So before touching architecture, you need to show that you understand what actually matters in this situation.

Before touching code or architecture, I restate the problem in system terms.

What we actually know:

• The system is already in production

• It fails under peak load, not average load

• Revenue depends on it

• You have 12 weeks, not a year

• A full rewrite is explicitly disallowed

That changes the goal.

Now the goal is not elegance; focus on stability under peak pressure.

Increase peak capacity, reduce blast radius, and buy time.

If a candidate doesn’t say this out loud, they usually miss the intent of the question.

Up to this point, everything has been about thinking clearly.

Now it is time to act.

The first phase is not about scaling up. It is about preventing things from getting worse, and it should take roughly two to four weeks.

Month 1: Stop the bleeding

This phase is about survival. Before you can start to think about scale, elegance, or future architecture, you need the system to stay up long enough to learn from it.

1. Add observability before changing behavior

If the system is failing and you don’t know why, every change is a guess.

So the first move is visibility:

• Request latency (p50 / p95 / p99)

• Error rates by endpoint

• Database query timing

• Connection pool saturation

• CPU and memory pressure

If tracing is missing, add basic tracing.

Why this matters:

Scaling without observability is like load testing blindfolded. You might change something, but it’s impossible to know if it helped.

2. Identify the hot path

Under peak load, not everything matters.

Usually it’s:

• Checkout

• Inventory reads

• Pricing

• Auth/session validation

I look for:

• N+1 queries

• Synchronous downstream calls

• Unbounded fan‑out

• Lock contention

• Chatty ORM behavior

System design is about leverage.

Fix the path every request touches. Ignore the rest for now.

3. Pull fast, reversible levers

No refactors yet. No rewrites.

Only changes I can roll back quickly.

Typical examples:

• Add read replicas

• Introduce cache‑aside for hot reads

• Add timeouts and circuit breakers

• Rate‑limit non‑critical endpoints

• Tune connection pools carefully

In parallel, I apply infra‑level scaling levers that are cheap and reversible:

This includes tuning instance sizes, autoscaling policies, container CPU and memory limits, and thread or connection pool sizes. These changes often buy immediate headroom and reduce cascading failures before any deeper work begins.

The goal of Month 1 is simple:

Survive the next peak.

Month 2: Increase capacity without redesign

Once the system stops falling over, the goal shifts. Now, you increase throughput and reduce pressure, without introducing risky structural changes you can’t easily undo.

4. Turn the monolith into a modular monolith

This is where strong candidates separate themselves.

They don’t jump to the network. They start with boundaries.

Actions:

• Split code by domain, not layers

• Introduce explicit interfaces

• Remove shared global state

• Make dependencies visible

This reduces:

• Change risk

• Accidental coupling

• Debug time

A modular monolith scales better than a distributed system built in panic.

5. Move expensive work off the request path

Peak traffic dies in synchronous work.

Common offenders:

• Email sending

• Payment callbacks

• Inventory recalculation

• Analytics writes

Fix:

• Convert sync → async

• Push work to background workers

• Return 202 Accepted when possible

Rule of thumb:

If it doesn’t need to block the user, it shouldn’t block the request.

6. Apply targeted data fixes

Most scalability problems show up in the database first.

I focus on real queries and real traffic patterns, not theoretical models. Missing indexes, unnecessary transactions, and overly strict isolation levels are common causes of peak‑load failures.

I focus on real queries, not theory.

Typical wins:

• Add missing indexes

• Remove unnecessary transactions

• Lower isolation levels where safe

• Precompute reports instead of running them live

• Cache computed aggregates

When caching is introduced or expanded at this stage, it must be done carefully.

Caching pitfalls matter:

Cache stampedes can take systems down faster than databases. I look for basic protections such as request coalescing or locking, jittered TTLs to avoid synchronized expirations, and clear rules around invalidation. I also make explicit decisions about where stale reads are acceptable and where they are dangerous, especially around pricing, inventory, and payments.

Caching should reduce pressure, not introduce a new failure mode.

No heroic schema redesigns. Only changes justified by metrics.

Month 3: Reduce blast radius and create options

At this point, the system should be stable under peak load. Month 3 is about turning that stability into confidence and making future failures smaller, slower, and easier to manage.

7. Isolate the biggest offender (optional)

If one domain dominates load and boundaries are clean:

• Extract one worker or service

• Keep the contract small

• Avoid creating new coupling

This step is optional.

Many systems stabilize without any extraction at all.

8. Load test peak, not average

By now, I want clear answers:

• Can we handle 2× peak?

• What fails first?

• Do failures degrade or cascade?

Good systems fail partially. Bad systems fall over completely.

9. Document decisions and limits

This is a senior‑level signal.

I document:

• What we fixed

• What we deliberately didn’t fix

• Known limits

• Next extraction candidates

System design is not about pretending trade‑offs don’t exist. It’s about making them explicit.

Why this question works so well

This question works because it exposes how engineers think when systems are already under stress.

Strong candidates naturally reason in constraints. They focus on the hot path first, not the entire system. They respect rollout risk and reversibility. And they understand that architecture, by itself, does not equal progress.

In short, it rewards judgment formed by real production experience, not familiarity with trendy patterns.

My Final takeaway

Scaling is not about rewriting systems.

It is about seeing the system clearly under pressure, choosing the smallest levers that create real impact, and reducing blast radius before adding complexity. Most importantly, it is about buying time so better decisions are possible later.

Good system design is not measured by how fast you can draw boxes.

It is measured by how long your system can survive peak traffic without falling apart.

If someone answers this question well, they can built systems that have had to survive peak traffic.

And that’s exactly what System Design interviews are trying to uncover.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

A customer reports an issue.
Support files a vague ticket.
Engineering tries to reproduce it.
Frontend blames backend.
Backend blames frontend.
Slack threads explode.
Everyone’s busy, but somehow, nobody actually knows what’s going on.

I used to live inside this loop for years.

And here’s what hit me:

The problem wasn’t the customer. It wasn’t the product.
The real issues was the team system itself.

If a team doesn’t have clarity, context, or energy…then nothing works.
It doesn’t matter how strong your architecture is; tired people ship tired products.

This realization forced me to rethink how teams actually build and debug software.
And it led me to six lessons I wish I’d learned sooner.

1. Context is everything

The root cause of most engineering pain is simple:
No one knows what actually happened.

Support writes:
“Checkout is slow.”

Engineering responds:
“Works for me.”

And both are right, because neither has the full story.

Once I started forcing myself (and my teams) to capture the full context once and make it travel with the issue, everything changed.

A good bug report includes:

• real repro steps

• real environment

• a timestamp

• screenshots or (ideally) a replay

• the trace or request ID

This alone removed hours of back-and-forth from every incident.

And the surprising thing?
It also reduced tension between teams.
Clarity is the best stress relief.

2. Replays beat meetings

Eventually, I realized how many meetings existed only because someone in the chain was missing context.

“Let’s jump on a call.”
Translation: “I don’t know what happened, please walk me through it slowly.”

A replay solved that in 10 seconds.

No storytelling.
No guessing.
No back-and-forth.

Just: “Watch this. Let’s fix it.”

This single shift cut a ridiculous number of meetings.
And when you remove unnecessary meetings, you protect the thing teams need most: ENERGY.

3. Too many tools destroy focus

Another lesson: even if you have the right data, it’s hard to use when it lives in four different places.

That was my life for years:

Replay in one tab.
Logs in another.
Traces in a third.
Metrics somewhere else.

By the time I pieced everything together, my brain was fried.

The real problem was that the debugging context lived across too many places.

The fix was simple:
put all the technical signals in one place so everyone sees the same thing.

Not fewer tools.
Just fewer places to hunt for the truth.

One path from “user clicked something” → “database wrote something.”

When the team stops switching tools, they start solving problems.

4. Automate the boring parts (the steps after debugging)

Once we know what broke, there’s a second phase: reproduce the bug reliably, run API calls, build integration tests, document edge cases, verify fixes. That’s where the manual cost really piles up.

That’s when a Notebook becomes handy, you can automate that second phase:

• Convert a session recording into a reproducible test or script.

• Sequence API calls or backend interactions exactly as they occurred, with real data, payloads, headers, and timing.

• Annotate, document, and version the flow so teammates, devs, QA, PMs, can see exactly what happened and how it was tested.

Something like this:

So debugging stays where it belongs, in the replay.
But all the follow-up steps, test scripts, documentation, and regression checks become automatic, repeatable, and shareable.

Simple result: way less repetition, way more time saved.

5. Energy is a system constraint

Startups obsess over uptime, latency, and error budgets.
But the thing that brings teams down isn’t CPU or RAM, it’s human bandwidth.

I saw it repeatedly: burned-out engineers write worse code, fix bugs slower, communicate poorly, and lose patience with customers.
None of that shows up on a dashboard, but you feel it everywhere in the system.

You can’t build great products with exhausted people.

Once I started treating team energy as an actual engineering constraint (not a “soft” idea), everything changed. Delivery became more predictable. PR quality went up. On-call incidents felt less chaotic. And the product itself noticeably improved.

One practical change made an outsized impact:

We implemented flow blocks.

Every engineer had protected windows where they could actually think:

• 2-hour focused work blocks

• no Slack

• no meetings

• no pings

• no “quick questions”

Protecting even one real flow block a day dramatically boosted output without increasing hours worked.
It gave people breathing room, restored cognitive bandwidth, and reduced the constant context switching that quietly drains teams.

Happy teams don’t happen by accident, you have to design your workflow to protect their energy.

And when you do, the quality of the product rises with it.

6. Problems get solved faster together

The last lesson was the most human: silos kill speed.

I watched this pattern over and over again, support sees one piece, frontend sees another, backend sees something completely different, and QA is working off a reproduction that doesn’t match what actually happened.
Everyone is smart.
Everyone is trying.
But everyone is solving a different version of the problem.

When teams don’t share context, they don’t share reality.

The turning point came when we made sure everyone, support, QA, frontend, backend, worked from the same source of truth.
Not notes.
Not Slack threads.
Not assumptions.

A single replay.
A single timeline.
A single root cause.

Once that happened, alignment was instant.
Speculation disappeared.
People stopped “defending their part of the system” and started solving the actual issue.

Shared context creates shared ownership.
And shared ownership is the fastest path to a real fix.

It sounds simple, but it’s the closest thing I’ve found to a productivity multiplier.

One tool that enables all of this: Multiplayer

Multiplayer gives your team a single place where the full story of an issue is captured automatically.

It records user actions, frontend state, network calls, backend traces, logs, and request/response data, all stitched into one connected timeline:

No tool-hopping.
No reconstructing.
No missing context.

It supports quick on-demand captures, continuous recording for hard bugs, and an upcoming conditional mode for issues users never report.

It also gives your AI tools the context they need to generate accurate fixes, tests, and explanations, not guesses.

The result is simple:
less friction, less context switching, and fewer interruptions.

More clarity, more focus, and more calm.

Try it here

My Takeaways

The longer I build software, the more obvious it becomes:
you can’t brute-force your way to great products.

You can push harder, add more meetings, and throw more people at a problem, but if the team doesn’t have clarity, focus, or energy, everything eventually slows down.

Most of the failures I’ve seen weren’t caused by a lack of talent.
They came from teams trying to build in an environment that worked against them:
too many tools, scattered context, constant interruptions, and workflows that drained more energy than the actual engineering work.

And that’s why the tools we choose matter.
Not because tools “fix culture,” but because they quietly shape the day-to-day reality of the team:

• whether debugging takes minutes or hours

• whether context stays intact or gets lost in handoffs

• whether people get real flow time or live inside Slack

• whether collaboration feels natural or forced

A good tool reduces friction.
A good workflow protects energy.
And when the environment supports the team instead of fighting them, everything gets easier; product quality, delivery speed, even customer outcomes.

Happy teams build better products.
And better products make happier customers.

You don’t get that by pushing harder.
You get it by designing the system, including your tools, to help the team do their best work.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

They spend days adding indexes, rewriting JOINs, and tuning queries…
only to realize the real problem isn’t the query; it’s the schema.

Most of the time, slow queries are symptoms, not causes.
The disease starts with schema design.

The Problem is that you can rewrite a query a hundred times; but if the schema underneath is broken, you’re just polishing rust.

When a database is poorly designed, every new feature adds friction:

• Queries grow more complex.

• Indexes pile up without real gain.

• Data integrity becomes optional.

• Performance quietly decays until it explodes, like a dead whale.

By the time you notice, you’re dealing with migrations, downtime, and brittle code paths that no amount of indexing can save.

Most teams don’t realize this until it’s too late.
The first few releases run fine.
Then, traffic spikes, queries slow down, and the database becomes the bottleneck no one planned for.

It’s at that point you discover what good engineers already know:
Performance is baked in at the schema level.

Every decision you make during CREATE (data types, constraints, normalization, indexes) decides how well your system will scale later.

Here’s how to build it right.

Real-time isn’t a buzzword, it’s the performance edge for companies like Vercel, Canva, and Framer. Tinybird helps them transform massive streams of data into instant insights and real-time user experiences.

To share what we’ve learned over the last 5 years running Petabyte-sized clusters, we built a free course on real-time analytical data foundations designed for developers who want to master the basics and build faster, smarter data products.

Join the course for free

1. Normalize First, Then Denormalize with Evidence

Normalization isn’t just a database theory; it’s your first defense against chaos.

When you normalize, you enforce discipline:

• Every column has a single, clear purpose.

• Relationships are explicit, not implied.

• Updates happen in one place, not ten.

That structure pays off when your system evolves.
New features don’t break old data. Schema changes stay localized.
You can scale without rewriting your business logic.

Start here. Get it right. Then measure.

If you later discover that query performance is suffering, that JOIN-heavy reports or analytical workloads are dragging, that’s when you earn the right to denormalize.

But denormalization should always be a measured trade-off, not a default decision.

Every denormalization is a bet: you’re trading consistency for speed.
Make sure it’s a bet you can afford to lose.

Denormalize when:

• You’ve identified repeated JOINs on high-traffic queries.

• You’re operating on immutable data (like historical logs).

• You’ve validated the benefit through metrics, not assumptions.

Denormalize with evidence, not fear.

Most systems suffer not because of normalization, but because of shortcuts taken before the first slow query ever appeared.

2. Use the Right Data Types

Your choice of data types decides how efficiently the database stores, compares, and retrieves your data.
It’s not a trivial decision, it’s a design constraint with real cost.

Most performance issues trace back to oversized or mismatched types.
Developers reach for VARCHAR(255) out of habit, or store timestamps as strings because it “just works.”
Those shortcuts add up, in memory, CPU, and I/O, especially when your table holds millions of rows.

Here’s what to remember:

• Use the smallest data type that fits.
  Smaller data types mean smaller indexes, faster scans, and less memory pressure.
  For example, SMALLINT (2 bytes) can handle values up to 32 767; often enough for bounded IDs.

• Avoid generic types like TEXT and BLOB unless absolutely necessary.
  They live off-page in most engines, increasing read latency and memory churn.
  If you must store large blobs or unstructured text, isolate them in a side table or a storage service optimized for that pattern.

• Match semantics to type.
  Don’t store dates as strings. Don’t store numbers in text. Don’t use floats for currency.
  Every mismatch forces implicit casts and wastes CPU cycles.

• Prefer fixed-width types when possible.
  Fixed-width data (INT, DATE, CHAR) makes row access predictable and cache-friendly.
  Variable-width types (VARCHAR, TEXT) require extra bookkeeping and page splits.

• Use JSON types with intention.
  JSON columns (JSON in MySQL, JSONB in Postgres) They are great for flexible metadata, feature flags, configuration, and sparse attributes.
  But they come with trade-offs:

  • They’re harder to validate at the database level.

  • They can’t always use traditional B-tree indexes.

  • They tempt teams to bypass schema design entirely.
    Treat JSON as an extension, not an excuse. Store structure when you know it; store JSON when you don’t.
    If you rely on JSON queries heavily, index specific keys with expression or GIN indexes.

Even though both versions “work,” the second tells the database exactly what to expect.
Databases are only as smart as the hints your schema provides.

The more precise your data types, the less work your database (and your team) has to guess.

Clean, intentional typing isn’t just about performance, it’s about truth in design.
A schema that reflects reality scales longer, breaks less, and reads like documentation.

3. Build Intentional Indexes

Indexes are like caffeine; they give your queries a quick boost, but too much of them will crash your writes.

They’re one of the most misunderstood performance tools in databases.
Engineers often add them reactively, every time a query slows down, until write operations start crawling.
The key is balance: index what’s read often, not what’s written often.

This simple hierarchy helps you reason about how indexes differ in purpose and structure:

• Primary vs. Secondary:
  Primary indexes are built on the primary key, ensuring logical ordering of data.
  Secondary indexes exist for non-primary-key lookups (like email or status).

• Clustering:
  Determines whether data is stored physically in the same order as the index.
  (e.g., In MySQL’s InnoDB, the primary key is the clustered index.)

• Dense vs. Sparse:

  • Dense indexes store an entry for every search key value.

  • Sparse indexes store entries for blocks of data, reducing storage but slightly increasing lookup time.

Here’s how to design them with intent:

• Index columns that filter, join, or sort.
  Indexes shine when they reduce search space.
  If a column appears frequently in WHERE, JOIN, or ORDER BY clauses, it’s a good candidate.

• Avoid indexing volatile columns.
  Every index must be updated on every write.
  Indexing frequently changing columns (like status, last_updated, or login_count) can double your write cost.

Composite indexes matter.
When queries filter on multiple columns together, a composite index performs better than separate ones.
But order matters; always align the index order with your query’s filter order.

• Understand the “Leftmost Prefix” rule.
  A composite index only helps when the query filters on the leftmost columns of the index definition.
  If you skip the first column, the index can’t be fully used.

• Use covering indexes for hot paths.
  A covering index contains all columns required by a query, allowing the database to serve it directly from the index without touching the base table.

• Watch out for index bloat.
  Indexes take space, slow writes, and can grow faster than the data itself.
  Periodically audit and drop unused indexes.
  Tools like pg_stat_user_indexes (Postgres) or sys.dm_db_index_usage_stats (SQL Server) can show you which ones actually get used.

• Use specialized indexes when needed.

  • GIN/GiST indexes for JSON, arrays, and full-text search (Postgres).

  • HASH indexes for equality lookups on simple keys.

  • BRIN indexes for large, sequential data (e.g., logs or time-series).

  The right index type can outperform dozens of generic ones.

A good index strategy is like caching: the wrong one helps no one, the right one changes everything.

Indexes are not about adding speed; they’re about removing unnecessary work.
Each index is a contract; a trade-off between faster reads and slower writes.
Treat them as investments, not defaults.

4. Enforce Integrity with Constraints

Most performance problems aren’t caused by the database doing too much; they’re caused by it trusting you too much.

When you skip constraints, you shift data validation from the database to the application layer.

The result?
Redundant checks, unpredictable data, and queries that have to work harder to clean up your mess.

Constraints are the foundation of both performance and correctness; because a database that knows the rules can optimize around them.

Here’s how each one helps:

• PRIMARY KEY → guarantees identity and access speed.
  It enforces uniqueness and creates a clustered index (in most engines), which improves range scans and lookup consistency.

• FOREIGN KEY → ensures relationships and prevents orphans.
  When properly indexed, foreign keys actually improve join performance by guaranteeing referential integrity; the optimizer doesn’t have to plan for missing matches.

• NOT NULL → simplifies query plans.
  When a column is guaranteed to be non-null, the optimizer can skip conditional branches and null checks, producing faster execution plans.

• UNIQUE → enforces business rules at scale.
  It prevents duplicates early and ensures consistency without expensive application-side lookups or retries.

• CHECK → enforces domain rules early.
  These simple guards (e.g., CHECK (age >= 0)) stop invalid data from ever being written, reducing downstream complexity and debugging time.

Constraints aren’t about restriction. They’re about communication; telling the database what’s impossible so it can stop checking for it.

Proper constraints lead to:

• Smaller, cleaner indexes

• Faster joins (less null logic and filtering)

• Fewer defensive checks in code

• Safer migrations and refactors

The only valid reason to skip a constraint is when the database physically can’t enforce it (e.g., external dependencies, event streams).
Otherwise, you’re just outsourcing correctness to a place with fewer guarantees and more bugs.

5. Partition What Grows Fast

Even a perfect schema will choke under scale if everything lives in one table.
Indexes get bloated, queries slow down, and maintenance becomes painful.

Partitioning is how you keep growth predictable.
It doesn’t make your database magically faster; it makes it stay fast as data volume explodes.

There are two main flavors:

Horizontal Partitioning (Sharding):

Split a large table into smaller ones that share the same schema but hold different rows.
For example, you might partition orders by region, tenant, or date range.

This keeps each partition small and manageable, so queries only scan what’s relevant.
In systems like Postgres, native table partitioning can automatically route inserts to the right partition and prune irrelevant ones during queries.

Time-based partitioning is especially common. It keeps operational queries fast and archival data isolated.

Vertical Partitioning:

Split columns across tables to separate “hot” from “cold” data.
Frequently accessed columns stay in the main table; large or rarely used columns move to a secondary table.

This keeps your main table lean, smaller I/O, faster scans, and cheaper caching.

When to Partition

Partitioning adds complexity, so don’t reach for it too early.
Use it when:

• Your tables cross tens or hundreds of millions of rows.

• Maintenance operations (vacuuming, backups, deletes) start impacting uptime.

• You need data lifecycle management; archiving, pruning, or region-based compliance (e.g., GDPR).

• Query performance drops even with proper indexing.

How It Improves Performance

• Query Pruning: Only relevant partitions are scanned.

• Faster Maintenance: You can rebuild or drop partitions independently.

• Improved Caching: Smaller working sets fit better in memory.

• Parallelism: Queries can run across partitions concurrently.

Partitioning doesn’t make queries faster — it keeps them from getting slower as data grows.

The Trade-offs

Partitioning comes with operational cost:

• More tables to manage and monitor.

• Cross-partition queries can be slower if not designed carefully.

• Secondary indexes must be rebuilt for each partition.

• Application logic or ORM support can complicate inserts and reads.

When done right, though, it buys you years of runway before you need to consider database sharding or distributed systems.
Partitioning is scalability’s quiet superpower; invisible when it works, painful when it’s missing.

After Almost 20 Years Designing and Re-Designing Databases

What I’ve learned is simple: databases never forget your early mistakes.
A careless column type, a missing constraint, or an unnecessary denormalization will come back, just when your system is at its busiest.

Good schema design doesn’t just make things faster; it makes them last.

The teams that win long-term are the ones who treat table design with the same respect they give code.

Some Realistic Considerations

• Over-normalization can hurt read performance. Let query patterns guide final design.

• Indexes require maintenance. Rebuild periodically to prevent bloat.

• Constraints add minor write overhead, a small price for guaranteed correctness.

• Partitioning adds complexity. Keep boundaries simple and visible.

I’ve seen teams spend weeks chasing query plans when a single redesign could’ve fixed the issue forever.

If you take one thing away, let it be this:

Performance isn’t something you tune; it’s something you design.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

I remember the first time I tried it, we had a legacy service, and the idea of streaming database changes without touching the code felt like magic.

And while it works, there are trade-offs you should know before jumping in.

Here are 7 lessons I’ve learned the hard way:

IcePanel’s a collaborative diagramming and modelling tool for designing software architecture. Create interactive and layered views for different stakeholders, based on a single source of truth.

Try it for free

1. Event Granularity: Business Fact vs. Domain Model

CDC streams are tied to tables, not to business events.

Imagine a customer places an order:

• orders table gets a new row.

• order_items table inserts multiple rows.

• payments table updates with a pending transaction.

CDC faithfully emits three different low-level events.

But what consumers actually wanted was a single, meaningful business event:

{
  “event”: “OrderPlaced”,
  “orderId”: “123”,
  “total”: 89.50,
  “items”: [ ... ]
}

Instead, they get a stream of row changes and have to reconstruct the business fact themselves. This adds complexity, invites inconsistent interpretations, and turns your supposedly decoupled consumers into fragile data archaeologists.

2. Database Schema Coupling

Your schema becomes your contract.

• Merge two tables? Consumers, depending on the old streams, break.

• Split one table into many? You may need to create new streams, forcing everyone to update.

A schema change that should be local to a team now requires cross-team coordination. The more consumers you have, the harder this becomes.

The result: your producer service loses autonomy. Every schema tweak becomes a release-management nightmare.

3. Sensitive Data Exposure

CDC publishes the full row by default. That often includes fields like:

• Credit card numbers

• Social Security numbers

• Internal audit fields

Now every downstream consumer sees them.

You can filter or encrypt, but that creates a new problem: some consumers need that data, some don’t. Which means custom filtering logic, duplicated pipelines, or complicated access policies.

Instead of protecting sensitive fields at the source, CDC spreads the risk system-wide.

4. Handling Relational Data

Most real-world domains aren’t flat. They have relationships.

• Orders belong to Customers.

• Orders contain Items.

• Items reference Products.

CDC splits these into different streams (orders, customers, products). Consumers now have to:

• Subscribe to multiple streams.

• Perform joins.

• Reconstruct context before doing anything useful.

Each consumer ends up reimplementing business logic that should have stayed in the producer service. This increases bugs, documentation debt, and wasted effort.

5. Configuration Complexity

CDC isn’t “flip a switch and done.”

You have to configure:

• Which tables to capture.

• How to serialize events.

• What schema format to use.

• How to handle deletes and updates.

Misconfigure one setting, and you risk malformed events or silent data loss. Consumers will keep processing, unaware they’re working with incomplete or corrupted data.

6. Database Migration Challenges

Switch databases, say, from MySQL to PostgreSQL, and you’ll quickly discover that CDC tools don’t guarantee consistent event formats.

Column types differ. Timestamps look different. Even the order of fields can change.

That means every consumer must adapt. A migration that should be a producer concern suddenly becomes an organization-wide coordination project.

7. Operational Overhead

The CDC processor (Debezium, GoldenGate, etc.) isn’t free. It’s another critical moving part in your system.

• If it crashes, your event stream stalls.

• If it lags, consumers drift behind real time.

• If it silently drops changes, producers and consumers fall out of sync.

And then comes the biggest question: who owns it?

• The producer service team?

• The DevOps/SRE group?

• A shared data engineering team?

Without clear ownership and robust observability, CDC can become your weakest link.

The Hidden Complexity of CDC

On paper, CDC decouples producers from event publishing.
In practice, it couples consumers to the producer’s database design.

• Consumers deal with row-level noise, not meaningful business events.

• Schema changes ripple across every service.

• Sensitive data spreads far too easily.

• Operations become fragile and political.

What started as a shortcut to avoid writing event code ends up creating a web of dependencies that slows everyone down.

An Alternative Path: Transactional Outbox?

CDC has its place, but for most event-driven architectures, a Transactional Outbox is safer.

Instead of exposing table changes, the producer writes business events into an outbox table as part of its transaction. A relay process then publishes them to the event bus.

The result:

• Events are high-level facts (OrderPlaced), not schema leaks.

• Schema changes stay internal.

• Producers own their contracts.

• Consumers stay shielded.

Takeaways

• CDC feels “free,” but it’s expensive for consumers.

• It leaks domain internals, exposes sensitive data, and ties everyone to your schema.

• It creates operational risks with an extra moving part in your pipeline.

• For autonomy and clarity, favor business events via a Transactional Outbox.

• That said, CDC is still one of my favorite tech for Legacy integration and analytics.

Every row change is data. Every business fact is an event. Know the difference.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

They lack fast, trusted access to the right data.

And no, BI dashboards aren’t the fix.
They still rely on manual digging, tool switching, or someone from data engineering.

Uber’s answer?
They built an AI agent that speaks the finance team’s language, literally.

As developers increasingly turn to CLI coding agents like Claude Code for rapid development, a critical gap emerges: who reviews the AI-generated code?

CodeRabbit CLI fills this void by delivering senior-level code reviews directly in your terminal, creating a seamless workflow where code generation flows directly into automated validation. Review uncommitted changes, catch AI hallucinations, and get one-click fixes - all without leaving your command line.

It's the quality gate that makes autonomous coding truly possible, ensuring every line of AI-generated code meets production standards before it ships.

Install CodeRabbit CLI

Here’s the typical flow inside a medium/large company:

• Data lives in multiple sources (Presto, Oracle EPM, Excel, Google Sheets…)

• Naming conventions vary wildly (e.g. “US&C”, “GB”, “Q4 FY23”)

• The Analysts either:

  • Know SQL (and spend their time writing/debugging queries)

  • Or submit requests to data teams and wait 3 days (if you are lucky)

It’s slow, frustrating, and error-prone.

And when data access becomes the bottleneck, everything downstream suffers.

• Decisions are delayed.

• Errors creep in from manual lookups.

• Analysts burn time fetching numbers instead of interpreting them.

• BI tools become static graveyards.

This is not just inefficiency; it’s an opportunity lost. And Uber knew it couldn’t afford that.

Uber’s Answer: Finch

Instead of adding more dashboards, Uber built Finch, an AI agent inside Slack.

You ask in plain English:
“What was GB value in US&C in Q4 2024?”

Finch understands finance lingo (“GB” = Gross Bookings, “US&C” = region). It identifies the right data source, generates SQL, runs it, and returns the result in Slack. Large datasets export directly to Sheets.

No dashboards. No context switching. No waiting in line.

General Architecture

Finch isn’t a chatbot with shortcuts. It’s a system of specialized agents, data layers, and safeguards.

• A semantic layer maps user-friendly terms to schema fields.

• Curated marts provide optimized, reliable datasets.

• RBAC checks enforce strict access control.

• A multi-agent system separates intent detection, SQL generation, and validation.

• Results appear where users already work: Slack or Sheets.

How the Finch Data Agent Flow Works

It’s one thing to say “an AI agent runs queries for you.”
It’s another to see how the moving parts actually work together.

Finch isn’t a black box; it’s a coordinated system of agents, metadata lookups, and real-time feedback.

Every query travels through a well-defined pipeline: from Slack input → to intent detection → to SQL generation → to validated results delivered back to the analyst.

Understanding Finch’s power is easier when you see the flow.

1. User query input
   A finance analyst asks a question in Slack, e.g.
   “What was GBs in US&C last quarter?”

2. Agent orchestration
   The Supervisor Agent receives the query and decides which specialized sub-agent should handle it (e.g., SQL Writer Agent).

3. Metadata retrieval
   The sub-agent queries an OpenSearch index to fetch relevant metadata:

   • Synonyms and aliases for table/column names

   • Valid values for filters (e.g. “US&C” → region_name)

   This mapping step is critical: it lets Finch bridge messy human shorthand with rigid schemas.

4. SQL query construction and execution
   The SQL Writer Agent uses the retrieved metadata to dynamically build a precise SQL query and execute it against the right data mart.

5. Guardrails
   Validates security permissions before executing the query.

6. Result delivery
   The final result is returned directly in Slack, complete with:

   • Natural language explanation of the SQL mapping

   • Generated SQL (with comments)

   • A Google Sheets link with full results

The Trade-Offs

Of course, an AI agent isn’t magic. Uber had to navigate real risks:

• Schema drift: keeping metadata aligned as tables change.

• Trust: analysts wanting to see how answers were computed.

• Backend load: too many heavy queries hammering systems.

• Hallucinations: models inventing columns that don’t exist.

The mitigations included caching common queries, regression testing generated SQL, exporting large results outside Slack, and adding human oversight for executive-level data.

The design isn’t just about speed. It’s about trust, security, and usability.

Takeaways: How to Build Your Own Finch

If you want to build something like this for your org:

• Start with one team and a few high-leverage metrics (finance is a great use case)

• Build a semantic metadata layer (OpenSearch, Postgres, whatever) to store aliases

• Use curated marts to simplify routing and avoid complex joins

• Separate responsibilities across multiple agents (intent → SQL → validation)

• Focus on observability: log every query, allow feedback, measure success

• Don’t ship without RBAC and audit logging; data privacy is non-negotiable

Final Thought

Most dashboards are slow, static snapshots.
They were built for looking, not doing.

Finch points to the better path:

Ask the question.
Get the truth.
Make the move.

If analysts are still stuck writing SQL or digging through dashboards, it’s wasted potential. I think dashboards had their moment, but they’ve become a bottleneck.

The future isn’t more dashboards. It’s agents that deliver answers when you need them.

“Dashboards show you data. Agents move you forward.”

Until next time,
— Raul

Reference

https://www.uber.com/blog/unlocking-financial-insights-with-finch/

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

• Some users double-book.

• Others get stuck in “processing” forever.

• And your pager lights up at 2 AM.

That’s the cost of ignoring distributed transaction design.

You get choices. You don’t get them all.

Let’s talk about the Three Big Guns in distributed transactions:

1. Two-Phase Commit (2PC)

2. Saga with Orchestrator

3. Saga with Choreography

Each solves the same problem: keeping distributed workflows safe.
Each comes with brutal trade-offs.

As developers increasingly turn to CLI coding agents like Claude Code for rapid development, a critical gap emerges: who reviews the AI-generated code?

CodeRabbit CLI fills this void by delivering senior-level code reviews directly in your terminal, creating a seamless workflow where code generation flows directly into automated validation. Review uncommitted changes, catch AI hallucinations, and get one-click fixes - all without leaving your command line.

It’s the quality gate that makes autonomous coding truly possible, ensuring every line of AI-generated code meets production standards before it ships.

Install CodeRabbit CLI

Two-Phase Commit (2PC): The Illusion of Atomicity

2PC coordinates a distributed transaction by introducing a “transaction Manager/Coordinator.” Each participant (database, service, or resource) is asked to prepare first. If all of them vote yes, the manager sends a final commit message. If any vote no, everyone rolls back.

What it delivers:

• Strong consistency: either all commit, or none do.

• Simplicity in mental model; behaves like a local ACID transaction.

The cost:

• Adding more participants degrades performance. Every resource is locked until the final decision.

• Global locks block unrelated work. A single slow participant stalls the entire transaction.

• Coordinator failures leave transactions “in doubt,” requiring manual intervention or complex recovery logic.

• If the Coordinator crashes after prepare, but before broadcasting the decision, the entire algorithm is blocked until the Coordinator recovers.

It’s like asking every wedding guest to raise their glass for a toast at the exact same moment. The picture looks perfect, until one late uncle delays everyone.

When to use:
Small, short-lived transactions where correctness outweighs throughput, such as seat reservations or atomic money transfers.

When not to use:
Anything requiring scale across services or regions. Throughput will collapse.

Saga with Orchestrator: Control Through Compensation

An orchestrator does things differently. Instead of enforcing atomicity with locks, a Saga breaks a workflow into independent steps. A central orchestrator issues commands in order: “reserve inventory,” “charge payment,” “create shipment.” Each step has a compensating action (undo payment, restock inventory) if something fails.

What it delivers:

• Business-level atomicity. Failures unwind with compensations rather than strict rollbacks.

• Visibility: the orchestrator provides a timeline, retry logic, and consolidated monitoring.

• Reliability under flaky conditions: compensating actions make the system resilient to partial failures.

The cost:

• The orchestrator itself becomes a target for scaling and reliability.

• Every service must implement both “do” and “undo,” doubling the contract surface area and the testing burden.

• State persistence, retry policies, and orchestration logic are non-trivial engineering work.

Think of a travel agent. They book your flight, hotel, and car in sequence. If the hotel fails, they cancel the car and refund the flight. The customer never sees the failure; they just get a consistent experience.

It is perfect for complex, multi-step workflows that map to real-world business processes, like e-commerce orders or financial transactions.

Saga with Choreography: Autonomy at Scale

Choreography removes the central brain. Services publish events, and other services react. For example:

• Order Service publishes “Order Created.”

• Payment Service reacts and emits “Payment Completed.”

• Shipping Service reacts and schedules delivery.

  

The workflow emerges from the event flow rather than a central authority.

What it delivers:

• Loose coupling. Teams own services independently and deploy without coordination.

• Horizontal scalability. Adding new participants is as simple as subscribing to the event stream.

• High autonomy. Systems evolve faster because no single orchestrator dictates flow.

The cost:

• Eventual consistency. Data may take time to settle, and systems must be able to tolerate temporary divergence.

• Debugging becomes difficult. Tracking who triggered what and when requires strong observability.

• Each service manages its own compensations; it must know how to undo its own work.

• Adding more services increases the risks of event storms, feedback loops, and hidden dependencies.

It works really well for autonomy. Large-scale systems with autonomous teams, where independence and scale are more important than immediate consistency.

Practical Considerations Beyond the Patterns

Event Storms and Hidden Dependencies

Choreography scales by removing the central brain, but the price is unpredictability. Poorly designed event chains can create storms: one service reacts, triggers another, which triggers the first again.

The loop grows until queues overflow. This isn’t theory; it happens when services emit broad events like Order Updated without clear contracts.

The fix: design events narrowly, enforce idempotency, and monitor for abnormal event volume.

Observability and Debugging

Distributed workflows are only as reliable as your ability to see them. Without end-to-end tracing, you must guess which service failed and when. Use correlation IDs in every event. Centralize logs for compensations and retries.

Add distributed tracing to visualize entire sagas across services. And don’t stop there, automated anomaly detection can catch unusual retry patterns before they become outages.

Choosing the Right Pattern

No single approach wins everywhere.
2PC fits short, critical operations where correctness beats throughput.
Orchestrator sagas shine when workflows need predictable undo steps.
Choreography works when autonomy and scale outweigh strict consistency.

Most real systems mix patterns. Payments might use strong consistency, while downstream shipping notifications settle eventually.

Map requirements, data criticality, latency tolerance, traffic volume, to the pattern that fits.

Failure Recovery in Practice

Every system fails. Recovery is the differentiator.

2PC needs recovery logs and leader election to survive Coordinator crashes.

Sagas need thoroughly tested compensations and idempotent steps so retries don’t double-charge or restock twice.

Transactional outbox and idempotency keys are battle-tested tools that make retries safe and events reliable.

Where the Industry is Heading

Modern systems rarely stop at 2PC or Sagas. Patterns like CQRS, event sourcing, and the transactional outbox are becoming standard ways to balance consistency with scalability.

They don’t eliminate the trade-offs; you still pay, but they shift the costs to places you can manage more easily.

If your system is stretching, these are the next patterns to explore.

The Trade-offs You Cannot Escape

Each pattern buys you one benefit by forcing you to accept a cost.

Two-Phase Commit gives you the strongest consistency guarantees: either all participants commit, or none do. That makes it the safest option when correctness is non-negotiable.

But the cost is scalability. Global locks stall unrelated work, a single slow participant blocks the group, and cross-region use is nearly impossible. You trade throughput and resilience for atomic truth.

The Orchestrator Saga provides centralized control. The workflow is visible, compensations are predictable, and the system mirrors real business logic, refunds, cancellations, and restocks.

The downside is complexity. You must build and operate the orchestrator itself, and every service must support both forward and undo actions. The price of control is additional engineering overhead.

Choreography pushes autonomy and scale. Services react to events, teams ship independently, and you can add new capabilities without central coordination.

But you inherit the mess of distributed compensations and debugging event chains. Consistency becomes eventual, not immediate. The gain is speed and independence; the trade-off is clarity.

In short:

• 2PC → Strong guarantees, but poor scalability.

• Orchestrator Saga → Predictable rollbacks, but central complexity.

• Choreography Saga → Scalable autonomy, but messy debugging.

The real question is which cost you’re willing to pay: latency, complexity, or chaos.

Outages don’t come from picking the wrong pattern. They come from pretending trade-offs don’t exist

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

They come from bad judgment, caching the wrong thing, caching too soon, or caching without a clear reason.

You added Redis. Slapped a .get() around a slow query. Threw in a .set() after it. The app got faster.

So you did it again. And again. Until cache wrapped everything.

Now you've got problems no profiler will catch:

• Users see outdated prices

• Debugging is a coin toss. Was that from the DB or cache?

• Your memory footprint tripled, and eviction is eating the wrong keys

• No one knows which keys still matter, or if they ever did

This is what happens when caching becomes a reflex instead of a decision.

It's not that caching doesn't work. It's that most engineers cache before they think.

Caching decisions shouldn't be ad hoc. To avoid that, start here:

Now let's break it down, question by question.

As developers increasingly turn to CLI coding agents like Claude Code for rapid development, a critical gap emerges: who reviews the AI-generated code?

CodeRabbit CLI fills this void by delivering senior-level code reviews directly in your terminal, creating a seamless workflow where code generation flows directly into automated validation. Review uncommitted changes, catch AI hallucinations, and get one-click fixes - all without leaving your command line.

It's the quality gate that makes autonomous coding truly possible, ensuring every line of AI-generated code meets production standards before it ships.

Install CodeRabbit CLI

1. Is the data accessed frequently?

If a piece of data is rarely used, caching it doesn't help, it just takes up space.

But if it's hit constantly, by the same user or across many users, then caching saves real work.

Example:

Caching the homepage's product recommendations makes sense; they're requested thousands of times an hour.

But a one-time CSV export from an internal dashboard? That'll never be read again. Let it hit the database.

You don't batch-cook a meal no one orders. Cache is like a prep kitchen; only prep what is popular.

Simply put: the more traffic a key absorbs, the more return you get per byte of cache.

2. Is it expensive to retrieve?

Not all reads are equal. Some come cheap. Others hit external services ($$$), three databases, join ten tables, and compute summaries that cost real CPU.

That's where caching makes a difference.

Example:
Generating a user's analytics dashboard might involve multiple service calls and heavy aggregation. Cache the final payload.
But fetching a flat record by primary key from a well-indexed table? It's already fast. Skip the cache.

Cache is a shortcut. Use it where the path is long.

3. Is the data stable or volatile?

Stable data makes a great cache. It can sit there for minutes, even hours, without anyone noticing.
Volatile data? Not so much. If it's stale, it's wrong, and now you're in trouble.

Example:

The list of supported countries won't change mid-session. Cache it for days.

But stock levels on a flash-sale item? That can change every second. Caching it without real-time invalidation will cost you trust and money.

If you cache volatile data, you either need:

• A short TTL (Time-To-Live)

• Or an explicit invalidation hook (e.g., when the DB updates)

If you can't tolerate being a little wrong, don't cache it at all; or cache it with a plan.

Once you've decided to cache something, the next question is:
How do you keep it fresh?

You have two main tools:

• TTL (Time-To-Live): Set it and forget it. Let the data expire after a fixed time.

• Invalidation: Explicitly remove or update the cache when the data changes.

Each has trade-offs. Use the wrong one, and you'll either serve stale data or waste cache space.

Here's a simple matrix to help you choose the right approach based on how the data behaves:

4. Is the data small and simple?

How big is this data object? Is it cheap to serialize and deserialize?

Big, messy data doesn't belong in fast memory.

Large payloads eat up space, increase GC pressure, and slow down serialization and deserialization.
Small, flat data is faster to work with and easier to evict if needed.

Example:

A compact JSON with 10 fields → good candidate

A massive blob with 1,000 nested items → probably not

A classic antipattern is caching the entire product catalog (100K items) instead of caching paginated views or product summaries.

Cache is a fast-access shelf, not cold storage. Store what fits and what you'll grab often.

Small keys, simple shapes. That's how cache stays fast.

5. Does it directly impact user experience?

Not all latency matters. But when it does, it matters a lot.

Anything on the critical path of a user interaction, loading a page, rendering a component, or hitting “submit” should feel instant.
If caching makes that possible, use it there.

Example:

The response time for a search query or product page load directly affects conversion.
But a background sync task running at 2AM? No one's waiting on it.

Cache where speed is felt, not just where it's measured.

6. Is it safe to cache?

Fast is good. Leaky is not.

Caching user-specific or sensitive data, PII, tokens, and financials without scoping or encryption is a security risk. One bad key and someone sees what they shouldn't.

Examples:

A shared cache key for user:profile might accidentally leak another user's data in a multi-tenant app.
Always scope keys and apply access control if needed.

Mitigations:

• Use per-user or per-session cache keys

• Encrypt values when possible

• Set short TTLs for sensitive data

A rule of thumb: If it can't go in a log file, it doesn't belong in a shared cache.

7. Will this scale?

Caching that works for 1,000 users can collapse at 1 million.

Unbounded keys, high churn, or poorly managed TTLs can overwhelm memory, reduce hit ratios, and cause eviction storms.

Example:

A per-search query cache key (search?q=...) sounds harmless—until it creates millions of one-time keys.
Normalize input. Bound cardinality. Use sensible TTLs.

Tactics:

• Use eviction policies (LRU, LFU)

• Set size limits

• Monitor hit/miss ratio and eviction churn

Don't keep every conversation in memory, just the most recent ones.

These questions work well when you can reason through them case by case.
But if you want a quick gut-check, here's a mental model to remember:

If any of these are near zero, caching that data won't give you much back.
If all three are high, you're sitting on a high-leverage cache opportunity.

Final Takeaways

Caching works best when it's boring, predictable, scoped, and justified.
Not clever. Not magical. Just correct.

Here's what separates high-leverage caching from tech debt in disguise:

• Cache what's used frequently

• Cache what's expensive to fetch

• Cache what stays valid long enough to be worth it

• Cache what improves something a user can actually feel

• And never cache what you can't safely explain or scale

Every cached key is a liability until it proves its value.
Design each one with the same intent you'd give a database schema or API contract.

If you can't answer why something is in the cache, it probably shouldn't be.

In high-performing systems, caching isn't an afterthought.
It's a deliberate, visible part of the architecture, with trade-offs, constraints, and clear justification.

Smart systems don't cache more.
They cache better.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

That’s been the hidden flaw in conventional caching, especially in AI and search-heavy systems.
You can store the output. You can key it smartly. But unless the input exactly matches, you're back to a cache miss.

The result?

LLMs answering the same questions phrased 100 ways.
Search engines recomputing nearly identical vector queries.
Agents re-processing states they’ve already seen.

Caching by key doesn’t work when your users speak in meaning.

Redis 8 offers a serious upgrade: semantic caching, made practical with new native features like Vector Sets, LangCache, and to also help, Redis Flex.

Let’s break down the shift and what it unlocks.

Something I’m excited to share with you:
I’ve been working with a couple of friends on a platform designed to make us better engineers—one challenge at a time.

It’s called Join Enginuity.
Every day, you get a system design problem that forces you to think, research, and sharpen your instincts.

No fluff. No endless reading. Just practical challenges that make you better.

Join The Challenge Today!

The Caching Problem in LLM Systems

Caching works great when your input is stable:

GET /product?id=123

You cache product:123 and call it a day.

Now compare that to an LLM workload:

User 1: What is Redis LangCache?

User 2: Can you explain how LangCache works in Redis?

User 3: Tell me about Redis' new caching for LLMs.

Semantically identical.
But unless you normalize or fingerprint aggressively, the cache sees them as totally different.

That means every slight rephrasing triggers a full LLM call. You pay again. You wait again.
Same output, burned compute.

What Redis 8 Introduces (And Why It Matters)

1. Vector Sets – Native Vector Similarity in Redis

Previously, you’d need a bolt-on vector DB like Pinecone or FAISS. Redis had no efficient built-in support.

Now it does.

Imagine you have a collection of colored balls. Each ball has:

• A label (like "red ball" or "blue ball").

• A set of numbers describing it (its "vector", which could be like [1.0, 2.0, 3.0]).

• Extra info, like "size=large" or "owner=John".

What do Redis vector sets do?

• They store all these balls, each with their numbers and labels.

• You can ask Redis: "Which balls are most similar to this one?" — It compares the numbers (vectors) to find matches.

• You can also limit your search to certain balls (like only balls with "owner=John").

Why use Redis for this?

• It’s super fast at searching for similar items, which is great for things like recommendations, finding related products, or grouping similar texts/images.

Example in plain steps:

1. Add balls with their numbers and info.

2. Ask for the most similar balls to a given one.

3. Filter results by extra info ("only large balls").

4. Remove balls or check how many you have.

It’s just a clever way to quickly store and search for things that are "close" to each other, based on their numbers. Perfect for finding related items, instantly!

The following commands are available for vector sets:

2. LangCache – Caching for Meaning, Not Just Keys

LangCache is a fully managed semantic cache built on Redis 8.

It flips the caching model:

Instead of: cache[exact_string] = result

Now: cache[embedding] ≈ result

LangCache uses your existing embedding model (OpenAI, Cohere, etc.) to generate a vector from the prompt. Then it searches the cache for similar past entries. If it finds one, it skips the LLM and returns the cached output.

You just saved money and time, without losing correctness.

LangCache is available via REST API and is fully managed on Redis Cloud.

The request flow looks like this:

This is ideal for:

• Chatbots and support agents

• RAG pipelines

• Multi-step planning agents

• Any system with input redundancy

No key shaping. No hashing. Just raw meaning.

Use this formula to estimate savings:

Example: If your monthly LLM spend is $200, 60% of that is output tokens ($120), and you get a 50% hit rate, that’s $60 saved per month

3. Redis Flex – Cache More for Less

Caching gets expensive at scale, especially if everything lives in RAM.

Redis Flex lets you offload less-used data to SSD, increasing cache capacity up to 5x without a 5x cost bump.

RAM storage holds:

• All keys (names)

• Key indexes

• Dictionaries

• Hot data (working set)

RAM can handle hot data, while your SSD handles warm data. You control eviction policies, thresholds, and performance trade-offs.

Frequently accessed data stays fast in RAM; less-demanded data is offloaded to SSD automatically. Applications use the same Redis API and don’t need to change to take advantage.

Redis Flex is ideal when your:

• working set is significantly smaller than your dataset (high RAM hit rate)

• average key size is smaller than average value size (all key names are stored in RAM)

• most recent data is the most frequently used (high RAM hit rate)

Redis Flex is not recommended for:

• Long key names (all key names are stored in RAM)

• Broad access patterns (any value could be pulled into RAM)

• Large working sets (working set is stored in RAM)

• Frequently moved data (moving to and from RAM too often can impact performance)

Redis Flex is not intended to be used for persistent storage.

With Flex + LangCache, you can serve semantic cache hits at scale affordably.

Why This Matters (Beyond Buzzwords)

Let’s be blunt: most current AI architectures are duct-taped.

RAG pipelines re-embed the same documents.
Agents reprocess state with no memory.
Teams pay LLM costs repeatedly for cached knowledge.

The tools weren't ready until now.

Redis 8 changes that:

• You can store and search semantic data directly.

• You can cache LLM responses intelligently.

• You can scale caching affordably with hybrid RAM/SSD.

It’s not just faster. It’s a different model.

What Could Go Wrong?

Every system has trade-offs. Here are a few worth flagging:

LangCache offers controls for most of this: TTLs, thresholds, scoped caches, etc.

Still, it’s not fire-and-forget. You’ll need to test thresholds and observe cache hit rates.

Takeaways

• Conventional caches will break under semantic load. Redis 8 fixes that with new primitives.

• Vector Sets allow Redis to natively support vector similarity search.

• LangCache turns prompt caching from string-matching into meaning-matching.

• Redis Flex makes it scalable without blowing up costs.

• Semantic cache isn’t just nice-to-have—it’s the only way to avoid repeated LLM waste.

Final Thought

Caching isn’t dead.
But the old rules don’t hold when your inputs are probabilistic, fuzzy, or phrased 10 different ways.

The next generation of infrastructure needs to understand meaning, not just match keys.

Redis 8 is the first real step toward that future.

If you’re building anything with LLMs, embeddings, or real-time retrieval, this upgrade isn’t optional.

It's overdue.

Every LLM call you make that could’ve been a cache hit is tech debt you’re paying in real time.

Until next time,
— Raul

References:

https://redis.io/docs/latest/develop/whats-new/8-0/

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

And then a payment went through twice.
Two orders shipped.
The same email got sent three times.

Welcome to the ugly side of retries no one warns you about: duplication.

In distributed systems, At-Least-Once delivery is the default. Messages get retried. Requests get replayed. But if your app isn't ready for it, you end up with corrupted data, angry customers, and incomprehensible logic

This post breaks down four battle-tested strategies to handle duplication at the application level, so your systems stay correct, even when the same work shows up more than once.

Thanks to our partners who keep this newsletter free for the reader.

CodeRabbit → Free AI Code Reviews in VS Code

CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsurf. Get free, real-time feedback on every commit, before the PR, helping you catch bugs, security vulnerabilities, and performance issues early.

• Per-commit reviews: Identify issues faster and avoid lengthy PR reviews

• Context-aware analysis: Understand code changes deeply for more accurate feedback

• Fix with AI and get AI-driven suggestions to implement code changes

Multi-Layered Reviews: Benefit from code reviews both in your IDE (free) and in your PR (paid subscription)

Install in VS Code

The Problem: Same Message, Twice the Effect

Let's say a user submits a payment. The request goes through your API, but the backend crashes right before saving the result.

Your retry logic kicks in.

Now the same request is processed again. And you've:

• Charged the user twice.

• Created two payment records.

• Triggered two downstream events.

This isn't just bad luck. It's by design.

⚠️ At-least-once delivery is the default.
Most systems, especially queues, retries, and distributed APIs, guarantee that a message will arrive at least once, but may arrive multiple times.

Why? Because retries are safer than data loss.
It's your job to make sure duplicates don't break your system.

What's at Stake?

Without duplication control:

• Your data becomes unreliable. Systems that rely on exact state, like billing or inventory, get corrupted.

• Your code becomes defensive. Every handler needs if-checks, patches, or compensating logic.

• Your operations team burns out. They spend hours deduplicating rows, refunding customers, and triaging inconsistencies.

You don't just lose trust. You will lose time.

4 Ways to Stop Duplication at the Application Layer

These approaches aren't theoretical. They show up in real production systems, depending on scale, latency tolerance, and system design.

1. Database Unique Constraints

Concept: Use a UNIQUE constraint in the database to prevent duplicate operations, backed by an idempotency key.

Let's say a user places an order.
The client generates a unique idempotency_key and sends it with the request:

CREATE TABLE orders (
  id UUID PRIMARY KEY,
  idempotency_key TEXT UNIQUE,
  ...
);

The server attempts to insert the order using this key. If the same request is retried (e.g. due to a timeout), the database will reject the duplicate with a constraint violation.

You catch that error and fetch the original order.

✅ Pros

• Ensures only one successful insert per key. Potential performance impact from high-concurrency insert conflicts or deadlocks.

• Clean fallback path for retries, return the existing result.

• Ideal for operations like checkout, registration, or payment initiation.

⚠️ Cons

• Adds write pressure to the database. In

• Doesn't protect downstream effects unless guarded (e.g. emails, inventory).

• Needs good key generation hygiene on the client or gateway.

📌 Key insight: The database isn't just for persistence. It's a gatekeeper that protects your system from replayed requests, as long as you give it something to gate on.

2. In-Memory Deduplication

Concept: Track processed request IDs in memory using a Set, Map, or LRU cache. On each request, check if the ID was seen before.

It's fast. It's simple. But it's only safe if your service is single-instance and doesn't restart frequently.

Why it works: The process itself remembers what it's already done, no I/O needed.

✅ Pros

• Blazing fast.

• No infrastructure dependency.

• Great for short-lived tools and batch processes.

⚠️ Cons

• Volatile memory: everything is wiped on restart or crash.

• Doesn't work in multi-threaded or multi-node systems without coordination.

• You need to manually clean up old keys or use a TTL mechanism.

📌 Best for: One-off scripts, CLI tools, test environments, or internal utilities where reliability isn't mission-critical, monoliths with low/moderate operation frequency.

3. Distributed Cache (Redis)

Concept: Use Redis to track processed operations across distributed nodes. Store a key for each request ID with a TTL:

SETNX request:<id> "processed"    # Only set if not exists
EXPIRE request:<id> 600           # Auto-expire in 10 minutes

This ensures:

• Only one node processes the request.

• Future retries are blocked.

• The de-dup record eventually expires to free memory.

Why it works: Redis becomes a shared memory layer across services and instances.

✅ Pros

• Fast lookup and write.

• Survives restarts and horizontal scaling.

• TTL provides automatic cleanup.

⚠️ Cons

• TTL tuning is non-trivial. Too short and legit retries slip through; too long and Redis bloats.

• Network partitioning or Redis downtime means you may process duplicates.

• You'll need to deploy and monitor Redis reliably. It can become a single point of failure.

📌 Tradeoff tip:
Pair Redis deduplication with a fallback to DB uniqueness to catch any misses due to cache failures or TTL gaps.

📌 Best for: API services, job processors, microservices with retry logic.

4. Message Broker Deduplication

Concept: Some brokers (e.g., Azure Service Bus, Pulsar) support built-in duplicate detection.

You assign a MessageId to each message. The broker keeps a deduplication window (e.g. 10 minutes). If the same ID appears again, it discards the message.

Why it works: The message infrastructure becomes the gatekeeper; duplicates don't even hit your app.

✅ Pros

• No application logic required.

• Prevents duplicate delivery, not just processing.

• Works great with at-least-once brokers.

⚠️ Cons

• Requires proper configuration (de-dup window, clock sync).

• Doesn't help if your app publishes duplicate downstream events.

• Broker-specific — not portable across infra.

📌 Best for: Event-driven architectures, high-throughput pipelines, or systems with built-in broker support.

Trade-Off Comparison

Final Takeaways

• Duplication is a design reality, not a bug. Plan for it.

• At-least-once delivery means your systems will get the same input more than once. Act accordingly.

• Use the database for strong guarantees, Redis for fast coordination, and brokers for message-level protection.

• TTL-based deduplication is usually good enough. Start there.

• Combine layers (e.g., broker de-dup + Redis + DB constraint) for critical flows.

You can't prevent retries, but you can prevent rework.

Resilience without deduplication is just chaos with retries.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

Let's say your app submits a critical order update to your PostgreSQL database.

The API returns 200 OK. You move on.

But then the server crashes.

You restart the database and check the orders table. No record.

What happened?

It turns out, the database acknowledged your write before it had truly committed the data to disk.

That’s the pain point that Write-Ahead Logs (WAL) solve.

Ignoring WAL = Risky Assumptions

If a system writes data directly and crashes before syncing to disk:

• Records can be half-written or corrupted

• Acknowledged writes may vanish

• Recovery might leave the system inconsistent

This isn't just a rare edge case. It's a recurring risk in any stateful system. Imagine a payment system that marks an order as paid but loses the transaction record after a crash. Or a logistics platform that acknowledges a shipment but forgets to actually persist the dispatch event.

The higher your write throughput, the more frequently this risk shows up, and the more expensive the consequences.

But how does PostgreSQL solve this problem?

Thanks to our partners who keep this newsletter free for the reader.

CodeRabbit → Free AI Code Reviews in VS Code

CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsurf. Get free, real-time feedback on every commit, before the PR, helping you catch bugs, security vulnerabilities, and performance issues early.

• Per-commit reviews: Identify issues faster and avoid lengthy PR reviews

• Context-aware analysis: Understand code changes deeply for more accurate feedback

• Fix with AI and get AI-driven suggestions to implement code changes

Multi-Layered Reviews: Benefit from code reviews both in your IDE (free) and in your PR (paid subscription)

Install in VS Code

The solution was the WALs

Write-Ahead Logging flips the order of operations. Instead of writing data directly, the system logs the intent to change first. Let's see a classic example; I bet you’ve used it without knowing.

Here's the general flow (PostgreSQL example):

1. Receive a change request (e.g. INSERT)

2. Write the change to a WAL buffer

3. Flush the WAL buffer to disk (fsync)

4. Acknowledge the write to the client

5. Later, apply changes to the actual data files

Step 3 is the critical point. Until the WAL is safely flushed to disk, nothing is considered durable.

Nothing is acknowledged until the log is safely on disk.

Even if the system crashes right after step 3, PostgreSQL is able to replay the WAL on startup and restore the exact state before the crash.

This mechanism transforms potential data loss into a recoverable state.

WAL Is Your Database's Black Box

When a plane crashes, investigators use the flight recorder (aka "the black box") to reconstruct what happened.

WAL is the equivalent for your database.

It captures every intended change, insert, update, and delete in a sequential, append-only format.

Even if your heap or index pages are inconsistent due to a crash, the WAL gives the system a source of truth to rebuild from.

That's how PostgreSQL achieves ACID durability guarantees. Without it, every crash would be a data loss lottery.

What's Inside a WAL Entry?

In PostgreSQL, a WAL entry for an INSERT may include:

• Table/relation ID

• Page number and offset

• The tuple data being inserted

WAL entries are not SQL statements. They are low-level descriptions of the physical or logical changes needed to replay the operation.

This structure ensures that recovery doesn't require full query re-execution, just a scan of the WAL entries.

PostgreSQL stores WAL data in files typically 16MB in size by default. These files are named using a 24-character hexadecimal format like:

This name breaks down into:

• 00000001: Timeline ID

• 00000003: WAL segment log ID

• 00000065: Segment within the log

These filenames are critical for:

• Streaming replication (so replicas request the right segment)

• Archiving (for point-in-time recovery)

• Monitoring (to detect lag or growth)

If you see a folder full of these files growing rapidly, that's either a high write workload, a replication lag, a misconfiguration, or a broken archiving process. Keep an eye on it.

WAL Enables Replication

Because WAL is a clean, ordered log of all changes, it's ideal for streaming replication:

• PostgreSQL sends WAL segments to replicas over a replication connection

• Replicas replay the WAL to stay in sync

• MongoDB uses its oplog the same way

• Kafka takes this idea further; its entire architecture is built around logs

Instead of building separate change streams, systems reuse their WALs to power real-time data distribution.

Things That Can Break

WAL helps, but it's not bulletproof. Watch for:

• fsync disabled: Some configurations or test environments skip disk flushing, giving a false sense of safety.

• Slow disks: Commit performance is gated by how fast you can flush WALs.

• WAL bloat: Without cleanup or archiving, logs can consume large amounts of storage.

• Missing archive strategy: For point-in-time recovery, you must safely copy old WAL segments to backup storage.

WAL gives you a recovery plan. But it only works if you configure and monitor it properly.

Key Takeaways

• WAL logs every change before applying it, which ensures durability and crash safety

• Writes are acknowledged only after the log hits persistent storage

• WAL is used for crash recovery, replication, and backup

• PostgreSQL, MongoDB, Kafka, and many other systems rely on WAL-like designs

• You must verify your environment uses fsync, manages disk I/O, and handles WAL archiving

Once you start to see your system as a stream of logged changes, replication, failover, and recovery become easier to reason about.

WALs aren't just a recovery mechanism; they're a design principle.

That's why your database writes everything twice.

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

And so is high availability.

If your system stays up during failure, it’s not because the universe smiled on you.
It’s because you made design choices, failover and replication, that absorbed the hit.

Availability isn’t magic. It’s math and architecture.

To design for it, you need to understand two ideas:

• Failover is about keeping the service up when a node dies.

• Replication is about making sure the data is still there and is fast to access.

They both solve different problems, but both are necessary and must work together. Let’s break down some key design patterns and their trade-offs for both so you can choose the right one before your pager buzzes.

They solve different problems but work together.

Thanks to our partners who keep this newsletter free for the reader.

CodeRabbit → Free AI Code Reviews in VS Code

CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsurf. Get free, real-time feedback on every commit, before the PR, helping you catch bugs, security vulnerabilities, and performance issues early.

• Per-commit reviews: Identify issues faster and avoid lengthy PR reviews

• Context-aware analysis: Understand code changes deeply for more accurate feedback

• Fix with AI and get AI-driven suggestions to implement code changes

Multi-Layered Reviews: Benefit from code reviews both in your IDE (free) and in your PR (paid subscription)

Install in VS Code

Failover Patterns

Failover is your answer to system failure.

When a node crashes or a zone goes dark, what takes over?

Failover is your solution to the problem: keeping the service running when the primary component becomes unavailable. It’s about redundancy, having backup instances that can step in without user impact. You don’t want downtime, so you need a standby.

It’s not about preventing failure. It’s about absorbing it with minimal disruption.

There are two main patterns to implement Failover:

1. Active-Active: All instances are live and share the load.

2. Active-Passive: One instance handles traffic, others stay on standby.

The choice between these two affects everything: latency, cost, failover time, operational complexity.

You don’t want downtime. So you need a standby.

1. Active-Active Failover

In an active-active setup, all nodes are live and actively serve requests.
They share traffic through a load balancer or service mesh. If one node fails, the others continue operating, no warm-up, no handoff.

This pattern gives you high availability and high throughput, but comes at the cost of complex coordination:

• You need consistent state across nodes, or idempotent operations if state sync lags.

• Your load balancer must detect unhealthy nodes and remove them from rotation fast.

• If nodes write to shared resources (like a DB), ensure concurrency safety and conflict handling.

✅ Pros

• No downtime on node failure

• Scales horizontally under load

• Better resource utilization

✘ Cons

• Requires robust load balancing

• Harder to manage stateful workloads

• Can mask underlying issues due to redundancy

📌 Use when:
You need zero-downtime failover and your service can tolerate or handle eventual consistency between nodes. Ideal for stateless APIs, edge services, and front-door layers.

Take into account this pattern maximizes uptime and capacity, but you’re also paying for that uptime 24/7.

2. Active-Passive Failover

In an active-passive setup, only one node serves requests.
One or more standby nodes remain idle until the active fails, then a failover mechanism promotes a standby to become active.

This pattern favors simplicity and consistency over speed of recovery.

But failover isn’t instant. Depending on detection and promotion logic (manual, DNS TTL, heartbeat monitoring), recovery can take seconds to minutes.

✅ Pros

• Simple to wrap you head around

• Easier to implement for stateful systems

• Reduced risk of data conflicts

✘ Cons

• Downtime during failover

• Backup node is underutilized

• Potential for data loss if replication lags

📌 Use when:
Failover speed is less critical than data correctness and operational simplicity. Common in primary-standby databases, DR setups, and tightly coupled monoliths.

3. Hybrid Active-Passive Failover

You don’t always have to choose between fully active or fully idle systems.

A hybrid model combines multiple active nodes with one or more passive backups.
For example, two nodes actively serve traffic, and a third is passive, ready to take over if either fails

This setup gives you the throughput and fault tolerance of active-active,
but with lower cost and simpler coordination, because not every node is participating at all times.

You’ve got:

• Redundancy from the passive instance

• Load distribution across the active ones

• Faster failover than cold standby

• Lower cost than scaling all nodes equally

✅ Pros

• Load is shared across fewer active nodes → less infra cost

• Passive node offers quick recovery without full duplication

• Easier to understand than full active-active with N nodes

✘ Cons

• Still requires sync between active and passive

• Uneven load may bottleneck if one active fails

• Failover logic must detect and rebalance

💰 Cost Consideration:

• Lower than full active-active

• Better utilization than pure passive

• Scales better as demand grows

📌 Use when:
You want to reduce infra spend while maintaining high availability and good performance.
Ideal for medium-scale services where failover speed matters, but cost discipline is required.

Failover can keep your services available when components fail.
But that’s only half the story.

To keep your data available and consistent, you need replication.

Let’s shift focus from infrastructure failover to data durability and distribution, and explore the core replication strategies.

Replication Patterns

Failover helps you survive infrastructure failure.
Replication helps you survive data loss and scale reads.

Replication means keeping multiple copies of your data in sync across nodes.
If one copy is lost, another can take over.
If your read traffic spikes, replicas can help offload the pressure.

There are two primary replication strategies:

• Single-leader replication: one node handles all writes

• Multileader replication: multiple nodes accept writes

Let’s break them down.

1. Single-Leader Replication

In this pattern, only one node accepts writes (the leader).
One or more read replicas sync from it asynchronously.

Clients can read from the leader or any replica, but all writes must go to the leader.

✅ Pros

• Simpler to conceptualize

• Strong write consistency

• Works well with read-heavy workloads

✘ Cons

• Single point of failure (without external failover logic)

• Replication lag → stale reads

• Leader can become a write bottleneck under heavy load

• Failover requires promotion and reconfiguration

📌 Use when:
You care about data consistency, simplicity, and read scalability.
Common in OLTP systems, dashboards, and APIs where write throughput is moderate.

2. Multileader Replication

In a multileader setup, multiple nodes can accept writes.
Each node replicates its changes to the others.

Write locally, sync globally.

Multileader setups are eventually consistent by default.
You’ll need logic to resolve conflicts, like "last write wins", version vectors, or CRDTs.

✅ Pros

• High availability across regions

• Local write performance (reduced latency for global users)

• Avoids a single point of failure

✘ Cons

• Complex conflict resolution

• Increased write latency due to cross-node sync

• Harder to figure out the data state

• Greater operational overhead

📌 Use when:
You need multi-region write availability, or your system must remain writable even during partial outages.
Used in collaborative tools, messaging platforms, and edge-aware databases.

Bonus: Leaderless Replication (For Context)

Some systems (e.g., Cassandra, Dynamo) use leaderless replication:
Clients write to multiple nodes directly, and consistency is managed via quorum reads/writes.

Trade-offs vary, but typically:

• Very high availability

• Eventual consistency unless tuned carefully

• Complex consistency guarantees

You likely won’t build this yourself, but you should at the very least understand it when choosing infrastructure.

Trade-offs in a table

Final Thought

Systems don't become highly available by accident.
They become resilient through intentional design.

• Failover absorbs failure.

• Replication protects data.

Here's what that means for you:

1. Don’t assume replication means zero data loss.

2. Don’t assume failover means zero downtime.

3. Don't pick a pattern based on popularity—pick based on recovery time, latency tolerance, and consistency needs.

Good architecture isn't about uptime. It's about controlled failure.

If you don't understand the trade-offs, you're not as "available" as you think.

What’s one thing your system can’t recover from today?

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

It instead lies hidden in the coordination.

And nothing makes coordination harder than distributed transactions.

What Are Distributed Transactions?

A distributed transaction is a single logical operation that spans multiple services or databases. You typically see this in microservices, where multiple components need to work together to fulfill a single user action.

For example, placing an order might involve:

• Charging a payment method (Payments Service)

• Reserving stock (Inventory Service)

• Scheduling delivery (Shipping Service)

Ideally, these should either all succeed or all roll back. But in distributed systems, that's far easier said than done. The moment you introduce separate services and network calls, you set the stage for failures, latency spikes, retries, and only partial successes.

The Challenge with Traditional Transactions

In monoliths, you might solve this with a database transaction (ACID guarantees). But in distributed systems, two-phase commits (2PC) become risky, hard to scale, brittle under failure, and expensive in performance.

You're left with a painful tradeoff: consistency vs availability.

That's where sagas come in.

👉 Quick sidenote: if you're gluing together retries, compensations, and half-broken workflows…

You might want to check out Kestra.

It's an open-source, event-driven orchestration platform that makes it way easier to build, monitor, and scale complex flows, including distributed ones like SAGAs. You write in YAML, plug into your existing stack (Terraform, Slack, DBs, etc.), and get full visibility when things go sideways.

I've been playing with it lately and it just… clicks.

Check their open-source repo here

They already had my ⭐️.

What Is a Saga?

A Saga is a design pattern for managing long-running distributed transactions through a sequence of local transactions, coordinated with events or commands.

Instead of relying on locks or global transactions, each service performs its own local operation, and if something fails later, a compensating transaction is invoked to undo previous steps.

In other words:
Forward-only steps, with explicit undo logic.

🧭 Orchestrated Sagas: The Central Brain

In an orchestrated saga, a central coordinator (called the orchestrator) defines and manages the order of operations.

Think of it like a conductor:
“Service A: place order.” → “Service B: charge payment.” → “Service C: ship package.”

The orchestrator explicitly calls each step. If anything fails, the orchestrator triggers the appropriate compensation, like cancelling the order or refunding the payment.

✅ Pros:

• Clear control flow and centralized logic

• Easier to test and debug

• Easier to visualize

❌ Cons:

• Orchestrator becomes a single point of failure

• Can become a coordination bottleneck between teams

• Slower to evolve when different teams own services

Real-world example:
An e-commerce company I worked with used orchestrated sagas for order management. It worked well until the orchestrator failed under holiday load. Half-completed orders piled up, and coordination across services broke down.

The orchestrator was technically fine, but organizationally fragile.

🔀 Choreographed Sagas: Domain Events + Distributed Ownership

In choreographed sagas, there's no central conductor. Each service listens for and emits events to progress the flow.

Service A publishes “Order Created.”
Service B listens and emits “Inventory Reserved.”
Service C listens and emits “Order Shipped.”

No central logic, just a chain of reactions.

✅ Pros:

• Team autonomy: Each team owns its part.

• Scales across domains: No need for central coordination.

• Loose coupling: Easier to evolve services independently.

❌ Cons:

• Flow invisibility: Harder to see the full process in one place.

• Complex compensation: Rollbacks require scattered logic.

• Harder to test end-to-end without observability tools.

Observability: The Missing Map

When you're dealing with distributed sagas, especially choreographed ones, observability isn't optional. It's how you make sense of chaos.

You need to answer:

• What triggered this flow?

• Which services ran, in what order?

• Where did it break?

• And what needs to be compensated?

Recommended Tools & Practices:

• Distributed tracing with OpenTelemetry, Jaeger, or Honeycomb:
  Trace execution across services end-to-end.

• Structured logging with correlation IDs:
  Pass trace IDs in every request to stitch logs into a coherent narrative.

• Centralized log aggregation (ELK, Loki, Datadog):
  Keep everything searchable across services, regions, and retries.

• Custom dashboards for saga metrics:
  Monitor failure rates, compensation triggers, and flow durations.

Without visibility, sagas devolve into distributed guesswork.

A Quick Cheatsheet

Here is how I decide which one to use

Personally, I've leaned more toward choreographed sagas, especially when the business process spans multiple teams. It's not perfect, but the autonomy and flexibility it gives outweigh the tracing complexity (most days 😅).

That said, orchestrated sagas can be a great choice when one team owns the entire process and wants clarity over autonomy.

One final tip: You can combine them by orchestrating critical paths (e.g., payments) with choreography for non-critical actions (e.g., notifications).

So… how do you handle distributed transactions?

Do you lean toward orchestration, choreography, or try to avoid the problem entirely through event stores, retries, or outbox patterns?

Hit reply or drop a comment, I'd love to hear your war stories.

“Most complexity isn't in code or data — it's in the space between them.”

Until next time,
— Raul

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

But here’s what really happens in most systems:

• Messages fail.

• They get moved to a DLQ.

• No one notices.

• No one investigates.

• Messages expire.

• And the failure becomes permanent.

Let’s break down why this happens and how to fix it.

Cut Code Review Time & Bugs in Half

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 5 million PRs, installed on 1 million repositories, and used by 50 thousand Open-source projects. CodeRabbit is free for all open-source repo's.

Get Started Today

❌ Problem 1: No Monitoring = No Recovery

Most DLQs are unmonitored.

There’s no alert when messages arrive. No dashboard tracking queue size. No process to investigate what failed and why.

Failures go unnoticed for days or weeks, until someone stumbles across missing data or a downstream report looks off.

By then, it’s often too late.

Fix:
Set up metrics for DLQ depth and message age. Trigger alerts when thresholds are breached. Make the DLQ part of your system health dashboard, not a hidden corner.

❌ Problem 2: Blind Redrives Create Loops

Some teams automate the reprocessing of messages from the DLQ.
It sounds efficient, until it’s not.

Redriving a failed message without understanding why it failed is asking for trouble.

• If the downstream service is still down, it will fail again.

• If the payload format changed, it will fail again.

• If your logic is flawed, it might break more than just that message.

Some systems allow messages to bounce between the main queue and the DLQ indefinitely. Others eventually discard them after repeated failure.

Fix:
Don’t redrive blindly. Attach metadata when a message is moved to the DLQ: error type, timestamp, originating service, trace ID.
Redrive only after the root cause has been addressed.

❌ Problem 3: Messages Disappear Without Warning

In systems like AWS SQS, messages in the DLQ are deleted after 14 days.
That’s not configurable.

If no one processes them by then, they’re gone forever.
No reprocessing. No audit trail. No accountability.

Fix:
Track the age of DLQ messages. Alert when messages approach the expiration threshold. Set a regular cadence, weekly or even daily, for reviewing and draining DLQs.

❌ Problem 4: DLQs Flatten Context

DLQs often act as a catch-all. Every failure, regardless of cause, ends up in the same place, with no context.

But not all failures are equal:

• Some are malformed payloads.

• Some are caused by missing downstream services.

• Some are just business rule violations.

And if you treat them all the same, you make triage harder and slower.

Fix:
Tag DLQ messages with structured metadata:

• Failure category

• Originating service

• Timestamp

• Retry count

• Root cause summary (if available)

This turns your DLQ from a black hole into an observable failure pipeline.

Most systems—like AWS SQS, Azure Service Bus, Kafka, RabbitMQ—automatically move messages to a DLQ after the max retry threshold is hit. But:

❌ They don’t let you customize the DLQ message payload during that move.
✅ They only move the original message, untouched.

So if you want to include useful debugging metadata:

• You must intercept the failure and publish to the DLQ yourself.

• You must implement the logic to inspect, validate, and retry the failed messages.

These are tradeoffs you have to take into account!

⚖️ The Tradeoffs: DLQ Pros and Cons

A DLQ is a safety valve, not a safety net.
It gives you a buffer, but only if you actively manage it.

✅ What Good DLQ Practice Looks Like

Here’s what a solid DLQ process includes:

• Monitoring & Alerting
  Real-time metrics on DLQ size, age, and throughput.

• Triage Automation
  Automatically classify failure reasons and separate transient from permanent.

• Structured Logging
  Include trace IDs, service names, error codes, and timestamps in every failed message.

• Human Ownership
  Assign a team to own the DLQ and act on it within defined SLAs.

• Safe Reprocessing
  Tools that deduplicate, validate, and replay only when the root cause is resolved.

The question is not “Do you have a DLQ?”
It’s “Do you actually know what’s in it, and are you doing something about it?”

Final Thoughts

A DLQ shows you where your system is failing.
But only if you’re looking.

Most teams aren’t.

They quietly lose data, miss signals, and assume everything is fine until a customer reports something missing, and no one knows why.

DLQs are not backups.
They are broken promises waiting to be discovered.
Own them. Monitor them. Triage them. Fix the root causes.

That’s how you build resilience, for real.

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

But you’ve already used a distributed system this week when you ordered coffee.

Even a coffee shop mirrors core principles of system design: APIs, queues, event-driven processing, bottlenecks, notifications, and more.

Let’s walk through it, step by step.

Your Code Can Shape the Future of AI; and Pay Up to $50/hr

Outlier is hiring top software engineers to help improve the world’s leading AI models.

It’s freelance, flexible, and remote.

👉 Review and critique AI-generated code.
👉 Use your expertise to raise the bar on code quality.
👉 No AI experience needed—just solid engineering skills.

If you’ve built real systems, know your way around Git, and love clean code, this is the smartest side hustle you’ll find.

GET PAID TO WRITE AND REVIEW CODE

1. The Cashier = API Gateway

The cashier is the system’s entry point.

You place a request:

“One oat milk cappuccino, no sugar.”

The cashier validates it, logs it, and returns a response:

“You’re order #37. Please wait to be called.”

That’s exactly how an API works:

• Takes in a structured request.

• Validates and transforms it.

• Sends back an acknowledgment (order ID).

This is synchronous, stateless, and isolated from the backend work.

📌 Design Insight: APIs should validate and log input early, then hand off work to background processes.

2. The Order Queue = Message Queue

Once your order is accepted, it goes into a queue for preparation.
Orders are processed in first-in, first-out (FIFO) order.

But here’s the catch:
Even if orders arrive in order, processing time varies.

A black coffee takes 1 minute.
An iced caramel macchiato with almond milk might take 5.

So even if you're earlier in the queue, someone after you might get their order first.

📌 Design Insight: Queues buffer load, but they don’t guarantee equal wait times. Task complexity affects throughput.

3. Baristas = Event Consumers

Baristas don’t wait for orders to be assigned.

They listen for new orders, then pull one off the queue when ready.
This is event-driven architecture: workers subscribe to a stream of tasks and pick them up independently.

More baristas = higher parallelism = better throughput.

No need to coordinate who does what. Each instance is stateless and autonomous.

📌 Design Insight: Event-driven systems are more scalable and loosely coupled.
You can scale consumers horizontally with minimal coordination.

4. Espresso Machines = Shared Resource Bottleneck

Each barista can’t make every drink right away.

Espresso drinks rely on shared machines—typically just one or two in the shop.

That introduces resource contention.
Even if you have 5 baristas, only 2 drinks can be brewed simultaneously.

This is like:

• Database connection limits

• CPU-bound microservices

• Shared memory or I/O locks

📌 Design Insight: Systems must manage contention and concurrency carefully.
Shared resources need pooling, backpressure, or load shedding.

5. Order Ready = Push Notification

Customers don’t walk up every 30 seconds asking, “Is my drink ready?”

Instead, the barista calls your name (or order number) when it’s done.

This is push-based notification, far better than polling.

Polling wastes bandwidth and computing resources. Push updates are efficient and timely.

📌 Design Insight: Use push notifications (WebSockets, SSE, Pub/Sub) over polling where possible, especially for status updates or async task completions.

Real-World Considerations

Distributed systems aren't perfect. Coffee shops aren't either.

Here’s what can go wrong:

• Load spikes: Morning rush = write spike. Queue fills up, latency increases.

• Skewed processing: One complex order delays others. Impacts fairness.

• Dropped task: Barista forgets an order. You need retry or reconciliation logic.

• Backpressure: When the queue is too long, the system should throttle intake.

All of these require planning, just like in software systems.

Final Thought

Distributed systems don’t start in the cloud.
They start with understanding how work flows, where it gets stuck, and who needs to be notified.

You don’t need Kubernetes to learn system design. You just need to watch how a coffee shop works under pressure.

Next time you're in line, look around and ask:

• Where’s the queue?

• What’s async?

• What’s the bottleneck?

• What breaks under load?

If you can model a coffee shop, you can model a backend system.

System design isn’t theory; it’s observation. And the world is full of working systems.

It’s all there, hiding in plain sight.

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.

Subscribe now

Then everything breaks.

You’ve seen it:

• Double charges in payment systems.

• Lost edits in collaborative tools.

• Inventory systems that sell stock that’s no longer there.

These aren’t edge cases. They’re the reality of modern software and will break your business if you ignore them.

Want to build real-world voice automation with serious engineering behind it?

Telnyx Voice AI gives you full-stack tools to launch voice assistants that feel fast, human, and production-ready.

• Build AI-powered call flows with real-time transcription, webhook logic, and natural language understanding.

• Bring your own model or use native tools — no vendor lock-in, full control.

• Scale instantly with built-in global telephony, low-latency infrastructure, and seamless fallback to human agents.

Thank you to Telnyx for sponsoring this newsletter and helping us keep it free!

Start Building Today!

What’s at Stake?

If you don't handle consistency properly:

1. You lose data,

2. You break trust,

3. And you create a system that’s impossible to debug.

But consistency is hard, especially at scale. You're working with multiple nodes and services, often across regions. Distributed writes mean several systems may try to update the same data at the same time. Throw in unreliable networks, retries, and partial failures, and suddenly, maintaining a single source of truth becomes a serious challenge.

Despite the complexity, there are three ways to get consistency right:

1. Write-Ahead Logging (WAL)

Used by: PostgreSQL, MySQL, etc.

How it works:

If the system crashes in the middle, you replay the log. That gets you back to a consistent state.

✅ Durable and safe.
❌ Doesn’t handle concurrent write conflicts.

2. Locking (Pessimistic Concurrency)

Used by: Traditional RDBMS

How it works:

✅ Prevents conflicting updates.

❌ Doesn’t scale. Blocks fast paths. Poor fit for distributed systems.

3. Data Versioning (Optimistic Concurrency)

Used by: DynamoDB, Cassandra, Event Sourcing

How it works:

No locks. Just version checks.

✅ Scales well. High throughput.
❌ You must handle retries and merge conflicts.

Case Study: How DynamoDB Gets Consistency Right

Let’s say you're building a diagram collaborative platform where teams co-edit architecture diagrams in real time.

Two users, A and B, open the same diagram. User A moves a component and writes it to the database (version 1). At the same time, User B renames that component and also tries to write (still thinking it's version 1).

Without coordination, whichever writer hits the database last will silently overwrite the other, resulting in either the position change or the rename being lost.

DynamoDB avoids this with Conditional Writes:

**ConditionExpression** — a clause in DynamoDB that ensures an update only occurs if a specified condition (like a matching version) is true: "version = :expectedVersion"

Key DynamoDB Features Used

1. ConditionExpression
   Enforces that writes only succeed if the item’s current version matches the client’s expected version.

2. Atomic Counters
   The version attribute acts as an atomic counter, ensuring sequential updates.

3. Optimistic Locking
   Clients assume no conflicts by default but handle them gracefully when they occur.

Why This Works

• Prevents Silent Overwrites: Only one writer succeeds per version, eliminating "last write wins" chaos.

• Merge Flexibility: Clients can implement domain-specific merge logic (e.g., Operational Transform for diagrams).

• Scalability: No centralized locking, making it suitable for globally distributed teams.

Trade-offs in Practice

Tools & Frameworks

• Write-Ahead Logging: Native in PostgreSQL, MySQL, and SQLite

• Locking: Handled via transaction isolation levels (e.g., SERIALIZABLE) in most RDBMS

• Versioning:

  • AWS SDKs for DynamoDB support Conditional Writes out of the box

  • Event Sourcing frameworks like Axon (Java), Eventuate (Java), or Marten (C#) enable version-based control

Wrapping up

• There are only 3 ways to ensure consistency: WAL, Locking, and Versioning.

• DynamoDB scales by choosing versioning and pushing retry logic to clients.

• Versioning is ideal for distributed, high-throughput systems, but comes with trade-offs.

• If you're building a system that involves concurrent writes, don't rely on duct tape and blind hope. Rely on versions.

• Consistency isn’t magic; it’s engineering.

Most teams overengineer consistency or overlook it until it breaks. DynamoDB shows that with the right model, you can keep both scale and safety.

Your Move: How does your system manage consistency today? Locking? Logging? Or versioning?

Shoot me a reply or share your approach; I’m curious.

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Articles I enjoyed this week

• How API Gateway Actually Work by

• 10 Must-Know Database Types for System Design Interviews by

• Essential Components of a Production-Ready Web Application by

• Why Composition Beats Inheritance In React by

Thank you for reading System Design Classroom. If you like this post, share it with your friends!

Share