Let me tell you about Dani. She’s a composite of engineers I’ve worked with, and her story combines failures I’ve seen more than once.
Her team shipped a checkout saga that passed every review. One step per service, a compensation for every step. By the book, everything looked covered.
Then a flash sale sold out the headline item in minutes. Hundreds of customers got a confirmation email, a charge, a refund, and an apology email, in that order. Dani, who owned checkout, summed it up: “The saga did exactly what we designed it to do.”
And that sentence captures the whole problem; nothing crashed here. Every compensation ran. The bug lived in the order of the steps, and most saga explanations never mention order at all.
Finding the right order starts with one uncomfortable fact: some steps can’t be undone.
Build for the long run.
This newsletter issue is free thanks to our sponsor, Inngest.
Long-running agents need endurance.
They must endure waiting for users, and weather tool failures. They must take each new jump in model intelligence in stride.
Inngest makes long-running agents durable, observable and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
Some steps have no undo button
A saga is a solution to the problem of distributing a transaction across several services. A saga allows a service to commit its local change and, if a subsequent step fails, undoes the prior commits using compensating transactions. In their 1987 paper, Garcia-Molina and Salem assumed that each step of a saga is compensatable.
This is a strong assumption that is often violated in practice. There are many examples in the real world where steps are irreversible. Once you send an email, it sits in someone’s inbox. Once a warehouse ships a package, a truck carries it away. Once you call a third-party API with no cancel endpoint, the effect stays.
Most of the time, undoing a step is either extremely expensive or highly impractical. Reversing the effects of a step often leaves a negative impact. Those are new business actions, not rollbacks.
Other steps can be undone, but it will cost you. Refunds return funds after capture, but the customer still sees a charge on their statement. You also eat the processing fee, and it takes a while for the money to arrive back in the customer’s account. So, while theoretically reversible, practically speaking, it isn’t a good look.
What is useful about a step is not simply the presence or absence of compensation. It is how much undoing the step hurts. If you answer this question for each and every step, the order almost writes itself.
The order: three zones and one point of no return
A well-ordered saga puts its steps into three groups, always in the same order:
Compensatable steps come first. These steps can fail for business reasons, like a declined card or an out-of-stock item, and you can undo them cleanly. If a step fails, the program retraces its steps and reverses any changes it made.
The pivot is the go/no-go step. If it is definitively rejected, you compensate everything before it and the saga ends without a committed order. If its outcome is unknown, you reconcile it before choosing a recovery direction. If it commits, the saga has crossed the line and must reach the end. Rolling back is no longer an option. The pivot does not have to be a separate, dramatic step. Microsoft’s guidance lets the last compensatable step or the first retriable step play that role. What is important is the line: one clear point where the saga commits.
Retriable steps come last. They must not introduce a business rejection that invalidates the committed outcome. Transient failures, such as timeouts or rate limits, are retried. Permanent exceptions follow an explicit terminal or operational recovery path. After the pivot, a failed step does not trigger compensation of the committed order; recovery works toward completion.
What is flipped at the pivot is the commitment to recovery. Leading up to the pivot, a definitive business rejection enables compensation. After a successful pivot, you do the recovery work to achieve the committed outcome. Transient failures can be retried prior to and following the pivot.
I think the home buying process provides a good analogy. There are a lot of things that can happen between an offer and closing that can make a buyer change their mind. There are inspections that can uncover problems with the home. The financing can fall through. Ultimately, it is signing the closing documents that legally makes the purchase. Once closing documents are signed, it's about getting moved into the house and not about regretting the purchase.
I’d say the shape of the design isn’t very complicated. I would say the hardest part is deciding which zone steps belong to, which then determines the final, ordered, step pattern.
Classifying a step to find its place
I run every step by asking three yes/no questions, in order.
Can it reject the business outcome?
If the answer is no, the step can be retried after the pivot. This question does not take into account the ability to undo the step. Sending the confirmation email is a classic case of post-pivot step placement. Of course you can’t un-send the email, and that’s the reason it’s placed after the pivot. You send confirmation only after the pivot has committed. There’s no need to reverse the step. The saga continues and the email is sent.
Is it cheap to undo, with limited customer impact?
If the step can say no, what happens when that step is taken back? Releasing a reservation or voiding a card authorization usually causes far less customer-visible disruption than refunding a captured payment. Refunding a charged amount is neither quick nor unnoticeable. It takes time and costs fees. The refunded amount appears on the customer’s statement.
If undoing it is cheap, the step is compensatable and goes before the pivot. Among these cases, place the most likely to fail first. An early failure means less work is done.
Do you already have a point of no return?
An expensive-to-reverse step that says “no” should be your pivot. If the saga doesn’t have one yet, this step is it. If it already does, whichever of the two runs second can fail after the first has already committed, stranding it.
Split one of them into a cheap “hold” and a final “commit,” then run each half through the questions again.
The last two rows of the process only work with an exact agreement. “Notify warehouse” means handing off a fulfillment directive for stock previously committed to the order. It does not mean checking stock availability. “Send confirmation” means the email service has accepted the email for delivery, or the saga logs a final notification failure, for example, a bad email address. We cannot validate email delivery, therefore, the saga will not wait for the email to be delivered.
First, let’s put these questions to work on a checkout flow, starting with the wrong order.
Redesigning the checkout
This is a simplified version of Dani’s checkout code. I’ve seen similar code in other projects. This order can feel natural because it shows what the user is interested in: tell them it was successful, take their money and finally communicate with the warehouse.
1. Send "Order confirmed" email
2. Charge card
3. Reserve inventory ← fails: out of stockIf Step 3 fails, the process compensates. The charge is refunded. The customer sees both on their banking app. The process attempts to “unsend” the email, and, being unable to do so, issues a second email, also compensating. The customer has now received two emails, one confirmation email and one apology email. Every step had a compensation. The order turned all of them into damage control.
Now the checkout, reordered and refined by zone:
Let's consider some scenarios for this revision. Say an item is unavailable at step 1. In that case, nothing is undone and the customer sees an error immediately. As for step 2, if a card is declined, the saga releases the reservation, and the customer only sees a decline message. What if a capture is firmly refused at step 4? In that case, the saga releases the allocation and cancels any remaining authorization, at most, the customer sees a pending authorization disappear. Now let's consider step 5a. What if the email service is unavailable? The money is moved and the order is considered final. Therefore, the saga keeps retrying the email while the fulfillment command at 5b goes out on its own. In general, a failure to send a notification should not block fulfillment.
The reservation has to survive the pivot. Step 3 is the one most designs skip, and if it’s not there, this order is still broken. Reservations usually have a time-to-live (TTL). Let’s say your capture eventually succeeds. In the meantime, the warehouse service is down and your reservation expires. Another customer purchases the item. Eventually, the warehouse service is restored, and “notify warehouse” succeeds for stock you no longer own. Checking the reservation right before the capture does not help, because of the race condition.
The inventory service must atomically convert a still-valid reservation into a non-expiring allocation, preserving exclusive ownership of the stock. If the reservation has expired or been released, conversion rejects. The allocation remains compensatable, so it belongs before the pivot.
Why not simply remove TTL altogether? Removing TTL means you're saying abandoned checkouts will lock the stock, forever. The TTL gives you protection before the customer completes the purchase. The allocation protects the customer after the purchase.
There is a second rule to this order: put the steps most likely to fail first. A failure on step 1 brings zero compensations. A failure on step 4 brings three compensations. Cheap, likely failures belong at the front.
Here is the control flow as pseudocode. Watch the order of two lines: the orchestrator records a step before calling it, not after.
// Control-flow pseudocode. A real engine persists every step of this.
const checkout: Saga = {
compensatable: [
{ execute: reserveInventory, undo: releaseReservation },
{ execute: authorizePayment, undo: voidAuthorization },
{ execute: allocateInventory, undo: releaseAllocation },
],
pivot: { execute: capturePayment },
retriable: [{ execute: sendConfirmation }, { execute: notifyWarehouse }],
};
async function run(saga: Saga, order: Order) {
const started: Step[] = [];
for (const step of [...saga.compensatable, saga.pivot]) {
started.push(step); // record intent BEFORE calling out
await sagaLog.save(order.id, started);
try {
await step.execute(order); // idempotent, keyed by sagaId + step name
} catch (err) {
if (!(err instanceof BusinessFailure)) throw err; // transient: retry later
await rollback(started, order);
return "FAILED";
}
}
// Past the point of no return: one durable task per independent step,
// each with its own retries and dead-letter path
for (const step of saga.retriable) {
await durableTasks.schedule(order.id, step);
}
return "COMMITTED";
}
async function rollback(started: Step[], order: Order) {
for (const step of [...started].reverse()) {
await retryUntilSuccess(() => step.undo?.(order)); // safe even if it never ran
}
}That ordering closes the most prevalent gap in saga recovery. A service may commit a reservation and be unable to respond because of a network issue. If the orchestrator recorded only completed steps, rollback would never release that reservation. Recording intent first, and making every undo safe to run against a step that never took effect, fixes it. In that case, releasing a reservation that may not exist should be a no-op.
One race survives, though. The reserve request is slow, the orchestrator times out, and the release runs and finds no reservation, so it does nothing. Then the original reserve request finally lands and creates the reservation. Both calls were idempotent, and the stock is still stuck.
A “nothing to release” no-op is only safe if the original request can’t take effect afterward. If it can still land after the timeout, the inventory service has to remember the cancellation: record it against the operation ID, and reject any reserve that shows up later with that ID. Once an operation is cancelled, nothing can bring it back.
A durable execution engine like Inngest (this issue’s sponsor) persists completed step results and reuses them when the workflow resumes.
But here’s the catch I always point out: the engine remembers where the workflow is, not what your services did. Your inventory service still owns its own data, and it’s still on you to make reserve and cancel safe.
Inngest’s own docs give a great example. A refund goes through, but something fails before the engine saves that step. On retry, the refund step runs again. If that call isn’t safe to repeat, the customer gets refunded twice.
The engine also can’t save to your database and publish an event as one action. When I need both, I write the change and a “publish this” note in the same database transaction, which is the transactional outbox pattern, and make every consumer safe to receive the same event twice.
All of that is still your job, along with the ordering decision.
Two contracts are hidden in the post-pivot code as well.
Scheduling must be idempotent.
Recovery must resume scheduling.
After a crash, recovery must schedule any post-pivot tasks that are still missing. COMMITTED guarantees the pivot was successful; it does not mean the related work has been completed or notified.
The same idea applies to keys. Your order ID tells you which purchase this is, but that’s not enough. Every step, and every payment attempt, needs its own key, something like {sagaExecutionId}:{stepName}.
Here’s the part that trips people up: one attempt gets one key, no matter how many times you retry it. If you generate a new key on every retry, you’ve quietly turned idempotency off. You’ll see this in the Stripe example below, where the auth- and capture- keys are built from an attempt ID that gets saved and reused.
Idempotency keys also expire. Stripe can prune them once they are at least 24 hours old, and after that, reusing your stored key can execute a brand-new request. Keeping the key in your own database doesn’t extend Stripe’s protection. So I persist the payment attempt, its idempotency keys, and the resulting PaymentIntent ID, and before retrying an old unresolved operation, I reconcile the existing payment’s status first.
The authorize/capture split in this example is not something I invented for the article. Payment providers built their APIs around exactly this problem.
When order is not enough, split the step
A single-call charge can serve as the pivot if all other business checks and resource commitments are already settled. A definitive decline releases inventory, a successful charge commits checkout, and an unknown outcome must be reconciled before the saga chooses a recovery direction. If another fallible, expensive-to-reverse commitment must coexist with the charge, ordering alone may not be enough.
Splitting authorization from capture can help avoid a settled charge by moving the payment rejection step up in the process. It gives you a clean pivot only if the remaining pre-pivot work is safely compensatable and nothing after capture can reject the committed outcome.
I like using Stripe to show this because the split is built right into the API. Create a PaymentIntent with capture_method set to manual, and Stripe only authorizes the card. The bank puts a hold on the funds, but no money moves yet.
The part people tend to miss: authorization is not always instant. A 3D Secure challenge can park the payment in requires_action while the customer approves it on their phone. So the code below assumes a card payment that has already reached requires_capture.
From there, you have two moves. Cancel, and the hold disappears without a settled charge. Capture, and the money moves. Even then, don’t treat the order as committed until you have confirmed the capture actually succeeded.
// Persist paymentAttemptId; reuse it for retries of this attempt.
// Compensatable: authorize only, the bank holds the funds
const intent = await stripe.paymentIntents.create({
amount: 4999,
currency: "usd",
payment_method: paymentMethodId,
confirm: true,
capture_method: "manual",
}, { idempotencyKey: `auth-${paymentAttemptId}` });
// Compensation, if a later pre-pivot step fails
await stripe.paymentIntents.cancel(intent.id);
// Pivot: the money moves
await stripe.paymentIntents.capture(intent.id, {}, {
idempotencyKey: `capture-${paymentAttemptId}`,
});Hotels have done this for years: they put a hold on your card before you arrive and only charge it when you check out. Look at what that split does to the saga. One step that was both risky and hard to undo becomes two. The authorization catches the declines early, while undoing them is still cheap, so it lives safely before the pivot. The capture becomes the pivot itself, and by then it rarely fails for a business reason.
This trick works way beyond payments. Inventory goes from “reserved” to “allocated.” A document goes from “draft” to “published.” A concert seat gets held for ten minutes, then ticketed. Same shape every time: a cheap hold you can let go of, then a commit that makes it real. So when your saga has no clean pivot, look for a step you can split that way.
Splitting buys you a clean pivot, but it also introduces time into the design, and time is where sagas get messy.
Where it gets messy
A good order gives you a clean model, and production shows you timeouts, expirations and other concurrent users. These are the issues I see most often on production applications.
The pivot times out. This one is the worst. You call capture, and the network drops, and you get no response. Did the money go through? If you assume it failed and compensate, you may release inventory for an order that was already paid for by the customer. If you assume it succeeded, you may have given a product for free. So, what do you do? Treat a timeout at the pivot as "unknown", and decide only when you get more information. This means your pivot should be idempotent and have the ability to be queried. If this is not the case, then you do not have a pivot, and you should not be taking that risk.
Compensations fail too. Just like other calls, release-inventory calls can time out and compensations need the same reliability treatment as forward recovery: they must be idempotent and retried, with operational escalation if they remain blocked. Post-pivot steps also need idempotency so a retry does not create duplicate effects, such as multiple confirmation emails.
“Bound to succeed” needs a backstop. Post-pivot steps can’t reject the order, but that doesn’t mean they always finish. For example, a bug or a long outage can leave one stuck for hours. My rule here is simple: retry transient failures with exponential backoff, and when the retries run out, park the step in a dead-letter queue (or whatever recovery state your platform offers) and get a human involved.
What I never do is compensate the order just because a step is stuck. The customer paid, the stock is allocated, and the order is real. A stuck notification is an operations problem, not a reason to undo a sale. And if the customer cancels next week, that is a separate business workflow, like a refund, not this saga rolling back.
Holds expire. Splitting a step into hold and commit adds a clock. Stripe documents that an online card authorization usually lasts about 7 days, and shorter for some cases, such as roughly 5 days for Visa merchant-initiated payments. Use the payment’s actual capture deadline rather than assuming every authorization lasts seven days: Stripe exposes it on the charge as payment_method_details.card.capture_before. When the hold expires, Stripe releases the money and the PaymentIntent is set to canceled. This causes the pivot to fail.
Inventory reservations also have holds with a time-out period. There can be a significant amount of time between the first step and the pivot. To give some examples, this can include backorders or fraud reviews. In general, you want to keep the time between steps as small as possible. Checking a hold right before the pivot steps gives you a temporary reassurance, but there is still a race condition. Converting the hold to a durable commitment gives you assurance.
Other requests see the middle. Sagas trade away isolation. While a saga sits between “reserve” and “capture,” other customers see reduced stock, and a dashboard might already count the order. Chris Richardson calls the fixes for this “countermeasures.” The one I reach for most is a semantic lock: the saga marks the record as PENDING, so other flows know it’s still in flight and can wait, reject, or display it as pending. Only the saga that set the flag can clear it, either by completing or by compensating.
A short compensatable zone helps too, because it shrinks the window in which anyone sees the middle. Even voiding an authorization isn’t perfectly invisible: Stripe notes that some card statements don’t distinguish a hold from a settled charge, so a customer may briefly see a pending line that later disappears.
“Most likely to fail first” is a heuristic, not a law. Sometimes the most likely failure is also the slowest or most expensive check, like a call to an external fraud model that adds real latency to every checkout. In this case, it is better to place a less expensive check with less failure rate in front of it. In general, you try to minimize the expected cost of a failure.
None of these cases changes the core idea. They just tell you where to look when you review a real design, which brings me to the checklist I use.
The checklist
When I review a saga design, I go through these in order:
Label every step as compensatable, pivot, or retriable.
Find the point of no return, then ask whether a later rejection can strand an earlier commitment. If it can, split a step.
Check that no post-pivot step can reject the committed business outcome.
Put cheap, likely failures at the front, and make anything the customer pays for durable before the pivot.
Make the pivot idempotent and queryable, so a timeout has an answer.
Record each step before you call it, make every step and every undo idempotent, and make sure a late request cannot win over its cancellation.
Know how long every hold lasts, and keep the compensatable zone shorter than that.
Give retriable steps a dead-letter path that ends with a human.
The mental model behind all of them fits in one sentence: a saga is a business recovery protocol you build across local commits, and the pivot is the point where recovery changes direction. Order the steps so everything before the pivot is cheap to undo and everything after it has a clear path to done.
Dani’s team didn’t write a single smarter rollback. They moved steps.
Reservation first. Allocation before capture. Authorize, then capture. The confirmation email, post pivot
The next flash sale sold out just as fast. This time, the customers who missed out got one message: “Sorry, this item is out of stock.” No charge. No refund. No apology email.
Same services, same compensation, but in a different order.
That’s the part I want you to take back to your team. Most saga reviews ask, “Does every step have an undo?” I ask a different question: “Where’s the point of no return, and did we put it there on purpose?”
Get the order right, and compensation becomes smaller, more predictable, and far less visible to your customers, which is exactly what you want.
Until next time,
— Raul
System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.