This agentic case starts with a request no business owner wants, but still we should plan for: a customer asks an AI support agent for a refund.
And the agent does exactly what it is told to do: it loads the order, it reads the refund policy, and it confirms that the request qualifies. Then it receives approval and calls the payment provider.
The provider does its job too and creates the refund.
But then, just before the agent records the result, the process ends up crashing.
A few seconds later, the workflow resumes from its last saved state, where “the refund never finished.”
So the agent tries again.
And in theory, the workflow recovered successfully; in practice, the customer was refunded twice.
This is one of the most dangerous failure modes in agentic systems: The agent can recover its reasoning without knowing what happened outside the workflow.
The problem starts with what we mean by state.
Build for the long run.
This newsletter issue is free thanks to our sponsor, Inngest.
Long-running agents need endurance.
They must endure waiting for users, and weather tool failures. They must take each new jump in model intelligence in stride.
Inngest makes long-running agents durable, observable and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
An agent has more than one kind of state
For some reason, when I talk about persisting an agent, many developers think I’m talking about saving its conversation history.
That’s useful, but conversation history is only one layer of this onion.
I separate the agent state into three categories.
The Conversation state is the messages, model responses, retrieved documents, and tool results.
The Workflow state is the record of which steps started, completed, failed, or require approval.
The Effect state is the changes outside the workflow: a refund was issued, an email was sent, a ticket was created, or a deployment started.
These states are related, but they are not interchangeable.
A saved conversation cannot prove that a workflow step completed. A completed workflow step does not always prove that an external system performed an action.
That difference creates a small but dangerous failure window that you have to take care of.
The danger gap is after success
Let’s take a look at this execution flow:
The payment provider committed the refund, but the application never recorded the response. From the provider’s perspective, the operation succeeded, but from the agent’s perspective, it never finished.
A timeout creates the same ambiguity. The agent may stop waiting after five seconds while the provider completes the refund after six.
This is an important detail because the timeout tells us the caller did not receive an answer. But it does not tell us whether the refund happened.
We have seen this before; this is where an agent workflow becomes a distributed-systems problem.
A distributed transaction without a transaction
In every distributed transaction, the workflow needs to update at least two independent systems (that is the root of the complexity).
Ideally, creating the refund and recording its result would happen in one atomic transaction. Either both operations succeed, or neither does.
But the workflow database and payment provider cannot share a transaction. We cannot commit both changes atomically, so the system has a dual-write problem.
This creates a partial failure: one system completes its operation while the other never learns that it happened.
A durable workflow usually recovers by retrying incomplete work. At the boundary of an incomplete external call, this behaves like at-least-once execution. The system prioritizes not losing the work, but it may attempt work more than once.
That trade-off is often acceptable for reading data, generating text, or calculating a value, but it becomes extremely dangerous when the operation changes the outside world.
At-most-once:
The refund may be lost, but it should not be duplicated.
At-least-once:
The refund should eventually happen, but it may be attempted repeatedly.
Effectively-once effect:
Repeated attempts create at most one provider refund for the approved business action.Retries alone cannot produce an effectively-once effect. An effectively-once business effect emerges from the complete design: a stable action identity, idempotency, durable state, and reconciliation.
And the first protection is to give the refund an identity that survives every execution attempt.
Give the action a durable identity
Every refund needs an identifier representing the business action, not the individual HTTP request.
For example:
refund:order-4821:request-3The workflow must now reuse this identifier after a timeout, crash, retry, or deployment.
A stable identifier lets the provider recognize that both requests represent the same intended action. If you are lucky enough, it supports idempotency keys, and repeating the request with the same key should return the original result instead of creating another refund.
One more thing: the identifier must match the action’s scope. An order may legitimately receive multiple partial refunds, so using only the order ID could block valid operations. The key needs to identify one approved refund while remaining stable across all its execution attempts.
Timing matters too. If a provider remembers idempotency keys for only a limited period, a retry several days later may no longer be protected. The application should retain its own action history for as long as the business needs it.
But a durable identity only works if we record it before contacting the provider.
Record the intent before executing it
I would not let the agent call the payment provider directly after deciding that a refund looks reasonable.
Instead, the agent should propose an action:
{
"actionId": "refund:order-4821:request-3",
"idempotencyKey": "refund:order-4821:request-3",
"payloadHash": "sha256:...",
"orderId": "4821",
"amount": 79,
"currency": "USD",
"reason": "damaged_item"
}A piece of advice: treat the action ID and its material request payload as immutable. A retry must reproduce the same amount, currency, target payment, and reason; if those details change, it is a new business action, and it requires new authorization.
Then, deterministic application code validates the proposal against the current order, refund policy, previous refunds, and approval limits.
If the action is allowed, the application can proceed and create a durable refund record:
I think of this record as an intent log. It says what the system has permission to do before it attempts to do it. You also need a durable dispatcher, often a worker consuming an outbox or scanning eligible AUTHORIZED records, so a crash between authorization and submission does not strand the work.
Also, a UNIQUE constraint on actionId should prevent two active records from representing the same refund request.
After the authorized record commits, a worker can submit the refund using the record's actionId as the provider's idempotency key. Just make sure your actionId format fits the provider's constraints. Stripe, for example, caps idempotency keys at 255 characters and restricts which characters are allowed.
This separation gives us a source of truth that does not depend on the model remembering what it intended to do.
The agent proposes → Application code authorizes → The action worker executes.
But the hardest transition begins when SUBMITTING does not receive a clear answer.
A timeout means unknown
Most applications model an external call as either successful or failed.
Unfortunately, that model is too simple for payments.
If the provider rejects the request because the amount is invalid, we know the refund failed. But if the request times out after reaching the provider, we do not know what happened.
The correct state is:
UNKNOWNUNKNOWN should not automatically trigger another refund.
First, a reconciliation process should query the provider using the action ID, idempotency key, or transaction reference.
If the provider confirms the refund, the workflow records the result and continues.
If the provider can authoritatively determine that no operation was accepted for that action ID, the worker may retry using the same action ID.
If lookup visibility is delayed or inconclusive, you keep the action in
UNKNOWNand retry reconciliation rather than issuing a new request. And always, always set a concrete escalation policy before you ship this. Something like: after three failed reconciliation attempts within 24 hours, route to human review and stop polling.If the provider cannot search by idempotency key or transaction reference, the integration carries more risk. For high-impact operations, you probably need to wait for human review.
FAILED_FINAL means the system has authoritative evidence that this approved action cannot succeed. Network errors and timeouts are not such evidence.
When you do reach FAILED_FINAL, have a plan for what comes next. The customer still expects a refund. That might mean store credit, a manual bank transfer, or a physical check.
The model should never decide that an uncertain payment “probably failed.” Uncertainty is a system state, not a reasoning challenge. A model may summarize the evidence for an operator, but deterministic policy and reconciliation must decide whether the system may retry or close the action.
Once the system can resolve uncertain actions, durable execution becomes much safer, but still…
Durable execution still needs idempotent steps
Durable execution lets a workflow save progress and resume after crashes, timeouts, deployments, or long waits.
For an agent, this avoids repeating expensive work. The workflow keeps completed retrieval, model, validation, and approval steps and resumes at the step that failed.
For example, with Inngest, you wrap each part of the workflow in step.run(). Inngest checkpoints successful steps and retries failed ones, so the agent does not repeat earlier work after a crash.
But the payment call still needs an idempotency key, because a checkpoint cannot record a response it never received.
If the refund succeeds right before the connection drops, the runtime only knows the step did not complete. So it runs the step again. The step body executes at least once, which is fine for the workflow and dangerous for the payment provider.
One detail matters here: generate the key outside the step, or derive it from the refund request. A uuid() inside step.run() produces a new key on every retry, so the idempotency key protects nothing.
Duplication can also happen one level up. The same refund event can arrive twice and start two separate runs. Each run has its own checkpoints, and both will reach the payment step.
This gives us three different protections:
Checkpointing prevents completed steps from running again.
Workflow idempotency prevents duplicate runs for the same event.
Business idempotency prevents repeated attempts from creating another real-world effect.
You usually need all three.
Even then, another timing problem remains: the world may change while the agent is paused.
Approval can become stale
Imagine that a manager approves a $79 refund.
Before execution, another support agent issues a $30 partial refund. The original workflow then resumes and attempts the full $79.
The approval was valid when someone gave it, but the underlying order changed.
A durable workflow should not treat approval as permanent permission detached from the state. The approval must bind to the important details:
Order: 4821
Refund request: 3
Amount: $79
Currency: USD
Order version: 18
Approval expires: 14:30 UTCFor example, with Inngest, step.waitForEvent() can pause the workflow until the approval arrives or times out. When it resumes, and before it submits anything, deterministic code (yes, an IF) should verify that the order version, refundable balance, approval, and action status are still valid. Make that check part of the state change itself: in one transaction, reserve the refund amount on the order only if the order is still at version 18, and move the action from AUTHORIZED to SUBMITTING. If no row changes, someone else got there first.
This is optimistic concurrency control applied to an agent workflow. If the state changed after approval, the system stops instead of executing an old decision against new data.
Do not ask the model to reinterpret the approval. If the authorized action no longer matches reality, request a new decision.
Handling that race in code is important, but we still need to prove that the recovery flow works.
Test the crash, not only the happy path
A normal integration test submits a refund, receives a response, and checks that the record says SUCCEEDED.
This test misses the dangerous cases.
I would deliberately stop the workflow before calling the provider, after the provider accepts the refund but before saving the response, while reconciliation is running, and after saving success but before advancing the agent.
I would also run two workers against the same action to verify that concurrency controls prevent both from creating a separate effect.
Every test should protect one business invariant:
One approved refund request -> must create no more than one provider refund.Then I would test delayed responses, expired idempotency keys, partial refunds, changed order versions, and providers returning an error after completing the operation.
Finally, I would monitor actions stuck in SUBMITTING or UNKNOWN. A durable workflow that never loses work can still leave money trapped in an unresolved state.
Those failure tests lead to the rules I would take into production.
My final thoughts
Agent memory, workflow state, and external effects are different things.
Persist the intended action before executing it. Give that action a stable business identity and reuse it across every attempt.
Treat payment timeouts as uncertainty, not failure. Reconcile with the provider before deciding whether to retry.
Keep approval and policy enforcement in deterministic code. The agent can propose a refund, but it should not own the transaction.
Most importantly, test the failure window after the provider succeeds and before your application records the result.
A checkpoint tells the agent where to continue. An idempotency key tells the outside world what must not happen twice.
If you ask me, the hardest part of building agents is not giving them tools. It is controlling what happens when those tools change the real world.
Until next time,
— Raul
System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.