When I think about agent memory, I don’t think about a chatbot forgetting someone’s favorite color. I think about an agent applying information that was once correct.
Imagine your e-commerce store reduced returns from 30 to 14 days for orders placed after September 1.
A customer asks if an order that arrived yesterday can be returned in three weeks. The agent says yes. It found a real 30-day return policy, but missed that the policy had expired. Now the customer expects a return the store may reject.
A larger context window or vector database might retrieve both policies, but neither should decide which one applies.
Here is my take: Agent memory is evidence, not truth. Each item needs a version, a scope, a source, and a check against authoritative data. Memory should work like a case file: it provides useful evidence, but it should never act as the judge.
And to achieve that, we need to start with the context window.
This newsletter issue is free thanks to our sponsor, Oracle.
Oracle partnered with DeepLearning.AI to create Building Adaptive AI Agents, a short course for developers who want to build agents that learn from experience instead of repeating the same mistakes.
The course covers three ways to improve an agent over time: turning execution traces into reusable, human-approved skills, updating its knowledge, and adapting the model when the problem requires it.
If you are working on agents that need to improve safely over time, I recommend checking it out.
N ow, let’s look at where an agent keeps the information it needs right now: the context window.
The context window is your workspace
The first instinct is to add more history. I have been there, and trust me, this typically just makes the problem worse.
Think about the context window like a desk. If you pile too many documents on it, it will be hard to find the right ones when you need them.
The same thing happens to the model. It has to work through more noise to identify the relevant information, while you have to pay for more tokens.
So ask yourself: what is the minimum useful context the agent actually needs for this task?
Once you limit that workspace, the next step is deciding what memory belongs behind it. To do that, you first need to classify it.
Four memory types do not require four databases
I use these categories to describe what the data means and how the agent uses it, not where you have to store it.
In the context of our example, it will look like this.
And just to be clear: four memory types do not require four databases.
In most systems, I would start with one durable store and separate each memory type through schemas, indexes, access patterns, and retention rules.
For example:
Working memory might disappear when the task ends.
Episodic memory needs timestamps and outcomes.
Semantic memory needs relevance search and freshness checks.
Procedural memory needs direct lookup and explicit updates.
The storage technology may stay the same; what really changes between memory types is how we write, retrieve, and remove the data.
Do not turn a useful memory taxonomy into an unnecessary distributed system. Start simple, and add specialized infrastructure only when scale, latency, or reliability gives you a real reason.
Once you classify the memory, the next question is more important: should you store it at all?
Treat every memory write as a proposal
Here is how I think about memory writes: the model can propose a memory, but it should never approve its own proposal.
Think of each memory write like a pull request. The model can extract information and suggest a change, but the application must validate it before merging it into long-term memory.
Imagine an employee says, “We usually make exceptions for loyal customers.” A summarizer could turn that comment into: “Loyal customers can return orders after 30 days.”
That sounds reasonable, but it is not an approved policy. It may describe one support decision, an informal habit, or simply the employee’s opinion. If you store it without validation, the next agent may present it to a customer as an official rule.
This is why you have to separate published policies, support decisions, customer claims, and model guesses. They may contain similar words, but they cannot have the same authority.
A proposed policy record might look like this:
The model may extract the policy text, but I would make the application supply and validate the scope, dates, priority, source, and approval fields.
I also prefer to draw a hard line between evidence and decision data. source_excerpt helps a person understand where the memory came from, but the system should make decisions using validated fields such as market, sale_type, and order_placed_from.
If structured fields cannot represent the policy, use a versioned rules engine.
Important: Never make the prompt your unofficial policy engine.
Once a proposal passes validation, you still cannot blindly replace the previous memory. You first have to decide when the new policy applies, what it replaces, and how it interacts with other rules.
A policy change needs update semantics
A policy change is not a simple database update.
Returning to the opening example: the store changes its return window from 30 days to 14 days for orders placed after September 1. You cannot replace 30 with 14 everywhere because the old policy still applies to orders placed before September 1.
If you overwrite the old rule, you break historical decisions, but if you keep both rules without defining when each applies, you can force the agent to guess.
So I came up with these terms:
Active: Approved and enabled for evaluation.
Effective: Covers the relevant business date.
Applicable: Matches the order and wins the precedence rules.
Superseded: Replaced by another version for the same scope.
Archived: Kept for history but excluded from normal retrieval.
Deleted: Blocked from serving and moving through the erasure process.
Now, take something into account: two different policies can overlap.
For example, one order might match all three of these rules:
US orders can be returned within 14 days.
Electronics can be returned within 30 days.
Final-sale electronics cannot be returned.
You need structured matching fields like tenant, store, market, product class, customer segment, fulfillment method, sale type, and order date. Then you need explicit precedence rules.
Your precedence might look like this:
One thing that can help is to reject overlapping policies at write time unless they have an intentional priority relationship. At read time, the resolver should rank every match and escalate if two policies still have the same precedence.
The resolver should also explain its decision. It should return the selected policy, the fields that matched, the effective date range, and the precedence rule that made it win.
For storage, I would use an append-only policy ledger as the source of truth. Think of it as a logbook: you do not erase old entries when something changes. You add a new version with its source, approval, change type, and predecessor.
You can build a serving projection from that ledger for fast lookups. The projection is an index, not the historical source of truth.
In one database transaction, write the new ledger version, update any local serving projection, and create an outbox event for external indexes.
That said, asynchronous indexing creates a read-your-writes problem. If an administrator publishes a critical policy change, the next customer request should not receive the old answer.
For low-volume, critical changes, I would index synchronously. Another option is a short-lived session overlay, but you have to keep it strict: only committed and validated records, correct tenant and authorization scope, strong version numbers, and a short freshness window.
Before building the prompt, deduplicate by stable policy ID and resolve the latest applicable version. Otherwise, your freshness fix can create another source of conflicting memory.
That is a lot, I know, but once the write path preserves policy history and resolves conflicts, the read path has to apply that history to the customer’s actual order.
Retrieve for the decision, not just the conversation
Here is my rule for reading memory: retrieve what the agent needs to make the decision, not everything that might be related to the conversation.
The customer is asking, “Can I return this order after three weeks?”
The naive approach is to search for conversations and documents containing words like “return,” “order,” and “three weeks.” You don’t want to do that; it may retrieve the 30-day policy, the new 14-day policy, and several exceptions from previous support cases.
All of those memories may be relevant, but relevance does not tell you which policy applies.
The first step is to load the order from the commerce system. Agent memory does not own the order date, market, product class, sale type, fulfillment method, or current return status. Those facts belong to the system that manages the order, so:
order = commerce.getOrder(orderId)Next, pass those authoritative facts to the policy resolver.
policy = policyResolver.resolve(
tenantId = order.tenantId,
storeId = order.storeId,
market = order.market,
productClass = order.productClass,
customerSegment = order.customerSegment,
fulfillmentMethod = order.fulfillmentMethod,
saleType = order.saleType,
orderPlacedAt = order.placedAt
)The resolver should use exact fields, effective dates, and precedence rules to select the applicable policy. It should also return an explanation of why that policy won.
Only after resolving the policy would I retrieve similar episodes, and only if the case needs them. For example, the customer may claim that support already promised an exception. That is when a previous conversation or support decision becomes useful evidence.
I think of vector search as a librarian. It can find documents that discuss the same topic, but it cannot decide which rule has legal or business authority.
A highly relevant result can still contain an expired policy. Similarity does not understand applicability, precedence, tenant boundaries, or authorization.
That means every retrieved policy must pass through the resolver before it reaches the prompt. If the evidence conflicts and the resolver cannot produce one clear answer, the agent should stop and escalate (yes, I’m talking here about human-in-the-loop review) instead of guessing.
The final context might contain:
Authoritative order facts
+ Applicable policy and resolution explanation
+ Relevant customer history
+ Optional supporting episodesThis gives the model enough evidence to explain the decision without turning the context window into a policy engine.
You also have to keep decision authority separate from action authority.
A previous support exception may help the agent understand the case, but it must not give the agent permission to issue a refund. The refund service still needs its own authorization checks, approval limits, and idempotency controls.
This separation is important:
Retrieval finds evidence.
The resolver makes the policy decision.
Authorization controls the action.
Once you separate those responsibilities, the architecture becomes much easier to reason about. The old trick of divide and conquer still works like a charm.
A good read path still collects old data, which brings us to forgetting.
Forgetting is more than setting a TTL
Here is another take: forgetting is a feature you have to design, not a cleanup job you add later.
I would keep approved policy history because you may need it for audits and historical decisions. But I would never treat personal customer memory as permanent.
Imagine the customer from our return example asks you to delete their data. Removing the original conversation is not enough. Their information may also exist in extracted facts, summaries, embeddings, caches, evaluation datasets, and prompt traces.
Deleting memory is like recalling a product from every warehouse. Removing it from the storefront does not mean every copy has disappeared.
A TTL only answers when one record should expire. It does not tell you where copies exist, how derived data should disappear, or how to stop an old background job from recreating the memory.
I would split deletion into four parts:
1. Logical deletion
Start by creating a tombstone and incrementing a deletionGeneration.
{
"customerId": "customer-728",
"status": "deleted",
"deletionGeneration": 7,
"deletedAt": "2026-09-15T10:30:00Z"
}The tombstone tells every read path to stop serving the data immediately. You do not have to wait for every physical copy to disappear before protecting the customer.
2. Protection from stale workers
Every asynchronous job should carry the deletion generation it started with. Before writing an embedding, summary, or extracted fact, the worker must compare its generation with the current one.
if job.deletionGeneration != current.deletionGeneration
or current.status == "deleted":
discard job resultWithout this check, a delayed worker can recreate data after you delete it. I have seen the same class of bug with stale cache writes: the delete works, but an older process puts the value back.
3. Physical erasure
Next, remove the customer data from active stores and every derived artifact you can trace back to it:
Messages and conversation history
Extracted facts and summaries
Document chunks and embeddings
Search and reranker caches
Evaluation datasets
Prompt and tool traces
This is why provenance matters. If you cannot identify which artifacts came from a customer record, you cannot reliably delete them.
4. Backup and audit handling
Backups usually follow a retention schedule instead of being edited in place. You should keep deleted content outside normal use until those backups expire.
If you restore an older backup, replay the deletion ledger before serving traffic. Otherwise, the restore can bring deleted memory back into production.
The important point is that deletion is, in fact, a distributed workflow. You have to stop serving the data, prevent stale jobs from recreating it, erase derived copies, handle backups, and verify that every system respected the request.
And because deletion can fail at any of those boundaries, the next step is to test the memory layer with real failure scenarios.
Test memory with real order boundaries
Here is my rule for testing agent memory (final rule, I promise):
Do not test whether the agent remembers something. Test whether it uses the right memory under the wrong conditions.
Let me explain; the happy path is easy. Store a policy, ask a matching question, and verify that the agent retrieves it.
Production failures happen at the boundaries: one second before a policy takes effect, one tenant away from the correct scope, or one stale indexing job after a deletion.
I would start by replaying the return-policy incident from the opening.
An order placed on August 31 should receive the 30-day return policy. An order placed on September 2 should receive the 14-day policy. The model may retrieve both policies, but the resolver must select the one applicable to the order date.
Then test what happens when the policy arrives late.
Record the 14-day policy on September 10 with an effective date of September 1. Verify that the system can answer both questions:
Which policy applies to the September 5 order?
Which policy did the agent know on September 5?Those questions use the same order but different clocks. If your tests cannot distinguish them, your implementation does not support the bitemporal model described earlier.
Next, test scope and precedence.
I would also delay the search index on purpose. Publish the 14-day policy, keep the vector index on the old version, and ask an immediate follow-up question.
The agent should still use the committed policy from the serving projection or the guarded session overlay. If it answers with the old rule, you have a read-your-writes bug.
Deletion needs its own failure tests.
Create a customer memory, start a delayed embedding job, and then delete the customer data. When the old job finishes, it must compare deletionGeneration values and discard its result.
I would also restore an old backup in a test environment and verify that replaying the deletion ledger removes the customer data before the system serves traffic.
Authorization is another boundary you cannot leave to the model.
Retrieve a previous support case where an agent approved a refund. The memory may help explain the current case, but it must not give the new agent permission to issue another refund. The action service should reject the request unless the current authorization rules allow it.
The goal is not more memory. It is better decisions.
The agent in our opening example did not fail because it forgot the return policy. It failed because it remembered a real policy without understanding when and where it applied.
That is the main lesson I want you to take from this article.
A good memory layer is not a large conversation history or a vector index full of everything the agent has seen. It is a system that controls what gets stored, tracks where each memory came from, knows when it applies, resolves conflicts, and removes it when the system should no longer use it.
You do not need a complicated memory platform to start. I would begin with one durable store, structured facts, clear versions, explicit time ranges, deterministic policy resolution, and a deletion workflow you can test.
Then add complexity only when real problems with scale, latency, or reliability require it.
More tokens only delay the architectural decision; memory is a data management problem.
Memory should help the agent explain a decision. It should never make the decision by accident.
The most dangerous agent memory is not what it forgets. It is what it remembers without knowing when, or whether, it still applies.
Until next time,
— Raul
System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.