Agent's memory is worse than not remembering it

In the previous article, I talked about context. It was a short-term memory in a conversation, and it was gone as soon as the conversation ended. But when old users come back to schedule their third trip, the Agent shouldn't still ask,"Do you prefer to stay in a B & B or a hotel?" He told you this six months ago. Remembering who a user is across conversations relies on another thing, the memory system. In Claude Code, it is called CLAUDE.md. Your project conventions and code style are written once, and it remembers every time after that. The corresponding thing in Wild Travel Y is a user portrait.

But memory is particularly easy to stumble in two places when doing it, and it is worse than not doing it. This article talks about these two pits and how to get around them.

Let's first explain clearly what you remember. There are two types of user portraits of Wild Xing Y. The first category is explicit preferences, which the user has set in the account: accommodation level, travel rhythm, budget, physical strength, taboos, interests. These are what he proactively tells the system and are the most reliable. The other category is behavioral memory, the trips he has done and participated in. There is no need to ask about this, just retrieve the last six titles from the database. So an old user opened a new conversation. The system that the Agent came up with was "Fixed preferences: staying in B & Bs, in-depth slow-down play, moderate budget; done: Western Sichuan Small Ring Road, Erhai Lake Around Lake." It opened its mouth and "remembered" this person, can say,"You have traveled through Western Sichuan before, but this time I will change your direction."

Here's one easy thing to clarify: memory and context are two different things. The context is in the window of the model and lives for one session; the memory is in the database and is stored across sessions and for a long time. The key is that the memory lying in the database is invisible to the model itself, and it will not "sense" it out of thin air. The model has to be read by refining it into a short paragraph at the right time and injecting it into the context window of this session. Memory is responsible for long-term storage, and context is responsible for current use. The two are relay relationships.

!

Now speaking of the first pit, it is also the most vicious one: I missed it.

The most dangerous part of memory is that you can still be confident if you remember wrongly. The model fully believes in what is injected into it. If you insert a wrong sentence into its memory, it will use the wrong sentence to mislead the user over the entire trip, and its tone is particularly determined. Therefore, there is an iron rule in this project that has been established from day one: every item in memory must be true and traceable, and must not be made up by the model itself.

Look at the previous paragraph of "Trip Done". It is a title that is actually found out from the itinerary table. The model did not intervene, but only showed out the real data. The same is true for preferences, which come from the user's real settings in the account, and the model does not participate in guesswork. This is consistent with the first principle of the entire Agent. The system prompt begins with the words "It is forbidden to make up cities, distances, and prices out of thin air." This rule holds true for tools and for memories.

Why are you so serious? Because an Agent that can compile memories is worse than an Agent that has no memories. If you don't have memory, you can at most ask it again every time to make it more annoying; if you compile memory, you use fake "I remember you said it" to mislead decisions, and it will not be easy for users to detect it. Therefore, the credibility of the memory system must rank above its richness.

There is a small but critical detail in the code: when a new user does not have any data, the portrait function returns an empty string without injecting anything. I would rather not inject it or make a guess that "this user may..." No memory, good memory of mistakes.

!

The second pit is more subtle: even if the memory is real, if it is executed as an order, problems will go wrong.

A user has been to western Sichuan, but it doesn't mean that he wants to go to the plateau this time; he prefers to stay in a B & B, but it doesn't mean that he has to take the elderly this time. Memory is the background and a layer of background that helps you understand it better. Don't use it as a hard indicator for this task. So when injecting the user's portrait, I specially loosened it in the prompt and wrote a clear sentence,"These are the background to help you understand him better. Don't move it, and refer flexibly according to the current needs."

This sentence is very important to loosen it. It tells the model that what the user says in the current conversation will always take precedence over historical memory. Memory solves the "default value" problem so that you don't have to ask from scratch every time, but it should not override the user's explicit expression at the moment. If this priority is reversed, the Agent will appear stubborn and disobedient, with a good memory but a worse experience.

In the end, when making a memory system for a dedicated Agent, I have learned three things: only remember what is truly valuable across sessions in this field, and don't put everything in it; every item must be truly traceable, from the database and from the user. Set it up, never let the model be compiled, and would rather leave it blank; when injecting it, it is clear that it is a background reference, and the intention at the time will always take precedence. By guarding these three points, the memory will make the Agent understand you more and more; if any point is broken, the more it will remember them more and more.

Here, the one-man show in the main cycle is over: a model, a set of tools, a context, a memory. However, it is not good to hand over some tasks to the same model and do them smoothly in the main loop. Either it is slow, and the ones that should be lined up in parallel are lined up; or it is not objective, and the person who arranged the plan checks his own plan. The next article will talk about sub-agents, when should they send out tasks, do them in parallel, or get a new brain to check them.