The context window is just that big, how should the Agent design be divided
Dedicated domain Agent series 04. Everything the model knows in every round is the words fed into the window this time. This article talks about contextual engineering: a tool result is fed to both model and user versions. The core of slimming and trade-offs is to determine which fields are untouchable, distill locally at the tool boundaries, and proactively inject the current date and user portrait.
The previous article talked about tools. What the tool retrieves and calculates must finally be stuffed into the context of the model so that it can make the next judgment. This article talks about how to manage this context.
Let's start with one thing that is easy to overlook: models have no memory. It does not "remember" what happened in the previous round. Every round, all it knows is the words fed into its window this time. Only if there is something in the window can it see; what is not in the window cannot exist for it.
Therefore, running this window well is a craft that requires special attention in a dedicated Agent. Ignore it and your Agent will be expensive, slow, and confused as you chat.
Why? Because windows are scarce. After a self-driving trip is scheduled, the model may have to adjust the tool more than a dozen times: searching for the destination, verifying five or six passes in parallel, estimating the driving time of seven or eight sections, and checking the aircraft and wine. If the original return of each tool is piled into the context as it is, the window will expand at a rate visible to the naked eye. Especially for online search, its returns are long (each result has a large summary) and many (one trip requires searching many times), making it the number one culprit with inflated context.
Regardless of it, the consequences are compound interest: when the model searches for the tenth time, the complete summary of the first nine pages will still be pressed in the window, and it will re-read everything that has been used long ago and no longer used again in each round. It's expensive, slow, and confusing.
So what should we do? The first thing I did in Bank Y was to serve two readers returned by a tool separately.
The return value of a tool actually has two places, and they want completely different things. One is to feed back the context of the model, and the model needs to use it for its next judgment. The more refined the better. Irrelevant fields are burdens. The other is an activity stream for users to see,"Found 5 items: Daocheng Aden Guide, Zhuoduo Mountain Road Conditions", allowing users to know what the Agent is doing.
Wild Y uses two functions to separate them. The part fed to the model should be slimmed down: For an online search result, the only thing the model can really use is the title and a short summary to determine whether the attraction is real and whether the road is not working. Urls, extra fields, and extra long body are not used at all when writing the model. So before entering the model, cut them off, leaving only the title and a summary truncated to 150 words for each item, leaving a maximum of six items. However, the part shown to the user remains unchanged, and the complete data is still taken and pressed into a single line of words. In the same return, we feed the model thinly and show the whole one to others, each taking what they need. After actual measurement, the token for a single search has dropped by more than half. Because it is compound interest, the amount saved by the entire wire arrangement is considerable.
There is one thing that is easy to make mistakes here, which is worth emphasizing: losing weight and cutting off the results of all tools without thinking.
In Wild Travel Y, only online search has been slimmed down, and other tools have been released as they are. Why? Because tools such as checking the real flight wine must be returned to the real flight and hotel selected by the user as it is. There must be no less field for price, flight number, and booking link. If you cut it, you will make an error. However, the tool used for estimating driving is already compact when returning and there is no need to use it. Determining which field model can be used and which tool's data cannot be touched is the core of context engineering. Anyone can write the action of compressing. Knowing what to compress and what to leave is a field judgment.
Slimming is "cutting fields", and there is another way to go further: Before information enters the window, distill it into conclusions.
The parallel verification tool mentioned in the previous article is the best example. The model gave it five questions to verify at a time, such as the road conditions of five passes. It searched in parallel and got five original search summaries. If you insert the context directly, there are five more paragraphs. But it didn't. Before returning, it first uses a model call to compress the summary of each topic into a factual conclusion, such as "There are many mountains: It is usually passable in October, and there are occasional temporary controls in case of heavy snowfall. Check road conditions before departure." What enters the context of the main loop is the conclusion of five sentences, no longer the summary of five pieces. Compression occurs in-situ at the tool boundary before information enters the window. Noise remains inside the tool, and dry goods return to the main cycle.
In addition to the dialogue history and tool return, there is also part of the content in the window that you have voluntarily inserted in. This part is also contextual engineering, and often determines whether the Agent looks like a person who knows what he is doing.
Every time Wild Xing Y runs, it will put "What month is it today" into the system prompt in real time. It sounds like a small matter, but without it, the model will use the default date you remembered during training to calculate your ticket and fill in a departure date that has passed long ago. One sentence of injection can cure a type of illusion. When an old user takes a new trip, the system will also refine his portrait, where he has been, and what he likes in daily life into a short paragraph and enter the first round. So the Agent "remembered" this person as soon as he came up and could say,"You were in western Sichuan last time, I'll change your direction this time." The attention is to refine it into a short paragraph and don't pour all the user's history into it, which will burst the window again.
Looking back, context engineering does just a few things: what to put in the window, what to block out, where to press if it is too long, and what to inject in addition to dialogue. It is the only basis for each round of judgment of the model, and it is worth running it like a scarce land.
However, this window only handles matters in a conversation. Who the user is, where he has been, and what he prefers should not start from scratch every time. There must be a place to remember across conversations. That's another thing called the memory system. The next article will talk about how to make the Agent remember you without misremembering and taking things that didn't happen as true.