What We Throw Away When a Conversation Gets Long

Every long-running agent conversation eventually hits the same wall: the history no longer fits in the context window. Something has to be thrown away, and what you throw away decides how the agent behaves for the rest of the conversation.

We run four strategies in a pipeline. This is what each one discards and why.

Measuring before deciding

You cannot decide what to drop without knowing what things cost, and calling a real tokeniser on the whole history every turn is more expensive than the problem.

So the estimate is deliberately crude:

const charsToTokens = (chars) => Math.ceil(chars / 4);

Four characters per token is roughly right for English and wrong in known directions for code and non-Latin scripts. That is acceptable because the number drives a threshold, not a hard limit, and it is off by a percentage rather than an order of magnitude.

Non-text parts get a flat charge of 4,500 characters. An image or a document has no character count to measure, so it gets a constant that is approximately right and, more importantly, never zero. A file that costs nothing in the estimate is a file that never triggers compaction, which is how a context window fills up while your accounting says there is room.

Tool results go first

toolResultStrategy({ keepLast: 2, maxResultChars: 200 })

Tool results are the cheapest thing to lose and usually the largest thing present. An API response is often kilobytes of JSON where three fields mattered, and once the agent has read it and acted, the raw payload is spent.

The last two are kept in full because they are probably still live: the customer may be asking a follow-up about the order just looked up. Everything older collapses to a short line naming the call and truncating the result to 200 characters.

What survives is the fact that the call happened and roughly what came back. What goes is the payload. In practice this alone reclaims more than the other three strategies combined, because the ratio of tool output to conversation is brutal in any agent that actually does things.

Then the window slides

slidingWindowStrategy({ keepTurns: 8 })

Keep the last eight turns, drop what precedes them. The oldest, bluntest idea in the book.

It works because conversations are mostly local. Turn thirty usually depends on turns twenty-eight and twenty-nine and rarely on turn four. When it does depend on turn four, it is generally because turn four established something durable, and durable things should not be sitting in the transcript hoping not to get evicted. They should be in the summary or in state.

Some messages are exempt regardless of age. Preserved artifacts and grouped atomic exchanges survive, because dropping half of a tool call and its result leaves the model looking at a call with no answer, which produces worse behaviour than dropping both.

Then truncation, on a budget

truncationStrategy({ tokenBudget: 30000, minKeep: 4 })

The backstop. If the history is still over budget, cut until it fits.

minKeep: 4 is the floor and it is doing safety work rather than quality work. Without it, a single enormous message can drive truncation until essentially nothing remains, and an agent with no context does not fail loudly. It answers, confidently, having forgotten what it was doing. Four messages is not much, but it is enough that the failure is visible rather than silent.

Then summarisation, last

Only when the cheaper strategies have run does anything get summarised, because summarisation costs a model call and adds latency to a turn a customer is waiting on.

It is also the only lossy-in-an-interesting-way step. Truncation loses things you can name. Summarisation loses things you cannot: the exact phrasing of a complaint, the specific model number, the hedge in "I think it was Tuesday." A summary reads well and quietly launders detail into paraphrase.

Which is why it runs last and only when needed, rather than being the elegant single answer it looks like on a whiteboard.

The order is the design

The four strategies compose in a pipeline, and the order encodes a single principle: discard by cost of loss, ascending.

A stale API payload costs almost nothing to lose. A message from twenty turns ago costs a little. An arbitrary cut costs more. A paraphrase of what someone actually said costs the most, because it is the only one where what remains is subtly untrue rather than merely absent.

Most compaction bugs we have seen are order bugs, not strategy bugs. Summarise first and you burn a model call on a history that was ninety percent recoverable JSON. Slide the window before collapsing tool results and you evict a customer's actual words while keeping a kilobyte of stock levels.

The thing to watch for

Compaction failures do not look like failures. The agent does not error, it does not warn, it does not go quiet. It gets slightly stupider in a way that is invisible in the response and obvious in the transcript.

It asks for the order number a second time. It re-explains something it already covered. It answers the question as if the previous three turns had not happened, which from its position they had not.

If you are debugging an agent that seems fine in short conversations and vague in long ones, you are not looking at a prompt problem. You are looking at what your compaction threw away.