Optimization Tips for Freeing Tokens in Long-Running Agents

The highest-leverage optimization for a long-running agent is conversation compaction. Instead of carrying every raw message into the window, periodically replace the oldest block of turns with a compact summary that preserves the decisions, facts, and unresolved questions. A summarised twenty-turn history can shrink from thousands of tokens to a few hundred while retaining everything the model actually needs. The matrix's long-conversation warning is the trigger; when utilization climbs past roughly 60%, schedule the next compaction point.

Memory eviction is the companion discipline. Long-term memory that grows without bound is a leaking bucket: every stale fact, outdated preference, and abandoned project still costs tokens on every request. Implement a retention policy that keeps high-value entries, compresses aging entries into shorter forms, and archives or deletes entries past their useful life. Run the matrix after a cleanup pass and watch the memory row shrink, which is the clearest possible proof the policy is working.

Cache and trim tool-call output aggressively. Agents often feed large tool responses back into the conversation, and those responses repeat on every retry or related call. Store the parsed result, extract only the fields the next step needs, and inject the condensed form rather than the raw JSON. Over a long session, this single habit can save thousands of tokens that would otherwise be spent re-reading the same response verbatim.

Trim the system prompt as a standing maintenance task. Instruction text is paid on every request with no degradation, so it is the most expensive content in the entire budget. Split rarely-needed rules into tool descriptions or a reference file the model reads only when relevant, and delete guidance that has been overtaken by model capabilities. Track the system prompt's token count in the matrix and treat any growth as a change that requires justification.

Use relevance filtering to inject less retrieval. The goal is not to retrieve everything that scores above a low bar, but to inject the few chunks that actually answer the current question. Raise the relevance threshold, use reranking, and cap the injected chunk count, because each extra chunk is paid on every turn of the session. The matrix's recommended memory allocation gives you a concrete budget, and a retrieval pipeline that fits inside it is one that can never overflow the window on its own.

Right-size chunking for the model, not just for embeddings. Shorter chunks, around 200 to 500 tokens, are easier for the model to consume and give retrieval finer granularity, but they also mean more chunks to reach the same coverage. Balance chunk size against the injection cap so the effective memory cost stays within budget. Experiment with a few chunk sizes and record the resulting utilization in the matrix until you find the combination that fits with margin.

Prefer one-pass compact designs over re-reading. If the agent needs a fact that appeared two hundred turns ago, either the memory layer surfaces it as a summary or the conversation must replay it, and replaying costs the full original tokens. Design the memory layer to answer lookups from stored summaries rather than forcing the model to scan the entire history. Good memory design is the difference between a constant-cost agent and one whose token spend grows linearly with session length.

Give the model instructions to be terse in replies. Assistant-side tokens are part of the conversation component, and verbose assistant messages inflate the average that feeds the budget. Instruct the model to answer directly, avoid echoing user content, and omit boilerplate. Across dozens of turns, small per-reply savings compound into real headroom, and the behaviour is easy to enforce in the system prompt.

Monitor utilization trends over time, not just at planning time. An agent that fits its window on day one can drift past 80% as features accumulate and memory grows. Log the utilization after each session, plot it against turn count, and alert when the trend line approaches the warning thresholds. Trending is the difference between catching a problem in planning and discovering it through a production truncation incident.

Finally, make the budget part of the release review. When a change touches the system prompt, retrieval, model, or average reply length, re-run the matrix and compare utilization before and after, like a performance regression check. Teams that gate changes on token-budget impact rarely meet context-overflow surprises, because every contributor knows the window is a bounded, monitored resource. It turns budget optimization from an occasional emergency into a continuous habit.