The Complete Guide to Budgeting an LLM Context Window

Every LLM has a hard ceiling on how much text it can see at once, the context window. A GPT-4o style model offers roughly 128,000 tokens, Claude Sonnet 4 about 200,000, and Gemini-class models a million. When the text you feed a model, its system prompt, the running conversation, the memory and retrieval content you inject, plus the reply it needs to produce, exceeds that ceiling, the request fails or, worse, silently truncates the oldest information. The Long-Term Memory Token Matrix exists to turn that vague risk into a concrete number you can plan against.

The matrix treats the context window as a shared budget with four spending categories. The system prompt is the standing instruction set that describes the agent's role, rules, and tools. The conversation is the accumulated exchange of user and assistant messages. Memory and RAG content is the injected long-term knowledge: user preferences, past summaries, and retrieved document chunks. Reserved output is the space the model needs to generate its reply. Summing these four gives the projected token count of a request, and comparing it against the model's maximum reveals utilization and headroom.

Estimating the conversation component is where most agents go wrong. You rarely know the exact token count of every message, so the matrix uses an average: the typical user message tokens plus the typical assistant message tokens, multiplied by the number of turns. A session with twenty turns, an average of 120 tokens per user message and 260 per assistant reply, contributes about 7,600 tokens. Keeping these averages realistic matters more than precision, because the purpose is early planning, not metering a specific request.

Memory and RAG content is the most volatile category and the one agents control most easily. A retrieval step that injects six relevant chunks at roughly 1,000 tokens each adds 6,000 tokens to every request. The matrix computes a recommended memory allocation, the amount of RAG and memory content the window can absorb while staying near a comfortable 70% utilization target after accounting for system, conversation, and output. That number tells you whether your retrieval pipeline can keep growing or needs chunk-size discipline.

Reserved output is the category that gets forgotten first. Models need space to generate, and a long reply plus tool-calling overhead can consume thousands of tokens. If you reserve nothing, a model that has spent its entire window reading input can fail mid-generation or cut itself off. The matrix defaults to 2,048 reserved tokens, roughly a long structured reply, and warns when the reservation is too small for tool-calling agents that emit verbose JSON.

Utilization is the scoreboard. Total tokens divided by the maximum context, expressed as a percentage, tells you how close a request sits to the edge. Under 60% is comfortable, 60 to 75% deserves monitoring, 75 to 90% is where compaction should start, and above 90% is where truncation becomes likely. The matrix color-codes the utilization bar green, amber, or red so the risk level is readable at a glance, and it reports the exact remaining budget in tokens so you know how much slack you actually have.

The breakdown table distributes the context window across components, showing each category's token share and its percentage of the whole. This is where budgeting disagreements surface: a table that shows memory eating 55% of the window explains why long conversations feel worse than they should, and a row for conversation that dominates explains why the model forgets the beginning of a session. The percentages make trade-offs visible and actionable.

Compaction warnings are the matrix's early-warning system. Rather than waiting for an overflow error in production, the tool flags thresholds as you type: critical above 90%, warning above 75%, and advisory notes when memory dominates the budget or a long conversation suggests summarization. Each warning is phrased as an action, reduce memory, summarize old turns, or move to a larger model, so the number leads somewhere instead of just alarming.

Every number in the matrix is a planning estimate, because tokens are not characters. The same English text can tokenize differently across GPT, Claude, and Gemini tokenizers, and code, punctuation, and non-Latin scripts inflate counts unpredictably. The matrix is a budgeting compass, not a billing meter; you still want a real tokenizer or the provider's usage reporting to confirm actual counts, but planning with estimates is far better than planning with nothing.

The practical workflow is to run the matrix before you architect the agent and again before each release. On the first pass, it catches a system prompt that is too fat or a retrieval design that assumes unlimited memory. On the release pass, it validates that the current turn counts and memory sizes still fit the chosen model. Because the tool runs entirely in the browser, you can experiment with model choices, turn counts, and memory sizes freely and settle on a configuration that fits comfortably before a single line of production code changes.