Optimising an AI's context: doing more with less

In short

• The context window is not a hard drive. The more you fill it, the harder it is for the model to find what matters: this is documented by research, not a hunch.
• The biggest gain is free: short, structured instructions, a project instructions file, and tasks split into pieces.
• Prompt caching sharply cuts the cost and latency of the parts that repeat, as long as stable content comes first.
• Search instead of pasting everything: an agent that reads the right file at the right time almost always beats a 200-page prompt.
• A summary can look perfect and still have lost the essentials. The only way to know is to test it with questions.
• I write from experience, my mistakes included, and about HCS, the open-source project I am building on this problem, still experimental.

What the context window really is

A language model "remembers" nothing between two requests. On every call, it receives a block of text, split into tokens, and answers from that block only. That block is the context. Its maximum size is the context window.

When you chat with an assistant, it feels like a continuous conversation. In reality, with every message the application sends back all or part of the history. With a coding agent it is even clearer: the context fills up surprisingly fast.

What actually fills the context

In a real working session with an agent, this is what occupies the window, in the order I see it grow:

1. The system instructions of the tool itself, which you never see.
2. Tool definitions: every available tool (read a file, run a command, search the web) is described to the model, with its parameters.
3. Project instructions files (AGENTS.md, CLAUDE.md) and persistent memory.
4. The conversation history: your messages and its replies.
5. Tool results: the content of files read, command output, web pages visited. This is almost always the heaviest item.
6. Documents pasted by the user.

A single 2,000-line file read in full, a command that prints a long error log or a whole web page can weigh more than the rest of the conversation combined.

What an overloaded context costs

Price. APIs bill every input token, on every call. A 100,000-token context sent ten times means one million input tokens billed. To estimate the impact by model, use the AI API cost calculator.

Latency. The model must read the whole context before producing the first word. The longer it is, the longer the wait.

Quality. This is the most counter-intuitive point, and it is well documented:

• The Lost in the Middle study (Liu et al., published in Transactions of the Association for Computational Linguistics) showed that models retrieve information placed at the start or end of a long context better than in the middle, including models designed for long contexts (arXiv 2307.03172).
• In July 2025, Chroma's research team tested 18 models and concluded that their performance becomes less and less reliable as input grows, even on simple tasks, and unevenly so (Context Rot).
• Anthropic sums up the principle in its article on context engineering: context must be treated as a finite resource with diminishing returns (Effective context engineering for AI agents).

A large context window is a capacity, not a goal. The right context is the smallest set of information that lets the model do the task well.

The levers, from simplest to most advanced

1. Short, structured, precise instructions

This is the most profitable lever, and it costs nothing.

When to use it: always.

What works:

• State the goal in one sentence, then the useful context (not the whole story), then the constraints, then the expected output format.
• Replace long explanations with lists and short examples.
• Remove what does not change the answer: repeated pleasantries, project history, "just in case" details.
• Ask explicitly for a short answer when possible: output is billed too, and it comes back into the context on the next turn.

Example: instead of pasting a whole file and writing "see what's wrong", I paste the function concerned, the exact error message and one sentence: "This test has failed since the cache was added; find the cause and propose the minimal fix." The prompt library applies this structure to 30 common cases.

2. A project instructions file

Instead of re-explaining the project every session, you write it once in a file the agent loads automatically: AGENTS.md (open format, see agents.md), CLAUDE.md for Claude Code (Claude Code memory documentation), or their equivalents.

When to use it: as soon as a project lasts more than one session.

What it should contain: the commands (install, build, test), the folder structure, the conventions, the areas never to touch, and the verification rules. Not the full project documentation.

The trap: an instructions file that is too long dilutes itself, and it is re-read every session. One or two pages are enough. The instructions file generator produces a clean base, with a quality score.

3. Split tasks, and use sub-agents

A single session that does "the whole site refactoring" ends up saturated with tool results that are no longer useful. Several short sessions, each with a clear goal, keep a clean context.

When to use it: for any task touching more than a few files, or mixing research and modification.

Sub-agents take the idea further: a lead agent hands a sub-task (explore a folder, check facts, write a file) to a sub-agent that works in its own context, then returns only a summary. The main context receives only the conclusion, not the hundreds of lines read to reach it. This is one of the architectures Anthropic describes in the article cited above.

From experience: to write the guides of this Lab, I gave each guide to a separate agent, with a precise brief. The lead agent only received each report: word count, verified sources, limits. Without that split, the main session would have drowned in web searches.

4. Prompt caching

When a large part of the context repeats from one call to the next (system instructions, tool definitions, a reference document), the major providers offer to keep it in cache. Subsequent calls that start with the same beginning are then processed faster and billed less.

How it works, according to the official documentation:

Sources: OpenAI, Anthropic, Google. Exact prices change often: the API cost calculator applies the published discounts, with their check date.

The golden rule, common to all three: put stable content first (instructions, tools, reference documents) and variable content last (today's question, the date, the user's data). Change a single character at the beginning and the whole cache after it is lost.

The traps:

• A date or a session ID at the very top of the prompt breaks the cache on every call.
• Changing the list or order of tools invalidates the cache at Anthropic and degrades reuse at OpenAI.
• Each provider has a minimum size below which nothing is cached, with no error message. Check the usage fields of the response to make sure the cache is actually used.

When to use it: chatbots with long instructions, assistants that always query the same document, agents that chain many calls.

5. Search instead of pasting everything (RAG and on-demand reading)

Pasting the whole documentation into the prompt "to be safe" is the most common mistake. Two alternatives:

• Retrieval-augmented generation (RAG). Documents are split and indexed in advance. For each question, the model only gets the few most relevant passages. The approach was formalised by Lewis et al. (arXiv 2005.11401).
• On-demand reading. This is what coding agents do: instead of loading everything, they keep references (file paths, queries) and read the content when they need it, with their search tools. Anthropic describes both approaches and their trade-offs: on-demand reading is slower, hence hybrid approaches.

When to use it: as soon as the knowledge base exceeds what you would reread yourself to answer a question. A two-page FAQ can be pasted; a 3,000-product catalogue or a whole code repository cannot.

The RAG trap: if the search brings back the wrong passages, the model answers confidently from wrong information. A good RAG is tested, with questions whose answer you know, as we will see below.

To connect an AI to your documents, tools and machines (APIs, MCP, agents), see the companion guide Connecting an AI to your tools and machines.

6. Summarise and compact long sessions

When a conversation nears the window limit, recent tools compact it: they replace the history with a summary and start again from there. A lighter form is clearing old tool results that are no longer needed.

It is essential for long sessions. It is also where the sneakiest losses hide.

My experience, unvarnished. During a very long working session on this site, the conversation was compacted automatically. The summary was good overall. But one detail had vanished: there were two different forms, one to notify visitors when an app is released, and a Google Form to enrol testers. After compaction, the agent mixed them up, then asked me for a link it could have found itself in the server's files. Nothing serious, but exactly the kind of error a summary produces: it keeps the topic and loses the distinction.

What limits the damage:

• Before a compaction, ask the agent to write the state of the work to a file: decisions made, files changed, next steps, known traps.
• After a compaction, require the agent to check the files rather than trust its memory. It is a rule I added to my instructions files: "check the repository instead of assuming".
• Prefer several short sessions, each started from a written state, to one endless session.

7. Persistent memory and project notes

Notes written outside the context window, then reread when needed, are the most robust form of memory. Anthropic describes this technique as structured note-taking: the agent regularly writes notes kept outside the context and reloads them later.

What I keep in notes, for each project:

• server paths, services and commands;
• decisions made and their reason ("why we don't do X");
• traps already met and their fix;
• the preferences of the person I work with.

What I never put there: secrets (passwords, tokens, API keys). A project note is read by the agent every session: a secret in it ends up in a conversation sooner or later. Before pasting sensitive text into an AI, the anonymiser replaces personal data and keys with placeholders.

The trap: a wrong note is worse than no note, because the agent believes it. A memory is reread and corrected, like an instructions file.

8. Token budgets

The last lever, more technical: set a limit and choose what goes into it.

• Cap answer length when you do not need a novel.
• Truncate tool outputs: the last 50 lines of a log are often enough.
• Fit the context to the model: a small local model has a shorter window and suffers sooner from a heavy context. The local AI hardware calculator also shows how much memory the context itself consumes (the KV cache).

Measuring what you lose

Why a summary "looks good"

A summary is written to be readable. It keeps the overall structure and the keywords. What it loses are the exceptions, the exact figures and the distinctions (two similar forms, two versions of a rule, an option disabled "for now"). Yet these are often what cause errors.

Rereading the summary is not enough, because you compare it with what you remember, not with the source.

The test-question method

The only reliable method I know: ask questions whose answer is in the source, and check whether the model, with the summary alone, answers correctly.

1. Write 10 to 30 precise questions: "Which header carries the idempotency key?", "What is the default branch?", "Which permission is the token missing?".
2. Note the exact answer, with where it is in the source.
3. Ask the questions to the model with the summary, then with no context at all: the difference shows what the summary really brings.
4. Count right answers, wrong answers and "I don't know". A wrong answer given confidently is the worst case.

It is the same idea as the test plans of the AI assistant designer, applied to context. Several tools already measure a compression or a summary with questions, for example the probe-based evaluation published by Factory, DeepEval's summarisation metric or Ragas faithfulness: these are methods and metrics, not file formats.

HCS: my open-source project on this problem

Running into these limits is what led me to start HCS (Holographic Context Standard), an open-source project under the MIT licence (github.com/Oufrid/HCS, presented among the Lab projects).

The idea: a context file where every claim cites its source, and which can tell when it becomes false.

• Traceable: every claim points to a precise passage (a function, a class, a heading, a line range).
• Verifiable: every citation carries a fingerprint (hash) of the cited text. The hcs check command detects, without AI, the claims whose source has changed; it ignores mere reformatting and follows code that has only moved. It can fail continuous integration.
• Measured: the file ships test questions whose answer is verified word for word in the source, and hcs score computes a fidelity score for a given model, with a confidence interval and a no-context comparison.
• Served over MCP: a built-in server lets the model read the compact claims first, then open the exact source when it needs it.

What you should know, honestly:

• HCS is experimental (version 0.2). The format, the AI-free verification and the MCP server are complete and tested.
• The AI-based features (compiling a file, measuring fidelity) are implemented and tested against simulated servers, but have not yet been measured on real models. Results will be published with their raw data, not as promises.
• None of the ideas is new on its own. HCS combines them in a portable format. It complements existing formats (AGENTS.md, CLAUDE.md, llms.txt), repository packers such as Repomix, Aider's repository map or documentation drift detectors, rather than replacing them.

If the topic interests you, feedback and corrections are welcome on the repository.

Summary table

Common mistakes

• Pasting everything "to be safe". The model gets lost in the middle, and the bill climbs.
• A date at the top of the prompt. The cache is lost on every call.
• A ten-page instructions file. It dilutes itself and costs every session.
• Trusting a summary because it is well written. Readability is not fidelity.
• Letting a session run for hours without ever writing down the state of the work.
• Putting secrets in project notes, which are reread every session.
• Choosing the model with the largest window to make up for a badly organised context.

The checklist

1. Does my instruction fit into goal, useful context, constraints, format?
2. Does my project have a short, up-to-date instructions file?
3. Can this task be split, or handed to a sub-agent?
4. Is the stable content at the start of the prompt, and the variable content at the end?
5. Have I checked in the usage fields that the cache works?
6. Am I pasting a whole document when a search would do?
7. Before a long session, have I planned a state file?
8. After a compaction, does the agent check the files?
9. Are my project notes accurate, and free of secrets?
10. Have I tested my summary or my RAG with questions whose answer I know?

FAQ

Does a larger context window solve the problem?

No. It pushes the limit back, but research shows that models make less good use of information buried in a long context, especially in the middle. A large window is useful for the cases that need it, not as an excuse not to organise the context.

Does caching change the model's answer?

No. It only avoids reprocessing an identical prompt beginning. The model receives the same content; only cost and delay change.

Do I need a RAG for a small site or a small FAQ?

Not necessarily. If all the useful content fits in a few pages, putting it in the instructions, with caching, is often simpler and more reliable. RAG becomes worthwhile when the base exceeds what the model can read efficiently for each question.

How do I know if my agent lost information after a compaction?

Ask it precise questions about the project's key decisions and distinctions, and compare with your notes. Better: before the compaction, have the state of the work written to a file, and ask the agent to refer to it.

Is HCS ready for production use?

Not yet. The AI-free verification and the MCP server are tested, but the compile and measurement features have not yet been validated on real models. It is an experimental project, to try knowingly.

Do small local models suffer more from a long context?

Usually their window is shorter, and the context consumes memory on your machine. A tight context therefore matters even more locally. The hardware calculator shows the memory taken by the context depending on the model.

Is optimising context only about cost?

No. Cost is the most visible part, but the main gain is often quality: a clean context reduces confusion, omissions and invented answers.

---

Want an assistant or an automation that stays reliable and cost-controlled, with a well-designed context from day one? That is exactly what I set up in my AI and automation service. Let's talk about your project.