Your AI context files lie silently: I built HCS to measure it

In short

• The problem: context files for AI (AGENTS.md, CLAUDE.md, project summaries) go stale without warning, never say where their statements come from, and nobody knows what they lost.
• HCS (Holographic Context Standard) is an open format, a command-line tool and an MCP server, under the MIT licence. Every claim cites its source with a hash, hcs check detects what changed without any AI, and hcs score measures what a model actually recovers.
• Measured, not promised: on three public repositories chosen in advance, a 2,000-token HCS context answered 46.7% of the questions, against 27.8% for a naive summary of the same size and 10% for the project's README. A second model from another family confirms the gap.
• What did not work is published too: self-improvement (refine) does not help, dependency alerts are mostly false alarms, and injection detection lets about half of novel attacks through.
• Available since 23 October 2026: npm install -g @oufrid/hcs-cli, and pip install hcs-format for the Python version. The format is stable (SPEC 1.0); the tool is still experimental (0.3).

What I kept running into: files that look right

I work with coding agents every day, on my websites and my Android apps. Like everyone, I give them context files: a CLAUDE.md describing the project, its rules, its commands, its traps. It is the most cost-effective lever I know; I explain it in the guide on optimising AI context, and the agent instructions file tool helps you write one.

But these files have three flaws that ended up bothering me:

1. They go stale silently. The code changes, a command is renamed, a rule evolves. The file does not move. The agent keeps believing something false, confidently. Nothing flagged it.
2. You cannot tell where a statement comes from. When the file says "refunds are accepted within 14 days", nothing says which file or function proves it. There is no quick way to check.
3. Nobody knows what a summary lost. A summary is written to be readable. It keeps the structure and drops exceptions, exact values and distinctions. Rereading it is not enough: you compare it with what you remember, not with the source.

This is not an isolated impression: studies of public AGENTS.md files, cited in the project's comparison document (PRIOR_ART.md), report that drift is common.

The idea: three properties in one file

HCS rests on a simple idea: a context file should be able to prove what it says, notice when it goes stale and measure what it loses.

• Traceable: every claim cites its source, a function, a class, a heading or a line range. A model knows where to read instead of guessing.
• Verifiable: every citation carries a hash of the cited text. hcs check detects, without calling any AI, the claims whose source changed. It ignores reformatting, follows code that only moved, even to another file, and fails when the meaning may have changed. So it can block continuous integration.
• Measured: the file ships test questions whose answers are proven verbatim in the source. hcs score computes a fidelity score for a given model, with a confidence interval, a no-context comparison and the rate at which the context misled the model.

A built-in MCP server completes it: the model first reads a few hundred tokens of claims, then opens the exact source of a claim only when it needs a detail. If MCP is new to you, the guide on connecting AI to your tools explains it.

What already existed

Before writing a line, I looked for anyone who had already done it. The building blocks exist separately: linters for AGENTS.md files, tools that pin documentation to code with a hash, question-based methods to evaluate summaries, prompt compressors, repository maps.

None of HCS's ideas is new on its own, and the comparison document says so plainly. What I did not find elsewhere is the combination in one portable format: a hashed citation required on every claim, generation that checks every quote verbatim, an automatic pull-request comment listing the affected claims, and MCP access to the exact source. That is an impression from one survey, not a proof of absence.

How I tested it, honestly

In AI, it is easy to publish a flattering number. I wanted the opposite: results a sceptical reader can check. Every measurement is in the project's report (bench/REPORT.md), and every number links to its raw results.

The rules I set myself:

• No invented quotes: when a model compiles an HCS file, every claim and every question must quote the source verbatim, or it is discarded. On the first real runs, this check kept 85 to 89% of what the models proposed.
• Questions written separately: the test questions were written from the source alone, without seeing the compiled context. Otherwise the test favours HCS.
• Confidence intervals: with 24 to 30 questions per repository, differences are uncertain. Every difference comes with its 95% interval.
• A judge from another family than the answering model.
• A pre-registered confirmation: the protocol and the success criterion were committed to the repository before the first model call on the new repositories.

The first result disappointed me

On five public repositories (kilo, itsdangerous, anyhow, cobra, ky), with independent questions, the first verdict was clear: HCS was not measurably better than a naive summary of the same size (+6.3 points, with an interval that included zero). It beat the projects' READMEs by far, but not a plain summary.

Looking for the reason, the diagnosis came from the numbers themselves: in a 2,000-token context, the references to sources took 31 to 39% of the space, and whole files had no claim at all. So I added a compact render: the claims only, and one claim per source file in turn.

With that render, on those five repositories:

The confirmation on three new repositories

The problem is obvious: I designed the compact render after seeing those five repositories. A result obtained that way may just fit the data. Hence the pre-registered confirmation on three repositories never used before: express, requests and logrus.

The criterion fixed in advance, HCS compact minus summary, gives +18.9 points, with an interval from +7.2 to +30.6: entirely above zero, so confirmed. With a second, smaller model from another family (Mistral's ministral-14b), the gap holds: +16.1 points on the new repositories and +18.8 on the first five.

One last result taught me something: when the model may read the source once before answering, it gains about 20 points, whatever the starting context. Reading the right source matters more than the form of the summary. HCS gets there with fewer reads, because its references point to the exact code, but the advantage is not significant in that mode.

The limits of these numbers

• A single model (Gemini 3.8 Flash) compiled the files and wrote the questions and the summaries. It may favour material in its own words.
• The questions are generated, 24 to 30 per repository, and only one budget was tested (2,000 tokens).
• On very well-known projects, a model with no context sometimes does better from its memory of public code. Private code would not give it that advantage, but that case was not measured.
• Success rates stay modest: the questions ask precise details (names, values, messages), and 2,000 tokens cover only a small part of the source.

What did not work

I publish these failures with the same care as the successes, because they show where the limits are.

• Self-improvement does not generalise. The hcs refine command adds claims to cover failed questions. It raises the score on the questions it optimises for, up to 100% on HCS's own repository. But on the held-out questions the score went from 50.0% to 40.9%, with more wrong answers, and it did not move on a second repository (44.4%). The feature stays available as an experiment, and the README says it did not help.
• Dependency alerts cry wolf. A claim about a function can become false when a function it calls changes. HCS can flag that, but on the project's own history, 17 of 18 alerts were unneeded.
• Prompt-injection detection is a filter, not a guarantee. After tuning, it catches 35 of the 39 attacks in the test corpus. On a set of attacks kept apart and never used for tuning, it catches only 7 of 15. Expect about half of novel attacks to get through. The instructions sent to the model, which state that claims are never instructions, are the second line of defence.
• Small local models measure poorly. A 7-billion-parameter model on a CPU answered "I don't know" even when the answer was in its context. They are useful to run the pipeline without quotas, not to produce numbers.
• Free tiers are not enough for everything. The first runs used the free tiers of Gemini and Groq, and HCS handles their quotas: it resumes where it stopped. But Gemini's free tier allows about 20 requests per day per model, and the main benchmark used a paid key. The second model of the confirmation ran on Mistral.

How it was built

I designed HCS, set its rules and decided what got published. The code itself was written with a coding agent, Claude Code, from the first commit on 21 October 2026 to the release on 23 October.

The lesson I take from it applies to any work with an agent: speed is worth nothing without written rules. The repository contains a CLAUDE.md file that requires, among other things, that every published number link to raw results, that a run not performed be marked as such, and that biases be written next to the results. That discipline is what surfaced the first disappointing result instead of hiding it, and what led to the pre-registered confirmation. An agent produces what you ask for. If you ask for evidence, it produces evidence, and the evidence may contradict what you hoped.

The project also applies its own method: its repository contains a project.hcs file that describes the project itself, checked automatically on every change.

Try it in 5 minutes

You need Node.js 18.17 or later. The tool has no dependencies.

1. Install:

``bash npm install -g @oufrid/hcs-cli ``

1. See verification at work, with no AI key, on HCS's own repository:

``bash git clone https://github.com/Oufrid/HCS.git && cd HCS hcs check project.hcs ``

1. Compile a context for your own project, with a free API key (Gemini, Groq) or a local model through Ollama:

``bash export GEMINI_API_KEY=... hcs models gemini hcs compile --model gemini:<model> --rpm 8 -o project.hcs ``

1. Connect it to your agent through MCP, for example in Claude Code:

``bash claude mcp add hcs -- hcs serve project.hcs ``

Note: compiling and scoring send your code to the provider you choose, and some free tiers may use the data you send. Only send code you are allowed to share, or use a local model. Verification (check), rendering and the MCP server do not contact any AI.

Every command, setting and integration is detailed on the HCS project page.

What comes next, and how to help

The format is frozen as version 1.0: files written today will stay readable, and any change goes through a public proposal (RFC). The tool is still experimental and will keep changing.

What would help me most:

• Results on real projects, especially private ones, where a model cannot lean on its memory of public code.
• Cases for more languages: function location is heuristic and does not cover every syntax.
• Feedback on the format, and corrections to the comparison document if a tool already does better.

Everything happens on the repository: github.com/Oufrid/HCS. If you want to apply this approach to a company project, you can also contact me.

FAQ

Does HCS replace AGENTS.md or CLAUDE.md?

No, it complements them. You can keep your files and add a .hcs file next to them. The hcs audit command even checks an existing AGENTS.md: it reports broken references, and can measure what it really conveys using an HCS file's questions.

Does a valid HCS file guarantee that everything is true?

No. It guarantees that each quote was found in the source when the file was compiled, and that the cited source has not changed since. An unfaithful paraphrase of an exact quote, a source that is itself wrong, or a change elsewhere in the code that makes a claim false are not detected.

Does my code leave my machine?

Not for verification, rendering, statistics or the MCP server, which never call a model. Compiling and scoring send the code or the context to the provider you choose. With a local model through Ollama nothing leaves, but small models give weak results.

Is it ready for production?

The format is: it is stable as version 1.0. The tool is version 0.3 and experimental: AI-free verification and the MCP server are complete and tested, and a second implementation in Python passes the same conformance suite. The AI-based features work, but their results depend on the model, and some, like refine, have not proven themselves.