chaffchaff

Concepts

Two ideas do most of the work: the resident/on-demand/dropped split, and a chaff% definition built to survive a hostile reading.

Resident, on-demand, dropped

A context file does not cost the same amount every turn. Claude Code’s own documented rules put every entry into one of three classes, and chaff models them separately instead of reporting one undifferentiated token count:

  • Resident, loaded in full every turn whether or not the turn needs it: the root CLAUDE.md, ancestor CLAUDE.md/CLAUDE.local.md files, their @-imports, tool definitions, .claude/rules/*.md files with no paths: key, and MEMORY.md up to its read limit.
  • On-demand, loaded only when something routes to it: skill and subagent bodies (their frontmatter description is resident; the body is not), .claude/rules/*.md files scoped with paths:, subdirectory CLAUDE.md files, and MEMORY.md topic files.
  • Dropped: never loaded at all. MEMORY.md content past the documented 200-line / 25 KB cut (whichever comes first) is cut on every load, silently, whether the author knows it or not.

For the generic harness, none of this is guessed: nothing is resident unless you declare it, and an undeclared entry prints a note rather than an assumption.

Resident, on-demand, and dropped tokens: a real chaff profile A horizontal bar chart with three bands, drawn to scale from a real chaff analysis of the committed fixtures/case-study/before context set (o200k_base tokenizer, --today 2026-08-09). Resident: 5,904 ref tokens across 2 file(s); this content loads into the model's context on every turn, filled solid. On-demand: 184 ref tokens across 1 file(s); this content loads only when a path-scoped rule matches the file being worked on, drawn as an outline. Dropped: 109 ref tokens; content past MEMORY.md's documented 200-line read limit that is silently cut and never reaches the model at all, filled amber, the one amber element in this diagram. Bars are linearly proportional to token count against a shared 0-6,000 scale; the two smaller bands use a minimum visible width for legibility, and the exact count is always printed as a label regardless of bar width. Reproduce with chaff analyze --today 2026-08-09 against fixtures/case-study/before. CHAFF'S OWN CASE-STUDY FIXTURE, BEFORE THE FIX resident vs. on-demand vs. dropped — o200k_base ref tokens, one real file set RESIDENT loads on every turn 5,904 tok · 2 files ON-DEMAND loads only when a rule path matches 184 tok · 1 file DROPPED past the read limit, never reaches the model 109 tok 0 1,000 2,000 3,000 4,000 5,000 6,000 ref tok (o200k_base)

The chaff% numerator, defined to survive a hostile reading

chaff% = (resident tokens under enforced-finding spans, deduplicated) / (total resident tokens), per set, per named tokenizer. Every clause in that definition closes a specific way the number could be gamed or misread:

enforced-finding spans only
Heuristic findings are excluded entirely. A believed-but-unmeasured claim cannot inflate a score that is supposed to mean “evidence-backed.” Their token weight still gets reported as the separate UNVERIFIED line, so it is visible without being counted.
deduplicated
Overlapping spans are merged before counting, so two rules flagging the same bytes cannot double the number. No rule may claim a span larger than the text it identifies either: a finding’s anchor cannot quietly annex neighboring, unflagged prose.
resident tokens only
On-demand and dropped tokens never enter either side of the ratio. A file that is mostly on-demand content cannot be made to look worse (or better) by folding tokens it never pays for every turn into the count.
set-level aggregates are excluded from the numerator, on purpose
budget.resident-exceeded asserts a fact about the whole set (“the total exceeds the declared budget”), not that any one span is removable. It still anchors on the largest resident contributor for reporting, and it still drives exit code 3. But its tokens are excluded from the chaff% numerator entirely, not partially: counting that anchor span would let one aggregate fact claim an entire file as chaff, which is exactly what “no rule may claim a span larger than the text it identifies” forbids. A structural finding can even exist with zero effect on chaff% for the same reason, when its span points at content that was never resident to begin with. The case study has a real example: memory.index-over-read-limit fires on content past the read limit, which by definition is already in the DROPPED band, not the resident one, so that enforced finding contributes nothing to the numerator either.

The absolute numerator is always printed beside the percentage, because a bare ratio is a vanity metric. And the honest caveat, stated plainly rather than buried: chaff% is the share of your always-loaded tokens that chaff can point at a specific, evidence-backed reason to remove. It is not a quality score. A 0% file can still be a bad file; it just means nothing in it cleared the enforced bar yet.