Rule admission pipeline

Three arms

Every battery runs the same file in three states, on the same items:

• present: the flagged pattern left in place.
• removed: chaff’s own suggested fix applied, and nothing else.
• control: an unrelated edit of equal token delta (±5%, verified from recorded counts, not asserted) elsewhere in the same file. The control is what separates “this pattern hurts” from “shorter context helps.” Without it, a rule that only ever removes tokens could look like it works for the wrong reason.

Battery and runs

At least 12 items per rule, each scored by a named deterministic scorer: did the run take the action the current instruction requires, did it select the right tool, did it read the on-demand file rather than guess. No LLM judge anywhere in the gate; that is what lets CI re-derive the verdict for free, forever, without spending anything. k ≥ 5 repeats per (arm × item), k ≥ 3 is the admissible floor, temperature 1.0 (the deployed condition, not a variance-suppressed one). Primary model claude-haiku-4-5; confirmation run on claude-sonnet-5. A rule admitted on one model is labeled with that model everywhere it appears. chaff never generalizes a measured effect past what was actually measured.

The admission test: pre-registered, recomputed by CI

All four must hold:

1. removed − control ≥ +0.15 absolute pass rate.
2. The 95% CI of removed − control from a seeded paired bootstrap over items (10,000 draws, fixed seed) excludes 0.
3. Sign consistency: removed ≥ control on at least 8 of 12 items.
4. Direction sanity: present ≤ removed.

The run file’s own verdict field is advisory only. CI recomputes the admission statistic from the raw observation matrix with the fixed bootstrap seed and ignores whatever the run file claims: the artifact is checked, not trusted. CI also verifies the config hash and battery hash in the newest run equal a recomputation over the current files; the shape gates (exactly 3 arms, items ≥ 12, k ≥ 3, control token delta within ±5%); that the run’s start time is after its pre-registration time; and staleness. A run older than 365 days, or naming a model absent from the current pricing manifest, fails the build outright with “re-run or demote.” Silent decay is the exact failure mode this whole design exists to prevent.

Graduation, and the graveyard

Heuristic to enforced happens in two PRs: the first commits eval.config.json plus the battery (pre-registration, merged before any run happens). The second commits the run data and flips the tier. A failed run is committed too, and the rule moves to the graveyard. “Rules we believed and could not measure” is the single most credible page a static-analysis tool can publish, and it costs nothing to keep honest. The reverse, enforced back to heuristic, happens the same way, dated, with the failing run kept rather than deleted.

Cost, capped, never in CI

The eval runner is local-only and needs a real API key. It never runs in CI, and CI never holds a key. A pre-flight estimate plus a running counter from each response’s own usage field hard-aborts on a per-rule cap (default $2.00) and a program-wide cap (default $25). The full three-arm, 12-item, k=5 matrix is roughly 180 calls; at that shape the two-model run for one rule costs on the order of a dollar. This is the only place in the whole product that can spend money, and it is the one piece explicitly out of scope for this build.