We make our scanner grade itself — on a holdout it has never seen

Preregister, then don't touch the holdout

Before running anything, we froze the scoring rubric and wrote down the pass/fail gates — precision floors, recall envelopes, false-positive ceilings, minimum evidence counts — and committed that document first, so we couldn't move the goalposts after seeing results.

Then a deterministic, seeded procedure selected a 200-repository holdout and froze it. The rule we held to: the holdout is selected once, never expanded, never adjusted after outcomes are visible, and no result is used to tune the rubric being measured. Every repository from every prior experiment was excluded, so nothing the detector had already seen could leak in. The scan is static and offline by construction — it reads tracked files at a frozen revision, never executes the repository's code, removes remotes before scoring, and treats “insufficient evidence” as a recorded outcome rather than a reason to guess.

How we label evidence

The results below are Measured. The preregistration, the frozen gate table and the terminal decision are Verifiable — they are published in the repository. Calibration frames are single-use: their membership is bound by a SHA-256 commitment when the frame freezes and stays closed while the evaluation is live. When a frame retires, its pinned members are published in a retirement reveal and the frame is never reused. The v17 frame is retired, and its sanctioned membership reveal records all 200 pinned revisions. The canonicalization and digest rules are published in the hash conventions. Adjudication labels and reviewer records remain closed; revealing membership does not make the accuracy figures independently reproducible.

The result

Rubric v17 passed every preregistered gate on the untouched holdout: finding precision 96.43% (95% lower bound 94.16%), rubric-agreement recall 95.64% (95% lower bound 92.23%), and a worst-case false-positive rate of 0.66% (95% upper bound 1.10%). This is not detection recall — four different things on this site are called recall, compared side by side. The finding precision is not detection precision — two different things on this site are called precision, compared side by side.

The defect-class census publishes the coverage-breadth boundary: which defect classes any shipped rule targets. It is not a recall claim.

And — this matters as much as the numbers — we published the boundary right next to them. This is finding-level calibration of one rule set on one preregistered population. It is not a universal security guarantee, not proof your code is safe, not a claim about every framework. We also disclosed the un-flattering statistic: the intervals are not adjusted for the number of rubric revisions tried during development, and we say so in the record rather than hoping nobody asks.

Why “publish either way” is the whole point

A scanner that can't fail its own test isn't measuring anything. Our harness can return NO-GO, and when a gate isn't met, that's the published result — not a quiet retry until the number looks good. If you build developer tools, we'd gently push this: don't just show a score. Show the evidence, show the boundary, and show what happened when you pointed the tool at itself. It's more work and it's less flattering, and it's the only version of trust that survives contact with a skeptical engineer.

The same method applies to other people's instruments. On 20 September 2026 we preregistered five predictions about a commercial judge model's ability to tell false agent completion reports from true ones, ran it against known outcomes, and published the result with three of the five contradicted: the Jev 1.13 judge calibration. A second specimen, on an open-weights model, is at the Laya specimen; its run departed from its preregistration in one recorded respect, stated at the top of that page. The method also applies to scanners: a known-answer test of Cisco's open-source skill-scanner, with every inserted defect known in advance, is at the skill-scanner specimen.