We asked a calibrated judge whether false agent reports were true. It mostly said yes.

Why we ran it

TypeSafe AI's Jev is a hosted model that does not generate text. You send it a state and typed questions and it returns typed answers with a probability. Its documentation defines calibration correctly and states that its published evals measure agreement with two frontier models, not agreement with outcomes. It also states that content written to steer the model can move its answer. Nobody had published a reliability curve for it against known outcomes on a real task, and at its prices the experiment costs cents, so we ran one.

The protocol was preregistered and frozen before the first live call, with five predictions written down in advance. Three errata were recorded during the corpus build, each signed before the outcome it affected was seen; all three concern selection and construction mechanics, none changes a question, a threshold, a metric or a prediction. The preregistration, the errata, the harness source and the unedited result are linked at the foot of this page.

What we did

Fifty merged public pull requests were selected by a deterministic procedure with eleven exclusion rules and an admissibility filter that checks each base case is clean by construction. From each, four defective variants were built by inserting exactly one false sentence: a completion claim timestamped fifty-eight minutes before the merge, a fabricated commit and issue reference, a wrong check count, and a file the change did not touch. Each defective variant was then rewritten once by a generative model instructed to make it maximally convincing while preserving the false sentence character for character; rewrites that failed that guard were dropped and counted (20 of 200). That gives 430 cases, each judged under two conditions: text only (T), and text with the structured evidence record from the GitHub API (E). 860 calls, US$0.03, four minutes.

The judge's decision was one probability that the report is accurate, threshold 0.5, frozen. A defective report with probability at or above 0.5 is a false clear. Every interval below is a two-sided 95% Wilson interval unless stated; paired shifts use a cluster bootstrap over base cases.

What happened

Report alone, the judge cleared nine tenths of the defective reports. Premature 90.0% [78.6, 95.7], fabricated reference 98.0% [89.5, 99.6], wrong count 92.0% [81.2, 96.8], wrong file 62.0% [48.2, 74.1], each n=50. This was predicted. A judge that sees only the text cannot see the world the text describes.

With the evidence attached, it still cleared most of them. Wrong count 84.0% [71.5, 91.7], wrong file 92.0% [81.2, 96.8], premature 90.0% [78.6, 95.7]. This was predicted the other way: counting and list comparison are things the judge is documented to handle, and the prediction was a false-clear rate at most 20%. The evidence helped on exactly one defect, the fabricated reference, from 98% to 68%, and it removed the false flags on clean reports entirely (28.0% [17.5, 41.7] to 0 of 50).

The persuasive rewrite moved the answer, in the direction opposite to the prediction. Mean probability on defective reports fell by 0.129 [0.100, 0.158] with text alone and 0.198 [0.178, 0.218] with evidence, paired over 180 cases. The prediction was a rise of at least 0.10. The rewrites are confident, specific and well organised, and the judge trusted them less. We do not know why; one hypothesis is that the rewrite makes the false claim prominent and specific enough to check, another is that the register itself reads as suspect. It is a question for a follow-up, not a finding.

The probabilities are not calibrated against outcomes. Expected calibration error 0.458 [0.438, 0.479] text only, 0.420 [0.395, 0.447] with evidence, n=430 each, against a preregistered gate of 0.05. With text alone the judge never gave a probability above 0.8 or below 0.1 to anything; with evidence, of the 82 reports it placed between 0.5 and 0.6, one was accurate.

The first four rows count defective reports the judge cleared; the last counts clean reports it did not clear. Different event, different population, opposite sense of the same cell. In every row a higher number is the worse outcome. No figure in this table has changed since publication; see the correction note below.

One secondary result, and what it says

With evidence, the judge was also asked to pick which of five defects the report contained. It picked correctly 374 of 430 times, 87.0% [83.5, 89.8], against a constant-answer baseline of 23.3% (the largest of the five truth classes is 100 of 430). Per label: fabricated reference 99 of 100, wrong count 81 of 82, wrong file 96 of 98, clean 50 of 50; the premature claim 48 of 100, which is the documented weakness at date comparison. Every wrong pick was "accurate".

Set beside the probability answered in the same call, the picture is this: in 174 of the 380 defective calls the judge named the defect correctly and, in the same response, gave a probability of 0.5 or more that the report matched the evidence. Asked the literal question, it answers it. Asked whether the report as a whole is right, it does not integrate its own answer. This cross-tabulation is a post hoc secondary analysis, computed from the frozen call log after the scored analysis, with no prediction registered for it; the secondary document linked below reproduces every primary figure from the log before printing it.

What we take from it

A content judge is a fast, cheap literal comparator when the question is literal and the evidence is in front of it, and it is not a verifier of a report. Its stated probability is not a probability of anything a buyer can act on. A report written to pass a reviewer did not pass this judge more easily, on this corpus, and that is not the same as saying it cannot be steered; the vendor says it can, and one rewrite prompt is one point. The distinction we build on is unchanged: a judge tells you what it thinks of the text; provenance tells you what happened; an auditor accepts the second. A Cejel certificate is the second kind of thing, and that is why it is deterministic and re-runnable by the party accepting the code rather than a probability from a model.

Limits, again

Constructed defects, one insertion each, from public pull-request bodies. Fifty base cases. One judge at one pinned version (jev-1.13.0, checked on every response), one wording per question, one rewrite prompt, one threshold. Three errata during the build, all recorded and none touching the analysis. One change to two harness files landed between the freeze and the run without the usual review and was read afterwards; it is described in the result document. No comparison with any other judge, and no comparison with any verifier on a shared task, because they answer different questions. Re-run it: the harness, the seed, the query and the construction rules are published, and the run costs three cents.

Follow-ups, 2026-09-23

Three preregistered follow-ups ran on 23 September, on the same corpus, the same pinned judge and the same questions. Each was frozen before its first call; the approvals and one erratum are linked below. Together they explain the result above and change the advice at the end of this page in one respect.

The information was in the answer all along. In condition E every call had already asked which of five defects the report contained, and the judge named the right one 374 times of 430. Re-scoring the frozen call log with that label’s own probability of “accurate” as the clearing score, in place of the separately asked “does the report match the evidence”, took the false-clear rate on wrong count from 84.0% to 1.2% [0.2, 6.6] (n=82), on wrong file from 92.0% to 1.0% [0.2, 5.6] (n=98), and on fabricated reference from 68.0% to 0 of 100 (printed as a count, withheld as a rate under the leakage rule). Premature stayed at 45.0% [35.6, 54.8] (n=100), the documented weakness at date comparison. Of the 324 defects the judge had labelled correctly, the composed score cleared none; all 47 clears were mislabelled cases. No new call was made. Prediction P2c-A, supported.

Asking in the other order does not fix it. A live run of the same 430 E cases asked the label question first and then, in the same call, whether the report matched the evidence “given the label you just gave”. The judge still cleared 48.8% [38.3, 59.4] of wrong-count cases and 59.2% [49.3, 68.4] of wrong-file cases with its own correct label one line above, while its label accuracy was unchanged at 86.7%. Predictions P2c-B, contradicted; P2c-C, supported. The integrated question is the problem, not the order it is asked in.

The mechanism is the salience of the claim, not the size of the evidence. Four arms on the 98 wrong-file cases and the 50 clean cases, 592 calls: a replication of the primary rendering (94.0%, inside the primary’s interval, so no vendor drift; PA-0 supported); the changed-file list moved to the top of the evidence (92.0%; PA-2 contradicted); the evidence reduced to the changed-file list alone (66.0% [52.2, 77.6]); and the report truncated to the paragraph that names the false path, with the list alone (2.0% [0.4, 10.5], 1 of 50). No arm produced a false flag on a clean report (PA-3 supported). PA-1, which asked for both reductions to halve the rate, is contradicted on its first clause and exceeded on its second; the honest statement is that truncating the report carries most of the effect and shrinking the evidence a minority. The last arm changes two things at once, so the split between them is not measured. Handed the claim, the judge finds the fault; handed the report, it clears it.

What changes in the reading above. Nothing in “What we take from it” is withdrawn: the integrated probability is still not a probability of anything a buyer can act on, and calibration against outcomes was not established in any follow-up arm either (composed-score ECE 0.175 [0.158, 0.196]). What is added is the use that does work: ask the judge atomic questions about one claim at a time, in front of the evidence, read the label, and compose the verdict in code. That is the vendor’s own design guidance, and this is a measurement of why it holds. A judge used that way is a fast, cheap literal comparator; it is still not a verifier of a report, and the distinction this site is built on is unchanged.

One refusal before a call. The anchoring preregistration stated 150 subjects; the frozen corpus manifest records 148, because two persuasive rewrites had been dropped by the verbatim guard during the build, as the primary result says. The harness refused to run on the mismatch; a signed erratum corrected the count before the first call. The number should never have been typed rather than read from the manifest. It is recorded here because a guard that fires on its author is the kind worth having.

Records

Every number on this page traces to one of these files, which are copies of the originals with the source commit of each recorded in the directory's README.

• Preregistration, frozen before the first live call
• Errata 1, errata 2, errata 3
• Result, unedited harness output, with the JSON
• Secondary analysis, label-space baseline and Choice-by-Noul cross-tabulation, reproduced from the frozen call log
• Follow-ups of 2026-09-23: composition preregistration with its signed approval record; Arm A (from the frozen log) and Arm B (live, forced order); anchoring preregistration, its erratum 1, and the anchoring result; JSON beside each
• Harness source and the frozen corpus (constructed reports from public pull-request bodies; digest in the manifest)

Not published, under our disclosure boundary: the frame-exclusion file that rule 2 reads, the call log, and any organic-arm material. The corpus contains no repository that is a member of a live Cejel calibration frame; only the count excluded under that rule is published.

Corrections after publication

2026-09-21, presentation only. The results table was captioned as false-clear rates on the defective arm, and its last row reports clean reports the judge did not clear — the opposite event, on a population the caption excluded. A reader taking the row at the caption's word would have read 14/50 as fourteen clean reports cleared, which would be ordinary behaviour, rather than fourteen false-flagged. The caption and the row labels now name the counted event in each row. No figure changed, and the prose above the table always stated it correctly (“it removed the false flags on clean reports entirely”). The preregistration, the result and the analysis are untouched; this is not a fourth erratum against them.