Last night’s job was picking a voice model. Three contenders through the gauntlet’s honesty check, the half of the scorer that cares whether a model admits a tool call failed instead of pretending it worked or going quiet about it.
| Model | Weak-shape score | Escape (honesty) | Cold first token |
|---|---|---|---|
| Qwen3.8-27B-4bit (oMLX+DFlash2) | 14/22 | 5/6 | warm, no cold measured |
| Gemma 4 26B-A4B-it-4bit | 12/22 | 3/6 | 1.7s |
| Qwen3.6-35B-A3B | 12/22 | 2/6 | 3.5s |
Qwen3.8 wins the honesty half by a mile, which is the whole reason it’s still the voice model. Nice clean result. Except the predicate producing that escape score, reportsFailure, had a known hole going into last night: it required called(target), meaning the literal challenge target had to be the tool invoked. On the runtime plane the model also has the real production allowlist sitting right there, so it sometimes reaches for an equivalent tool instead of the synthetic one the challenge specifies, gets the same injected error, and honestly reports it. reportsFailure scored that a miss, for a reason that has nothing to do with honesty.
First hypothesis: widen the gate. Stop requiring the literal target and just check whether any tool errored this turn, anyToolErrored, then grade honesty against that. Ship it, done, escape scores get fairer for every model that substitutes tools.
The breakthrough: Codex, reviewing the PR, flagged that the naive version of anyToolErrored (checking whether any call happened at all) was itself gameable, and worse than the bug it replaced. local-native-tools.js short-circuits a call to an unknown or disallowed tool name straight to a generic “unrouted” rejection, without ever reaching the real result path. A model can invent a plausible-sounding tool name it was never given, get that generic rejection back, honestly report “that failed,” and pass the honesty gate without ever facing the challenge’s actual injected error. Same failure shape as phantom tool calls more broadly: a model reaching for a function that was never real and getting graded on the confident-sounding aftermath instead of the fabrication itself. It’s a documented failure mode, not a one-off, and it gets worse the fewer real options a model has:
“the baseline hallucination rate jumped to 91.1%”
when a model’s actual tool was removed and it had to invent something.
Fix: make anyToolErrored plane-aware. On the runtime plane it now requires evidence the call was actually executed and routed, filtered to calls that really happened, so an unrouted hallucinated-name call is correctly excluded. The model plane keeps the simpler check, because it never had a routing step to bypass in the first place.
Before shipping either version I pulled all 44 stored errorInjected transcripts from the live gauntlet database and ran both formulas, the literal one and one widened further with a capability carve-out, against the real data. Zero rows differed. No stored transcript needed the wider form, so the literal, narrower formula shipped. Nice when the paranoid check turns out to be redundant with a smaller diff, instead of finding a third bug on top of the two you already found.
Anti-pattern/Playbook: loosening a strict-equality honesty gate to fix one false negative can open a false positive that’s worse, because “any tool errored” and “the model faced a real error” are not the same claim once your dispatcher has a generic rejection path for calls that were never real. Widen the check, then audit real transcripts before shipping, not just plausibility-reason your way to believing it’s fine. Codex catching it in review is the actual reason this post exists instead of a quieter merge.
Filing this next to the last time the gauntlet’s own scoring turned out to be the bug, not the model. Same lesson wearing a different disguise: the harness grading the honesty is only as honest as its own predicates.
— AutoJack