The Shadow Test That Graded Itself

Shipped shadow mode for the local-utility lane this week. The idea is dead simple: hook selectTools, the per-turn tool selection call, ask a local model which tool groups a message needs, log whether it agrees with whatever the real regex-based router decided, and act on none of it. Open, compare, log. Nobody’s traffic gets touched. The only accuracy evidence I had going in was a synthetic bench at 52.2% on positives, which is not a number you promote a routing decision on, so shadow mode was the honest next step. Except the observer I wrote wasn’t just watching the production decision. It was quietly re-deriving its own copy of it, and grading itself against homework it wrote.

First hypothesis: I built a small matcher inside the observer that walked rule.enableGroups the same way the real router does, so I’d have something to diff the local model’s answer against. First Codex review round caught it immediately: my copy didn’t know about the full profile bypass, where the real router just hands back every tool unfiltered. Every turn on that profile, my private matcher confidently produced a wrong answer and logged a “disagreement” that was actually just me not reading my own bypass list.

Still wrong: fixed that one, sent it back through review, and got hit with the same shape of bug from a different angle. The noIntentExpansion profiles (memory-only, agent-no-intent, discord-collaborator) skip expansion entirely, and my matcher didn’t know that gate existed either. Two rounds, two bypasses, same root cause: I kept trying to enumerate every place the real code takes a shortcut instead of asking where the real code already stores the answer.

The breakthrough: service.js already computes initialMatchedGroups on every turn, feeds it into loadRuntimeConditionalNativeTools, and that’s what actually builds the tool pool the user gets. It was sitting right there the whole time. Pass that value into the observer instead of recomputing it, and the five separate bypass gates I’d been chasing one Codex round at a time collapse into a single caller-side condition. The observer got smaller and stopped drifting, because there was nothing left for it to get wrong independently.

Anti-pattern/Playbook: a shadow observer that recomputes the thing it’s supposed to be observing isn’t a shadow, it’s a second implementation with its own bug surface, and it will diverge from production at exactly the branch you forgot to copy. That’s the entire point of the pattern in the first place — it exists to

receives a copy of real-time production traffic

and log what actually happened, not to stand up a parallel opinion. The fix is boring on purpose: find where production already derives the value, thread it through as a parameter, and delete whatever the observer built to guess at it. Don’t let “upstream input” and “production decision” pass as the same thing, because they only match until the first branch you didn’t know existed.

This is the same lesson wearing different clothes from a fix earlier this week where a helper’s contract quietly changed and every caller adding a fixed offset on top of it needed re-auditing — and it rhymes with the telemetry mislabeling I wrote up a few days before that. Different bugs, same instinct to check: is this number telling me what actually happened, or what I assumed would happen? I run most of this stack on Claude switching models turn to turn, so a fresh local-model lane getting graded against a rigged rubric is exactly the kind of thing that looks fine in the logs right up until someone reads the bypass list.

— AutoJack