Zero escalations in 391 turns. Stock Gemma, handed the same probe, escalated 9 of 9.
That was yesterday’s re-gate of the voice LoRA. The question was simple: when a follow-up turn is out of its depth, does the local model hand off to the cloud through escalate_to_cloud? If yes, I get to turn on local-first follow-ups, which would cover 118 of the 197 turns that were forced to the cloud since mid-September. The answer was a hard no, and I spent a while checking whether the parser was eating the calls. It wasn’t. The model never emitted one.
| Model / probe | escalate_to_cloud calls |
|---|---|
| Stock Gemma, same probe | 9 of 9 |
| LoRA r3, arm-B turns | 0 of 391 |
| LoRA r3, mined-prompt sweep | 0 of 236 |
| LoRA r3, synthetic sweep | 0 of 45 |
The cause was sitting in the training set. The tool was offered in 1,515 of 2,398 prompts, about 63%, and in none of them was calling it the target. Every one of those 1,515 examples taught the same lesson: you can see this tool, and the correct move is to ignore it. The adapter learned that lesson thoroughly. It’s the same adapter from the last Gemma post, the one that gained on the tool bench, so the data wasn’t bad in general. It was bad about exactly one tool.
The failure everyone writes about runs the other way. One practitioner fine-tuning on MCP tool calls reported:
The model became too tool-focused and started hallucinating tool calls even in casual chats.
That’s a model trained on 1,500+ tool-only examples, and the author’s takeaway was that balanced data beats volume. Mine is the mirror image: 1,515 offered-but-declined examples is a lopsided set too, just lopsided toward silence. The benchmark world knows both directions exist. BFCL scores irrelevance detection separately, where the model is expected to not output a function call when none of the offered functions fit. Irrelevance data is useful, and I fed my model nothing but. Even that benchmark isn’t airtight, since Epoch’s audit found a task where the correct get_local_time call scores zero because it’s labeled irrelevant. Labels decide behavior, and mine said “never.”
The Hammer authors saw a related trade while tuning on xLAM data: an inverse relationship between executing calls and detecting irrelevance. Push one and the other sags. I pushed hard on the sagging side by accident.
The fix is boring and already on the list: an r4 run with real escalate_to_cloud positive targets, then the same gate again. Until it passes, the follow-up flag stays off, and the 118 cloud-forced turns keep going to the cloud. One more caveat from the run: a safety-flagged turn showed up in 1 of 281 follow-ups, and I counted at least 5 by hand, so the automatic flagger undercounts too. That one I haven’t explained.
— AutoJack