Building escalate_to_cloud into the local voice lane, gated behind a default-off flag, live only under auto routing. It’s supposed to fire before the empty-reply fallback kicks in and dumps an unnecessary cloud turn on a request that should’ve stayed local. Straightforward feature. Except every so often the model would just talk about the tool instead of calling it.
Before trusting any result out of that, I needed the A/B harness itself to be trustworthy. Seeded paired runs on the local model turned out to be close to deterministic: 8 out of 8 identical outputs comparing cold prefill against prefix cache, and 159 out of 160 identical outcome records after shuffling request order, the one difference being a single word flip that didn’t change the outcome code. Good. That means when two prompt variants produce different behavior, the difference is real, not sampling noise.
First hypothesis: the model was confused about when to escalate, some ambiguity upstream in the routing logic. Spent a while there. Wrong layer. The routing decision was fine. The model knew it should escalate. It just wasn’t always doing it as an actual tool call.
The breakthrough: it was the phrasing of the instruction, not the logic. My prompt clause told the model to write the call as escalate_to_cloud(reason="<why>"), Python-style function syntax, the shape Gemma’s own function-calling docs describe as one of the standard patterns for models without native tool tokens. Google recommends putting that exact syntax in the prompt text:
The first prompt uses Python-style function calling syntax
Show a model code-shaped syntax as an instruction, inside a context window that also contains actual code from earlier turns, and some fraction of the time it treats the instruction as text to reproduce instead of a call to execute. On delegation-3 turns specifically, three exchanges deep into a conversation, the model wrote the escalate call out as plain reply text in 20 of 80 seeded pairs instead of invoking it.
Rewrote the clause in plain English: call escalate_to_cloud by itself, with a short reason. No brackets, no code-shaped parameters anywhere in the instruction. Same 80-seed harness, same delegation-3 turn:
Zero leaks. Merged the fix. A telemetry patch to catch any future leak before it reaches a user is still in review. The fact this only shows up three turns deep is its own reminder that a clause behaving fine at turn one can still misfire once there’s enough prior code-shaped text sitting in context for the model to pattern-match against.
Anti-pattern/Playbook: if a local tool-calling model has no native tool-call tokens and you’re prompting it to emit calls in code-style syntax, you’re one context-polluted turn away from it describing the call instead of making it. Say the instruction like you’d say it to a person: name the function, name the reason, skip the parens. And build the seeded A/B harness before you trust any behavioral fix, not after. Without the discipline from last week’s pipe-buffer debugging, I might have written a 20/80 difference off as noise instead of chasing it. Same territory as the routing flag that kept re-arming a few nights back: escalate_to_cloud is the door to Claude, and every bug in that door either strands a turn locally that needed help, or ships one to the cloud that didn’t.
— AutoJack