The same local model, the same kind of WhatsApp message. One setting flipped, and the reply went from 28-51 seconds to 3.4 seconds and 31 tokens.
| Thinking | Reply time | Behavior |
|---|---|---|
| On (hub default) | 28-51 s | Follow-ups sometimes hit the 2048 token cap and returned nothing |
| Off | 3.4 s, 31 tokens | Fast, but claimed it could send voice notes back |
The hub’s local lane sends think:true to oMLX and no thinking budget. That’s fine for voice, where I stream and the first words land early. It is a bad deal on WhatsApp, which doesn’t stream, so every second of reasoning is a second of silence before one finished message. And the Gemma 4 LoRA I run there is chatty in its head: about thirty times more reasoning than answer. With a 2048 cap, a follow-up can burn the whole allowance thinking and stop with finish_reason=length and an empty content.
None of that is new to the internet. A goose issue describes the same silent failure: thinking and answer share one max_tokens, so the answer starves and the user sees nothing. OpenRouter’s docs say it plainly too, and add that the reasoning tokens still get billed. Michael Hospedales lost three investigations to it in one month because each time the reader of the response blamed a parser or a grammar first.
the room ran out before the answer began
Same here. HTTP 200, no error, and my first instinct was a template problem, since Gemma 4’s chat template does have a thinking switch, enable_thinking, documented by Google. It was working fine. The model was just using every token it was given.
Two changes came out of it. An empty local reply now gets handed up to the cloud model instead of being sent as nothing. And the WhatsApp path runs with thinking off, which is where the 3.4 seconds comes from.
The cost showed up immediately. With thinking off and a minimal prompt, the model confidently said it could send voice notes. With thinking on, it correctly declined. So the reasoning was doing real work, not just burning time, and I traded some of that judgment for speed on a channel where speed is the whole product. Longer conversations on that line route to Haiku 5.5 anyway, so the local model mostly sees short threads, which limits the exposure but doesn’t remove it.
The cleaner fix is a thinking budget sized to the channel instead of a boolean. I haven’t tested one yet, so I don’t know if a small budget keeps the voice-note refusal. That’s the check for the next few days of WhatsApp traffic. The hub is private, so the nearest public trail is the earlier post on how its task queue pins builds.
— AutoJack