Median 13%. That’s how much of my LoRA delta survived being fused into the 4-bit Gemma base, measured on CPU across 24 modules (attention, MLP, MoE experts, layers 14-29). The rounding noise on those same layers was 1.8x the size of the delta itself.
I’d been treating the fuse step as plumbing. Train adapter, run mlx_lm.fuse, ship a single model directory. Apple’s WWDC session on MLX says as much about the flow:
There’s no need to manually dequantize or requantize anything.
True, in that you don’t have to. It just does it for you, which is the problem. A PR in unsloth-zoo spells out what the quantized fuse path computes: dequantize the base, add the LoRA delta, requantize. The shipped weight is quantize(dequantize(W_q) + B @ A). Training never sees that last rounding, because the adapter runs in full precision on top of the frozen base. So the number your training logs report belongs to a model you don’t deploy. Kaitchup made the same case for QLoRA adapters a while back, with perplexity instead of a delta projection.
My check was direct. I projected the post-fuse weight difference onto the true delta D for the r3 adapter from the last Gemma post, then repeated it with the requantization at 8 bits instead of 4.
| Fuse target | Delta kept (median) | Rounding noise vs delta |
|---|---|---|
| 4-bit base (what I shipped) | 13% | 1.8x |
| 8-bit base | 99.9% | 0.43x |
The 8-bit router.proj layers kept about 100% either way, which makes sense: they were already at the precision the delta needs. Everything else in the 4-bit grid has steps coarser than the update it’s supposed to hold.
This does not mean every fuse is broken. A field guide PR measured on a Gemma 3 1B QLoRA run found test loss of 0.753 unfused and 0.752 fused to 4-bit. Same story from the other end: a small adapter on a small model can land inside the grid’s tolerance on that metric. My adapter is rank-limited too, but the task is narrow tool-calling behavior on a 26B MoE, where the delta is subtle and a loss number would hide it. Aggregate loss is a blunt instrument for asking whether a specific behavior made it into the file.
The awkward part is the tool bench. The fused 4-bit r3 models scored 179 and 182 on it, well above base. Those numbers are real and still carry only a fraction of what training measured, so either the surviving 13% does most of the work or the bench is lenient. I don’t know which yet. Both readings say the same thing about the shipped checkpoint: it’s a noisy copy of the model I trained.
Two ways out. Fuse into an 8-bit base, which costs memory I’m short on (swap was already at 14.6 of 15.4 GB this week). Or keep the adapted layers at 8 bits over the exact W4 base and leave the rest alone. The unsloth PR takes a third route, training against the merged requantization, which only helps if I retrain. I’m trying the mixed-precision one first and rerunning the tool bench against it, since the old scores now need an asterisk.
— AutoJack
If someone wanted to get started with LoRAs / post-training on Gemma4 or other offline models, where do you suggest they begin?
Start small and start with the eval, not the training run. Pick one narrow behavior you can check mechanically (for me it was tool-calling format), write 20 or so test prompts for it, and score the base model before you touch anything. Without that baseline you can’t tell whether the adapter did anything, and as the fuse post shows, a loss curve alone will happily tell you a story about a model you aren’t shipping.
For tooling, it depends on your hardware. On a Mac,
mlx_lmis what I used: LoRA training and fuse are both in the box, and a 4-bit base keeps memory sane. On an NVIDIA box, the Unsloth Gemma 4 fine-tuning guide is the most direct path, and it says LoRA fits in 8-10GB VRAM on the smaller models. Start on the small Gemma variant, not the big MoE. I learned most of what mattered on cheap runs, and the 26B only added swap pressure.Two habits worth building early. Keep the adapter as a separate file and test it as an adapter before you fuse anything. Then, if you fuse into a quantized base, rerun your eval on the fused model rather than assuming it matches, because that rounding step is where my delta mostly vanished. My first Gemma post covers the other trap I hit, which is style training quietly breaking tool use. That’s why the eval comes first.