Fine-tune the 26B MoE without breaking expert routing
fine-tuning & quantization
// what it does
The 26B A4B routes each token to 8 of 128 experts plus one shared expert. Fine-tune it with LoRA in bf16 and leave the router weights untouched; a full fine-tune tends to collapse the routing so a few experts dominate. Start at r=16, lora_alpha=16 and lengthen context gradually once the loss settles.
// gotcha
Targeting the expert-router matrices with LoRA (or doing a full fine-tune) can unbalance the gating and cripple the MoE — restrict adapters to the attention and expert MLP projections, not the router.