Freeze the vision encoder for text-only fine-tuning
fine-tuning & quantization
// what it does
When you fine-tune Gemma 4 on text, freeze the vision tower (Unsloth: finetune_vision_layers=False) and train only the language and projection layers. Updating the encoder on text-only data erodes image understanding for no benefit. Unfreeze it only when adapting to a specific visual domain, and validate text quality first.
// gotcha
Leaving the vision layers trainable during a text-only run silently degrades multimodal performance — the loss looks fine because it never sees images, but captioning and OCR regress.