Grab Google's QAT checkpoints to save memory
fine-tuning & quantization
// what it does
Beyond the bf16 repos, Google ships Quantization-Aware Trained int4 checkpoints — google/gemma-4-<size>-it-qat-q4_0-gguf for llama.cpp and Ollama, plus matching loadable variants — that hold most bf16 quality at roughly one-third the memory. They are the recommended way to run the 31B on a 24 GB card. Download them with huggingface-cli.
// shell
$ huggingface-cli download google/gemma-4-31B-it-qat-q4_0-gguf// gotcha
QAT weights are already int4: do not re-quantize them. Stacking bitsandbytes 4-bit on top corrupts the values and tanks quality — load them as-is.