Use Google's QAT int4 weights for best quality per byte
llama.cpp & gguf
// what it does
Google publishes Quantization-Aware Trained checkpoints (google/gemma-4-<size>-it-qat-q4_0-gguf) that were fine-tuned to tolerate 4-bit weights, recovering most of the bf16 quality that naive post-training quantization loses. They cut memory roughly 3x versus bf16 while staying close on benchmarks, and -hf pulls them directly.
// shell
$ llama-cli -hf google/gemma-4-E4B-it-qat-q4_0-gguf -p "Explain QAT in one sentence."// gotcha
QAT GGUFs are int4 only — you cannot up-cast them back to bf16 quality. For 8-bit or full precision, download the safetensors repo (google/gemma-4-E4B-it) and quantize it yourself.