> llama.cpp & gguf
llama.cpp runs Gemma 4 from GGUF files on CPU, GPU, or a hybrid of both, and its -hf flag downloads a model straight from Hugging Face in one command. llama-cli gives you a shell completion or chat; llama-server exposes a browser UI and an OpenAI-compatible API on port 8080. Google's QAT int4 GGUFs give the best quality at 4-bit.
// llama.cpp & gguf
5 commands$ llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -p "Write a haiku about the sea."$ llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -sys "You are Hong Gildong." -p "Who are you?"$ llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -c 8192 -ngl 99$ llama-cli -hf google/gemma-4-E4B-it-qat-q4_0-gguf -p "Explain QAT in one sentence."$ llama-cli -hf ggml-org/gemma-4-12B-it-GGUF -c 32768 -ngl 99 -p "Summarize this file."// faq
How do I run Gemma 4 in llama.cpp without converting anything?
Use the -hf flag with a GGUF repo, e.g. llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -p "Hello". It downloads the GGUF (caching under ~/.cache/llama.cpp) and runs it. Pointing -hf at the safetensors repo google/gemma-4-E4B-it fails because it contains no .gguf files.
What are the Gemma 4 QAT GGUFs and why use them?
Google publishes Quantization-Aware Trained int4 checkpoints (google/gemma-4-<size>-it-qat-q4_0-gguf) that were trained to tolerate 4-bit weights, so they keep most of the bf16 quality that naive post-training quantization loses, at about a third of the memory. They are the recommended way to fit the 31B on a 24 GB GPU.
How do I run Gemma 4 on the GPU with llama.cpp?
Pass -ngl (n-gpu-layers) to offload transformer layers to the GPU; -ngl 99 pushes all of them and silently caps at the real layer count. Without -ngl the model runs entirely on CPU, which is slow for the 12B and larger. Set context with -c, but keep it modest since a large KV cache can exhaust GPU memory.
Does llama-server work with OpenAI SDKs?
Yes. llama-server serves an OpenAI-compatible API at http://localhost:8080/v1/chat/completions plus a chat UI at http://localhost:8080. Point any OpenAI SDK at that base URL with a placeholder key; the model name is whatever you loaded.