Extend context length and offload layers to the GPU
llama.cpp & gguf
// what it does
-c/--ctx-size sets the context window in tokens; Gemma 4's 12B/26B/31B reach 256K but you must ask for it, since the default is small. -ngl/--n-gpu-layers moves that many transformer layers to the GPU. A larger context allocates a bigger KV cache, so raise -c only as far as memory allows.
// shell
$ llama-cli -hf ggml-org/gemma-4-12B-it-GGUF -c 32768 -ngl 99 -p "Summarize this file."// gotcha
Requesting -c 262144 on a small GPU allocates a KV cache that can dwarf the weights and trigger an out-of-memory abort before the first token. Start around -c 8192 and increase as needed.