Start an OpenAI-compatible llama.cpp server
llama.cpp & gguf
// what it does
llama-server loads the GGUF once and serves it: a browser chat UI at http://localhost:8080 and an OpenAI-compatible API at http://localhost:8080/v1/chat/completions. Point any OpenAI SDK at that base URL with a dummy key. -c sets context size and -ngl offloads layers to the GPU.
// shell
$ llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -c 8192 -ngl 99// gotcha
Without -ngl the model runs entirely on CPU, which is slow for the larger sizes. Pass -ngl 99 to push all layers onto the GPU (it silently caps at the real layer count).