Shard a large Gemma 4 across multiple GPUs
serve an api
// what it does
--tensor-parallel-size splits the model's weights across N GPUs so a 31B in bf16 (~58 GB) fits on two 40 GB cards. --gpu-memory-utilization (0-1) caps how much of each GPU vLLM claims, leaving room for other processes. Combine with --max-model-len to bound the KV cache.
// shell
$ vllm serve google/gemma-4-31B-it --tensor-parallel-size 2 --gpu-memory-utilization 0.90 --max-model-len 16384// gotcha
--tensor-parallel-size must divide the number of attention heads evenly, and all GPUs should be identical — an indivisible split or mismatched cards fail at load time.