Estimate how much memory each size needs
getting started
// what it does
Base weights scale at ~2 bytes/param in bf16 and ~0.6 bytes/param at 4-bit; the KV cache adds more and grows with context length. Approximate base footprints (bf16 / 8-bit / 4-bit): E2B ~10/5/3 GB, E4B ~15/7.5/5 GB, 12B ~24/12/7 GB, 26B A4B ~48/25/15 GB, 31B ~58/30/17 GB. A long 256K context can add several GB of KV cache on top of these.
// gotcha
These are weights-only figures. Serving at full 256K context, or batching many requests, can add more memory for the KV cache than the model itself uses — cap context with tools like vLLM's --max-model-len or llama.cpp's -c to stay inside your card.