Llama 3 8B
8B paramsFP16 weights are about 16 GB; with overhead this lands near 18 GB. At 8K context the KV cache (32 layers, 8 KV heads via GQA) adds only ~1 GB.
Fits a single 24 GB RTX 4090 comfortably in FP16, or an 8 GB card in 4-bit.
Precision Tool
VRAM Calculator
Estimate memory footprint by model size, precision, sequence length, and batch configuration before you commit to a deployment or local setup.
Weights + KV cache + overheadUseful for local inference and server planningBest used before renting GPUs or scaling prompts
Precision Tool
How GPU memory for an LLM is actually calculated
The memory a large language model needs on a GPU is not a single number — it is the sum of three components, and getting the balance right is what separates a deployment that works from one that crashes with an out-of-memory error on the first long prompt. This calculator breaks the estimate into model weights, the key-value (KV) cache, and runtime overhead, using the model's real architecture from its Hugging Face config rather than a generic multiplier.
1. Model weights
Parameter count × bytes per parameter. FP16/BF16 uses 2 bytes, INT8 uses 1 byte, and 4-bit formats (GPTQ, AWQ, GGUF Q4) use roughly 0.5 bytes. This term is fixed for a given precision and does not change with prompt length.
2. KV cache
2 × layers × kv_heads × head_dim × context × batch × 2 bytes. It grows linearly with context length and batch size, stays in FP16 even for quantized models, and becomes the dominant cost at long context.
3. Runtime overhead
CUDA context, framework buffers, activation working set, and memory fragmentation. Budget roughly 1–2 GB plus 10–15% of the total as a safety margin before you commit to a card.
Worked examples
These examples show how the same model can fit very different hardware depending on precision and context length. Run your own model through the calculator above to get exact numbers.
Mixtral 8x7B
46.7B (MoE)A mixture-of-experts model holds all experts in memory even though only two are active per token. FP16 weights are roughly 90 GB, so a single 24 GB GPU is out of the question.
Needs 2× A100 80GB in FP16, or drops to ~24 GB in 4-bit for a single high-end card.
Llama 3 70B
70B paramsFP16 weights are about 140 GB — firmly multi-GPU territory. In 4-bit the weights fall to roughly 40 GB, which fits a single 80 GB card with room for the KV cache.
Single A100/H100 80GB in 4-bit, or 2× 48 GB GPUs with tensor parallelism.
Frequently asked questions
How much VRAM do I need to run an LLM?
As a fast rule of thumb, weight memory equals the parameter count multiplied by the bytes per parameter: 2 bytes for FP16/BF16, 1 byte for INT8, and roughly 0.5 bytes for INT4. A 7B model therefore needs about 14 GB in FP16, 7 GB in INT8, and 3.5 GB in INT4 for the weights alone. On top of that you must budget for the KV cache, which grows with context length and batch size, plus 1–2 GB of framework and CUDA overhead. The calculator above computes all three terms for a specific model.
Why is the KV cache so important for long context?
The KV cache stores the key and value tensors for every token already in the context window. Its size is 2 × layers × kv_heads × head_dim × context_length × batch_size × 2 bytes (it stays in FP16 even when the weights are quantized). Because it scales linearly with context length, a model that fits comfortably at 4K tokens can run out of memory at 128K tokens. This is why two models with identical parameter counts can have very different memory profiles at long context.
Does quantizing to 4-bit cut my total VRAM by 4x?
No. Quantization shrinks the weights (from 2 bytes to about 0.5 bytes per parameter), but the KV cache and activations usually stay in FP16. At short context the weights dominate, so 4-bit gets close to a 4x reduction. At long context or high batch size the KV cache dominates, and 4-bit weights barely change the total. Always size the KV cache separately before assuming a quantization saving.
What is the difference between grouped-query attention (GQA) and multi-head attention for memory?
GQA shares key/value heads across multiple query heads, so the KV cache scales with the smaller number of key-value heads rather than the full attention head count. A model with 64 attention heads but only 8 key-value heads has an 8x smaller KV cache than a naive estimate would suggest. The calculator reads num_key_value_heads from the model config to get this right.
Can I run a 70B model on a single 24 GB GPU?
Not in FP16 — a 70B model needs roughly 140 GB just for FP16 weights. In 4-bit it drops to about 40 GB, which still exceeds 24 GB. On a single 24 GB card you are realistically limited to models up to about 13B in 4-bit, or 7B in FP16, once you leave headroom for the KV cache. For 70B you need either an 80 GB GPU in 4-bit, or multiple GPUs with tensor parallelism.
Are these numbers exact?
They are planning estimates, not a substitute for profiling. Real usage is affected by the inference runtime (vLLM, TGI, llama.cpp), memory fragmentation, paged-attention efficiency, CUDA graph capture, and optimizer state during training. Treat the estimate as a lower bound and keep 10–20% headroom before committing to hardware.
Keep going
Once you know the memory footprint, the next questions are which GPU to buy and whether to quantize. These guides go deeper: