Data center · NVIDIA · 40 GB VRAM

What can you run on the A100 40GB?

The A100 40GB has 40 GB of VRAM. Of the 38 models we track, 35 run on this card — 28 at full FP16 precision and 7 once quantized. Every figure below counts model weights plus the KV cache at a 4,096-token context.

Runs at full FP16 precision

These models fit without quantization, so you keep full output quality.

Runs with quantization

These need an INT8 or 4-bit build (GPTQ, AWQ, or GGUF). Quality stays close to full precision for most chat work, but re-test structured output and tool calling before shipping.

Too large for this card

These exceed 40 GB even at 4-bit. You would need multiple cards with tensor parallelism, a larger GPU, or a smaller model.

How to read these numbers

Whether a model runs on the A100 40GB comes down to three memory costs measured against its 40 GB. First the weights: parameter count times bytes per parameter — 2 bytes in FP16, 1 in INT8, roughly 0.5 at 4-bit. Second the KV cache, which holds attention keys and values for every token in the context window and grows linearly with prompt length; it stays in FP16 even when the weights are quantized. Third, an allowance for activations, CUDA context, and fragmentation. A fit is only called comfortable when the total leaves about 10% headroom.

The practical consequence is that the table above is a starting point, not a guarantee. A model listed as fitting at 4,096 tokens can still run out of memory once conversations get long or several requests run at once, because the KV cache term grows with both. If you plan to use long context or serve concurrent users, size with the VRAM Calculator at your real context length before committing.

Compare other data center GPUs

Frequently asked questions