Consumer · NVIDIA · 8 GB VRAM

What can you run on the RTX 4060?

The RTX 4060 has 8 GB of VRAM. Of the 38 models we track, 23 run on this card — 9 at full FP16 precision and 14 once quantized. Every figure below counts model weights plus the KV cache at a 4,096-token context.

Every model on the RTX 4060, at every precision

Total VRAM for each model at FP16, INT8, and 4-bit, including the KV cache at 4,096tokens. Green fits with headroom, amber fits but leaves under 10% spare, red exceeds the card's 8 GB. Models are ordered largest first, so the row where the colours change is the capacity limit of this card.

Weights scale with precision — roughly 2 bytes per parameter at FP16, 1 at INT8, 0.5 at 4-bit — but the KV cache column does not. It stays in FP16 regardless of how the weights are quantized, which is why 4-bit stops helping once the cache alone approaches the card's capacity. Speed is quoted at the best precision each model actually runs at on this card, for a single request.

Fitting is not the same as usable

A model can fit the RTX 4060and still be too slow to use, which is the gap the speed column closes. Generating text is memory-bandwidth-bound rather than compute-bound: producing each token requires reading every weight once, so throughput is roughly the card's bandwidth divided by the bytes those weights occupy. The RTX 4060 moves about 272 GB/s at peak, and a well-optimized runtime realizes roughly 60% of that on a single stream once attention, sampling, and kernel launch overhead are counted.

Two consequences follow. Quantization buys speed as well as memory — dropping from FP16 to 4-bit quarters the bytes read per token and so roughly quadruples generation speed, which is often the stronger argument for it. And bandwidth, not TFLOPS, is the spec that predicts how a card feels: a GPU with more compute but slower memory will generate text more slowly on the same model. As a rough guide, above 30 tokens per second output arrives faster than most people read, 10 to 30 is comfortable for chat but sluggish for long answers, and below about 3 the model is only practical for background or batch work.

These are single-request estimates and deliberately conservative. Serving several users at once raises total throughput well above these figures, because one weight read is shared across the whole batch, while each individual response gets no faster. Speculative decoding and heavily tuned runtimes can also beat them. Treat the numbers as a floor for interactive use rather than a benchmark result.

Where this card hits its limit

Full precision on the RTX 4060 runs out after Qwen2.5 3B Instruct at 3.09B parameters, which uses about 7.2 GB of the available 8 GB. That is the first cliff, and it is the one that costs you output quality rather than the ability to run at all. The second cliff is harder. Phi-4 needs roughly 9.2 GB even at 4-bit, past the point where quantization can rescue it. Below that size you are choosing a precision; above it you are choosing a different card, or roughly 2 of these with tensor parallelism.

The practical reading is that VRAM is a step function, not a slider. Between the two cliffs, dropping from FP16 to INT8 costs very little measurable quality on most chat and summarization work, and 4-bit is usually acceptable too — but structured output, tool calling, and code generation degrade first and degrade quietly, so re-run your own evaluations after quantizing rather than trusting the benchmark deltas.

What happens at longer context

The table above assumes a 4,096-token prompt. Weights do not change with context but the KV cache grows linearly with it, so a model that fits comfortably on a short prompt can run out of memory in a long conversation. These are the largest models that fit the RTX 4060, recomputed at their best precision as context grows.

Any row that turns amber or red before the last column is a model you can demo but cannot ship on this card at that context. Two levers help before you buy more hardware: switch to a model with grouped-query attention, which cuts the cache by the ratio of attention heads to key-value heads, or cap the served context below the model's maximum. Batching works against you here — every concurrent request carries its own cache, so serving four users at 8K costs roughly the same as one user at 32K.

How to read these numbers

Whether a model runs on the RTX 4060 comes down to three memory costs measured against its 8 GB. First the weights: parameter count times bytes per parameter — 2 bytes in FP16, 1 in INT8, roughly 0.5 at 4-bit. Second the KV cache, which holds attention keys and values for every token in the context window and grows linearly with prompt length; it stays in FP16 even when the weights are quantized. Third, an allowance for activations, CUDA context, and fragmentation. A fit is only called comfortable when the total leaves about 10% headroom.

These are architecture-aware estimates, not benchmarks. Each model's layer count, key-value head count, and head dimension are read from its published config rather than assumed from parameter count, which matters because two 7B models with different attention layouts can differ by several gigabytes of cache. What the estimates cannot capture is runtime overhead: vLLM preallocates a large block of VRAM by design, llama.cpp can offload part of the model to system RAM, and driver and framework versions each take their own cut. Treat a comfortable fit as a green light and a tight fit as something to verify on the actual card before committing.

For numbers at your own context length, batch size, and quantization scheme, use the VRAM Calculator. To work the problem the other way — starting from a model and finding the cheapest card that runs it — use the GPU Picker.

Compare other consumer GPUs

Frequently asked questions