Training tool
Fine-tuning VRAM calculator
Can you fine-tune this model on your GPU? Compare full fine-tuning, LoRA and QLoRA with every memory term accounted for — weights, gradients, fp32 master copies, optimizer state and activations — at your real sequence length and batch size.
QLoRA · Llama 3.1 8B Instruct · RTX 4090
6.81 GB needed · 24 GB available
Fits comfortably
This leaves about 17.2 GB spare on the RTX 4090. Enough margin to raise batch size or sequence length somewhat before running out.
Training 16,777,216 adapter parameters — 0.209% of the model.
Where the memory goes
| Component | Size | Share |
|---|---|---|
| Base weights | 4.11 GB | 60% |
| Gradients | 0.03 GB | 0% |
| fp32 master weights | 0.06 GB | 1% |
| Optimizer state | 0.13 GB | 2% |
| Activations | 1.39 GB | 20% |
| Runtime overhead | 1.09 GB | 16% |
| Total | 6.81 GB |
All three methods on the RTX 4090
Same model, same sequence length and batch size — only the training method changes.
Full fine-tune
127.9 GB
Needs 6x cards
Every weight updated. Best quality ceiling, highest cost.
LoRA
18.19 GB
Fits comfortably
Frozen bf16 base plus small trainable adapters.
QLoRA
6.81 GB
Fits comfortably
Frozen 4-bit base plus adapters. Cheapest way to train.
Not sure the model even fits for inference? See what the RTX 4090 runs.
How training memory is calculated
The reason fine-tuning surprises people is that weights are the smallest part of the bill. Serving a model needs one copy of the parameters. Training needs that copy plus everything the optimizer requires to update it, and with mixed-precision AdamW that adds up to roughly eight times the inference figure.
The four terms
Weights are 2 bytes per parameter in bf16, or about 0.55 in 4-bit once quantization constants are counted. Gradients add another 2 bytes for every parameter being trained. Master weights add 4 more: mixed precision keeps an fp32 copy so that small updates are not lost to rounding. Optimizer state is the largest single term at 8 bytes per trainable parameter, because AdamW stores two fp32 moments — a running mean and a running variance — for each one. Add those and a full fine-tune costs about 16 bytes per parameter before a single activation is stored.
Why LoRA and QLoRA change the picture
Three of those four terms scale with the number of parameters you are training, not the number the model has. LoRA freezes the base model and inserts small low-rank matrices, so gradients, master weights and optimizer state apply to well under one percent of the parameters. The base model still sits in memory at 2 bytes per parameter, which is why LoRA on an 8B still needs roughly 18 GB. QLoRA goes further by storing that frozen base in 4-bit, cutting the last large term and bringing the same job under 7 GB — the difference between needing a data-center card and using the one already in your desktop.
Activations, and why sequence length hurts
Activations are the intermediate tensors kept from the forward pass so gradients can be computed on the way back. Their size depends on sequence length, batch size, hidden dimension and layer count — and crucially, attention contributes a term proportional to the square of sequence length. That is why moving from 2,048 to 8,192 tokens costs far more than four times the memory, and why cutting sequence length is the most effective response to an out-of-memory error. Gradient checkpointing discards most activations and recomputes them during the backward pass, trading roughly a quarter more compute for an order of magnitude less memory. Leave it on unless you have a specific reason not to.
A worked example
Take Llama 3.1 8B at 2,048 tokens, batch size 1, with checkpointing on. A full fine-tune needs about 128 GB: 16 GB of bf16 weights, 16 GB of gradients, 32 GB of fp32 master weights and 64 GB of Adam state. That is two 80 GB cards minimum. LoRA at rank 16 drops it to about 18 GB, because the trainable set collapses to roughly 17 million adapter parameters and only the frozen 16 GB base remains large — that fits a single 24 GB card. QLoRA quantizes that base to 4-bit and lands near 7 GB, which runs on an 8 GB laptop GPU. Same model, same data, an eighteen-fold spread driven entirely by method.
What these estimates do not cover
The model assumes single-GPU training with standard mixed precision. It does not account for DeepSpeed ZeRO or FSDP, which shard optimizer state and gradients across devices and change the arithmetic substantially. It also assumes an efficient attention implementation; older kernels that materialize the full attention matrix use considerably more. Framework overhead, memory fragmentation, and whether your trainer keeps a separate evaluation batch in memory all move the real number. Treat a comfortable fit as a green light, and a tight fit as something to verify on the actual hardware before committing to a run.
For the inference side of the same decision, size the deployment with the VRAM Calculator and check which card runs the finished model in Can I Run It. If you are still deciding whether to fine-tune at all, RAG vs Fine-Tuning covers when each approach is the right one.
Frequently asked questions
How much VRAM does fine-tuning need compared with inference?
Roughly eight times more for a full fine-tune. Inference needs about 2 bytes per parameter for bf16 weights. Full fine-tuning with AdamW needs about 16: 2 for weights, 2 for gradients, 4 for fp32 master weights, and 8 for the two Adam moments. An 8B model that serves happily in 16 GB needs around 128 GB to train fully.
Why does QLoRA use so much less memory?
It attacks both large terms at once. The base model is frozen and stored in 4-bit rather than 16-bit, cutting the weight term by about four times. Because the base is frozen, gradients, master weights and optimizer state exist only for the small adapter — typically well under 1% of the parameters. What remains is mostly activations, which gradient checkpointing keeps modest.
What LoRA rank should I use?
Start at 16. Rank controls adapter capacity, and its memory cost is almost negligible compared with the base model, so the trade-off is quality and overfitting rather than VRAM. Ranks of 8 to 32 cover most instruction-tuning and style-adaptation work; go higher only when you are teaching genuinely new capability and have the data to support it.
Does sequence length or batch size matter more?
Sequence length, by a wide margin. Activation memory contains a term that grows with the square of sequence length because of attention, while batch size scales it linearly. Doubling the sequence length costs far more than doubling the batch. If you are out of memory, cut sequence length first, then use gradient accumulation to keep the effective batch size you wanted.
Should I turn gradient checkpointing on?
Almost always yes. It discards intermediate activations during the forward pass and recomputes them during the backward pass, trading roughly 20 to 30 percent extra compute for an order of magnitude less activation memory. Without it, activations rather than weights are usually what exhausts VRAM on long sequences.
How accurate are these numbers?
They are planning estimates built from the standard memory model, and they track published results closely — QLoRA on a 70B lands near 44 GB here, consistent with the QLoRA paper fitting a 65B on a single 48 GB card. Real usage varies with framework, attention implementation, fragmentation and whether Flash Attention is active. Treat a comfortable fit as a green light and a tight fit as something to test before relying on it.