Use with real traffic assumptions
The right GPU for one-user local testing may be the wrong GPU for concurrent production traffic. Always connect the result to expected request volume and context length.
Precision Tool
GPU Picker
Enter a HuggingFace model ID and let the app auto-detect parameters, precision, and VRAM needs before ranking practical GPU options for inference or deployment.
Model-aware GPU shortlistBudget and intent filtersUseful after model choice, before infra commitment
Pick a model and usage intent to get GPU recommendations.
Pair with comparison
If multiple models fit your hardware budget, move back to compare and choose based on license, context, and ecosystem rather than GPU fit alone.
Related reading
Read Choose an AI Model by GPU and Budget for the broader framework behind these recommendations.
How to choose the right GPU for an AI model
Picking a GPU for AI work is a balance of four constraints that rarely point at the same card: how much memory the model needs, how fast you want tokens generated, how much you can spend, and whether you are running inference or training. A card that is perfect for serving a quantized 13B chatbot is wildly over- or under-specified for fine-tuning a 7B model or batching thousands of embedding requests. This tool starts from the model you actually want to run and works backwards to the hardware.
Memory comes first
If the model plus its KV cache does not fit in VRAM, nothing else matters — the workload will not run. Always confirm the fit at your target precision and context length before comparing speed or price.
Then throughput
Once several cards fit, memory bandwidth and compute (TFLOPS) decide tokens per second. Generation is usually memory-bandwidth bound, so bandwidth often matters more than raw FLOPS for chat workloads.
Then budget tier
Consumer cards (RTX 3060–4090) offer the best value per GB; workstation and data-center cards buy you capacity, ECC memory, and multi-GPU scaling at a steep premium.
Finally, intent
Inference, batch serving, and fine-tuning have very different memory profiles. Tell the picker your intent so it does not hand you an inference-sized card for a training job.
Reading the recommendations
Each recommended card shows the precision it was matched at and the resulting VRAM utilization. A card at 95% utilization technically fits but leaves no room for longer prompts or larger batches — prefer options in the 60–80% range for production. The quantized alternatives panel shows how dropping from FP16 to INT8 or 4-bit changes which cards become viable, which is often the difference between a $1,600 consumer card and a $15,000 data-center card. When several cards fit comfortably, decide on price-to-performance and availability rather than squeezing into the smallest possible option.
If no consumer card fits even at 4-bit, the model is genuinely large and you should plan for a workstation card, a data-center GPU, or a multi-GPU server. In that case, weigh a hosted inference API against self-hosting: for occasional use, per-token pricing is usually cheaper than owning 80 GB of GPU.
Frequently asked questions
What is the best GPU for running LLMs locally?
For local inference the RTX 4090 (24 GB) and RTX 3090 (24 GB) are the sweet spot in 2024–2025: they run 7B–13B models in FP16 and up to ~34B models in 4-bit. If you only run small models, a 16 GB RTX 4060 Ti or a used RTX 3060 12 GB is enough. For 70B-class models you need a 48 GB workstation card (RTX A6000) or a data-center 80 GB card (A100/H100). The picker above matches a specific model to concrete cards rather than guessing.
How much GPU memory headroom should I leave?
Never size a GPU to exactly the weight requirement. Leave 20–30% headroom for the KV cache (which grows with context length), activations, and memory fragmentation in the inference runtime. A model whose 4-bit weights are 22 GB will not run reliably on a 24 GB card once you add a long prompt. The picker applies a safety margin automatically.
Should I buy a GPU or rent one in the cloud?
Rent when your usage is spiky, experimental, or below roughly 15–20 hours per week; the pay-per-hour math usually beats hardware depreciation. Buy when you have steady, high-utilization workloads or strict data-residency needs — a 24 GB consumer card pays for itself in a few months of heavy use versus cloud A10/L4 instances. Consider power and cooling costs in the buy case.
Is one big GPU better than two smaller ones?
A single GPU with enough VRAM is almost always simpler and faster than two smaller cards, because splitting a model across GPUs (tensor parallelism) adds inter-GPU communication overhead. Use multiple GPUs only when the model genuinely does not fit on one card, or when you need the extra cards to raise throughput with data parallelism. Two 24 GB cards do not cleanly equal one 48 GB card for a single large model.
Do AMD GPUs work for AI inference?
Yes, increasingly. AMD data-center cards (MI250/MI300) and consumer RDNA cards run PyTorch and llama.cpp through ROCm, and vLLM has growing ROCm support. The ecosystem is still narrower than NVIDIA CUDA, so some quantization kernels and libraries may lag. If tooling breadth matters more than price, NVIDIA is the safer default; if raw VRAM per dollar matters, AMD is worth evaluating.
What GPU do I need to fine-tune a model?
Fine-tuning needs far more memory than inference because you also store gradients and optimizer states. Full fine-tuning of a 7B model can need 60–80 GB, while parameter-efficient methods like LoRA or QLoRA bring a 7B model down to a single 16–24 GB consumer card. If your goal is fine-tuning, filter for the QLoRA-friendly cards and expect to trade some quality for the memory savings.