Intermediate Inference
Tensor Parallelism for LLM Inference
Tensor parallelism splits model computation across multiple GPUs so larger models or higher throughput deployments can become practical.
What You Will Learn
- - Tensor parallelism is used when one GPU cannot comfortably hold or serve the model.
- - It introduces communication overhead between GPUs.
- - Fast interconnects matter for performance.
- - It should be tested against quantization and smaller-model alternatives.
Author and Review
Author: Dhiraj
Technical review: InnoAI Technical Review
Review process: Content is reviewed for technical clarity, deployment realism, and consistency with currently published product pages and tools.
Key Takeaways
- - Tensor parallelism is used when one GPU cannot comfortably hold or serve the model.
- - It introduces communication overhead between GPUs.
- - Fast interconnects matter for performance.
- - It should be tested against quantization and smaller-model alternatives.
1. What tensor parallelism means
Tensor parallelism splits parts of transformer computation across multiple GPUs. Instead of putting a full copy of the model on every GPU, layers or matrix operations are partitioned so each device handles a slice. This can make models practical when their weights, KV cache, or throughput target exceed a single GPU. It is common in large-model inference, but it adds coordination cost.
2. Why it is used
The obvious reason is memory. A model with an FP16 footprint above one GPU capacity needs sharding, quantization, offload, or a smaller model. Tensor parallelism can also improve throughput when a single GPU is too slow. However, splitting work means GPUs must communicate partial results. If interconnect bandwidth is poor, the communication overhead can erase much of the benefit.
3. How it changes deployment planning
Single-GPU planning asks whether the model and KV cache fit. Tensor-parallel planning asks whether the model fits across GPUs and whether communication stays acceptable. The number of GPUs, GPU memory, NVLink or PCIe topology, batch size, sequence length, and runtime implementation all matter. The same model can perform differently on four GPUs in one server versus four GPUs spread across slower links.
4. When to prefer quantization instead
If a model barely misses single-GPU memory, quantization may be simpler than tensor parallelism. A strong 8-bit or 4-bit variant can reduce operational complexity and avoid communication overhead. Tensor parallelism becomes more attractive when quality requirements demand higher precision, the model is too large even after conservative quantization, or high throughput justifies multiple GPUs.
5. Runtime support
Runtimes such as vLLM, TensorRT-LLM, and other distributed inference systems can support tensor parallel patterns, but configuration details differ. Developers should read runtime-specific docs and avoid assuming flags are portable. Pay attention to supported architectures, quantization compatibility, maximum context length, and how the runtime handles KV cache across devices.
6. Failure modes
Common problems include out-of-memory errors despite sharding, slow generation due to communication overhead, uneven GPU utilization, unsupported quantized formats, and unexpected latency spikes under concurrency. These issues are easier to diagnose when you log model revision, tensor parallel size, context length, batch settings, GPU type, and interconnect topology.
7. Measurement strategy
Measure baseline single-GPU or quantized performance first if possible. Then test tensor parallel sizes such as two, four, or eight GPUs. Track throughput, latency, memory per GPU, and utilization. Watch for cases where adding GPUs increases throughput but hurts p95 latency. Production systems often need a balance rather than maximum aggregate tokens per second.
8. Practical recommendation
Use tensor parallelism when model quality or throughput justifies multi-GPU complexity. For many teams, the simpler path is a smaller model, quantization, or routing between small and large models. When tensor parallelism is necessary, choose hardware with strong interconnects, keep runtime versions pinned, and test representative prompts before committing spend.
9. How the split actually works
Tensor parallelism partitions the big matmuls inside each layer. In the Megatron pattern, attention projections and the first feed-forward matrix are split column-wise, the second feed-forward matrix row-wise, and an all-reduce combines partial results after each block. Every token therefore triggers cross-GPU communication twice per layer. This is why tensor parallelism is latency-sensitive and wants NVLink or NVSwitch: on PCIe-only links the all-reduce traffic can erase the compute savings, especially at small batch sizes where there is little work to hide the communication behind.
10. Tensor vs pipeline vs data parallel
These three axes solve different problems. Tensor parallelism splits within a layer to cut per-GPU memory and latency, but needs fast interconnect. Pipeline parallelism assigns whole layer ranges to different GPUs; it tolerates slower links but introduces pipeline "bubbles" that hurt latency at low batch. Data parallelism replicates the full model to raise throughput and does nothing for a model that does not fit. Large clusters combine them — tensor parallel inside a node, pipeline across nodes — but for a single 8-GPU box, plain tensor parallelism is usually the simplest path to serving a 70B model.
11. Pipeline parallelism, and why the two are not interchangeable
Tensor parallelism splits individual matrix operations across GPUs, so every device participates in every layer and they must synchronize several times per token. Pipeline parallelism instead assigns whole contiguous blocks of layers to each GPU, so a token passes through GPU 0, then GPU 1, and so on. The communication profiles are opposite. Tensor parallelism moves small tensors very frequently and is punishing over slow links, which is why it wants NVLink and generally stays inside a single node. Pipeline parallelism moves activations once per stage boundary, tolerates PCIe or even Ethernet, and therefore scales across nodes — but it introduces pipeline bubbles, because with a single request in flight most stages sit idle waiting their turn. That makes pipeline parallelism a throughput technique that hurts single-request latency, and tensor parallelism a latency technique that demands fast interconnect. Large deployments combine them: tensor-parallel within each node where the links are fast, pipeline-parallel across nodes where they are not. For most teams the decision is simpler than the theory suggests. If the model fits in one node, use tensor parallelism sized to a divisor of the attention head count. Only reach for pipeline stages when a single node genuinely cannot hold the model, and expect to feed it concurrent traffic to keep the bubbles filled.
Implementation Checklist
- - Check whether quantization alone clears the memory gap before adding a second GPU.
- - Confirm the interconnect topology — NVLink versus PCIe changes the outcome more than GPU count does.
- - Set tensor parallel size to a divisor of the attention head count, or the runtime will reject the config.
- - Compare p95 latency across TP sizes, not just aggregate throughput; more GPUs can worsen tail latency.
- - Verify your quantization format is supported under sharding — several are single-GPU only.
- - Confirm the interconnect (NVLink vs PCIe) before choosing a tensor-parallel size.
- - Set tensor-parallel-size to a value that divides the attention head count.
- - Compare against a 4-bit single-GPU run before committing to multi-GPU complexity.
- - Measure p95 latency, not just aggregate throughput, as you add GPUs.
- - Pin runtime and driver versions; sharded paths are the most version-sensitive.
FAQ
Does tensor parallelism make inference linearly faster?
Not usually. Communication overhead prevents perfect scaling.
Do I need NVLink?
Not always, but faster interconnects usually improve large-model sharded inference.
Is tensor parallelism the same as data parallelism?
No. Tensor parallelism splits model computation; data parallelism replicates the model for separate batches.
Can I mix different GPU models in a tensor-parallel group?
Technically sometimes, practically no. Every rank synchronizes at each layer boundary, so the slowest card sets the pace and the fastest one idles. Mismatched VRAM is worse still, because the shard size is bounded by the smallest card. Keep a tensor-parallel group homogeneous.
Why must the tensor-parallel size divide the number of attention heads?
Each GPU handles a whole-number slice of the attention heads. If the head count is not divisible by the tensor-parallel size, the heads cannot be split evenly and the runtime will reject the configuration.
Related Guides
Decision Resources
Sources and Methodology
This guide combines public model metadata with practical deployment heuristics used in InnoAI tools.
Continue Your Journey
Editorial Disclaimer
This guide is for informational and educational purposes only. Validate assumptions against your own workload, compliance requirements, and production environment before implementation.