Why hardware knowledge matters
AI teams often jump straight to model benchmarks, but hardware limits are what decide whether a model can actually run, how much it costs, and where latency problems show up.
Hardware / Architecture
Streaming Multiprocessor
The SM is the main architectural unit in modern GPUs. This page summarizes the key hardware blocks that matter most for AI and inference performance tuning.
What this page helps decide
Use this page to build intuition around the blocks that determine throughput, memory pressure, and deployment feasibility before moving into execution or performance tuning.
Best next step
After this overview, continue to Execution for warp behavior or Performance for roofline reasoning. If you already have a workload, jump to the VRAM calculator and GPU picker.
Streaming Multiprocessor
Core execution block where warps are scheduled and tensor pipelines execute.
Tensor Cores
Specialized matrix units for BF16/FP16/FP8 workloads and high-throughput AI kernels.
Memory Hierarchy
Registers, shared memory, L2, and HBM/GDDR bandwidth shape achievable performance.
SM Diagram Snapshot
Warp Schedulers
4x
Dispatch Units
8x
CUDA Cores
128x
Tensor Cores
4x
Related Navigation
How to read GPU hardware pages
Start with blocks
Focus first on what each hardware block does: scheduling, matrix math, caches, and memory transport. That foundation makes later performance discussions much easier to understand.
Then connect to workload
The useful question is always: which of these blocks is likely limiting my model, kernel, or serving pattern? This page is meant to build that habit.
The memory hierarchy is the real constraint
GPU specifications lead with core counts and teraflops, but for AI workloads the memory system decides almost everything. Each level of the hierarchy trades capacity for speed. Registers sit closest to the arithmetic units and are effectively free to access, but there are only tens of kilobytes per streaming multiprocessor and they are divided among every resident thread. Shared memory — a programmer-managed scratchpad inside the SM — offers roughly a hundred kilobytes per block at latency measured in tens of cycles. Global memory (HBM or GDDR) holds gigabytes but costs hundreds of cycles per access.
That last gap is the one that matters. A read from global memory can cost several hundred cycles, during which the arithmetic units would sit idle if the scheduler had nothing else to run. Every major GPU optimization technique is, at bottom, an attempt to avoid paying that cost: tiling data into shared memory so it is read from HBM once and reused many times, coalescing accesses so 32 threads fetch one contiguous block instead of 32 scattered ones, and keeping enough warps resident that the SM always has other work available while one warp waits.
For running language models this shows up immediately in a practical form. Weights must live in VRAM, and the KV cache grows alongside them with context length — which is why a card is chosen by capacity first and speed second. Once the model fits, generation speed tracks memory bandwidth far more closely than it tracks peak FLOPs, because producing each token means streaming the whole weight matrix out of memory again. A card with more bandwidth generates faster even at identical compute. You can size the capacity side precisely with the VRAM calculator or check a specific pairing in can I run it.
SMs and tensor cores in practice
The streaming multiprocessor is the unit that actually does work. A GPU is essentially many SMs plus a memory system to feed them, and each SM contains CUDA cores for general arithmetic, tensor cores for matrix math, a register file, shared memory, schedulers, and load/store units. Work arrives as thread blocks assigned to an SM, and each block is subdivided into 32-thread warps that the schedulers issue from. How many blocks an SM can hold at once depends on how much of its register file and shared memory each block claims, which is why resource usage in your kernel translates directly into how well latency can be hidden.
Tensor cores are the reason modern GPUs are fast at AI specifically. Rather than computing one multiply-accumulate per cycle per lane, a tensor core performs a small matrix multiply-accumulate as a single operation, delivering an order-of-magnitude throughput increase on exactly the operation that dominates transformer workloads. The catch is that they only engage under the right conditions: reduced precision (BF16, FP16, FP8, or INT8 depending on generation), and tensor shapes that align to the hardware's expected tile sizes. A model running in FP32, or with awkward dimensions, quietly falls back to the standard cores and gives up most of the advertised performance — one of the most common reasons real throughput lands far below a datasheet figure.
Frequently asked questions
What is a streaming multiprocessor?
The core execution block of a GPU. It contains CUDA cores, tensor cores, a register file, shared memory, and warp schedulers. A GPU is many SMs plus the memory system that feeds them.
Why does VRAM bandwidth matter more than FLOPs for LLMs?
Generating each token requires reading the full set of weights from memory to perform relatively little arithmetic. That makes decoding memory-bound, so bandwidth sets the speed while peak compute goes largely unused until you batch requests together.
When do tensor cores actually get used?
When the operation is a matrix multiply in a supported reduced precision (BF16, FP16, FP8, or INT8 by generation) with shapes that match the hardware tile sizes. FP32 kernels or misaligned dimensions fall back to standard CUDA cores.
How much VRAM do I need for a given model?
Roughly two bytes per parameter in FP16, one at INT8, and about half at 4-bit, plus the KV cache which grows with context length and concurrency. Add headroom for activations and fragmentation rather than sizing to the exact weight footprint.