Memory-bound clue
If arithmetic intensity is low and bandwidth is saturated early, your biggest wins usually come from data movement, coalescing, caching, or tiling.
Performance / Bottleneck Analysis
GPU Performance Analysis
Use this page to understand whether a workload is limited by memory bandwidth or compute throughput. The roofline model is useful because it turns low-level measurements into a decision you can act on: optimize memory movement, optimize arithmetic intensity, or accept that the kernel is already near a practical limit.
Compute-bound clue
If arithmetic intensity is high and the roof is compute-limited, look at tensor core usage, instruction mix, occupancy, and whether your kernel is already close to hardware limits.
Related workflows
Pair this page with the occupancy estimator and warp divergence visualizer to connect bottleneck analysis to kernel behavior.
Arithmetic intensity decides your ceiling
The roofline model rests on one ratio: arithmetic intensity, the number of floating-point operations a kernel performs per byte it moves from memory (FLOPs/byte). Every kernel sits somewhere on that axis, and where it sits determines which hardware limit it hits first. Below the ridge point — where the memory-bandwidth roof meets the peak-compute roof — you are bandwidth-bound and adding FLOPs is free. Above it you are compute-bound and only more math throughput helps.
The numbers are stark for AI workloads. An H100 delivers roughly 3.3 TB/s of HBM3 bandwidth against hundreds of teraflops of tensor-core throughput, putting the ridge point far to the right. That means most LLM inference kernels are firmly memory-bound. Token-by-token decoding is the clearest case: each new token requires reading the entire weight matrix from memory to perform a single matrix-vector product, giving an arithmetic intensity close to 1 — hopeless on a machine that wants hundreds. This is why decode speed tracks memory bandwidth almost linearly and barely responds to a faster compute unit.
The practical consequence is that batching is the single most effective optimization in LLM serving. Running one request reads the weights to produce one token. Running thirty-two requests together reads the same weights once and produces thirty-two tokens, multiplying arithmetic intensity by thirty-two and shifting the kernel toward the compute roof where the hardware is actually fast. That is the entire reason continuous batching exists, and why an idle-but-quantized model can still be slow while a busy one is efficient.
Reading a roofline plot without fooling yourself
A point far below both roofs is not automatically a bandwidth problem. It usually means something else is wasting cycles: uncoalesced memory access, where threads in a warp read scattered addresses and force the memory system to issue many transactions instead of one; low occupancy that leaves no warps available to hide latency; or warp divergence serializing execution. Fix those before concluding the hardware is the limit — they move the measured point upward without any change to the roofs.
Be careful which roof you compare against, too. Peak FLOPs figures quoted by vendors usually assume tensor cores at low precision with perfect utilization. If your kernel runs FP32 on the standard CUDA cores, the relevant ceiling is far lower than the marketing number, and a kernel that looks like it is achieving 10% of peak may already be near its real limit. Use the roofline analyzer with the precision your kernel actually uses, and confirm with measured timings rather than trusting a single plotted point.
Frequently asked questions
What is arithmetic intensity?
Floating-point operations performed per byte moved from memory. It places a kernel on the roofline plot and determines whether bandwidth or compute is the binding constraint.
Why is LLM inference memory-bound?
Generating one token requires reading every weight once to do a single matrix-vector multiply, so arithmetic intensity is near 1 while modern GPUs need hundreds of FLOPs per byte to saturate compute. Decoding speed therefore scales with memory bandwidth, not peak FLOPs.
Does batching change the bottleneck?
Yes. Batch size multiplies arithmetic intensity because the same weights serve many sequences per read, moving the kernel from the memory roof toward the compute roof. It is the highest-leverage change in most serving stacks.
My kernel is below both roofs — what now?
Look for uncoalesced memory access, insufficient occupancy to hide latency, or warp divergence. Those waste cycles without changing arithmetic intensity, and fixing them raises the measured point.