Advanced Performance
CUDA Graph Optimization for LLM Inference
CUDA graphs can reduce CPU launch overhead by capturing repeated GPU work, but they require stable shapes and careful runtime support.
What You Will Learn
- - CUDA graphs reduce repeated kernel launch overhead.
- - They work best when execution shapes are stable.
- - Dynamic request patterns can reduce their effectiveness.
- - Measure latency improvements separately from memory or quality changes.
Author and Review
Author: Dhiraj
Technical review: InnoAI Technical Review
Review process: Content is reviewed for technical clarity, deployment realism, and consistency with currently published product pages and tools.
Key Takeaways
- - CUDA graphs reduce repeated kernel launch overhead.
- - They work best when execution shapes are stable.
- - Dynamic request patterns can reduce their effectiveness.
- - Measure latency improvements separately from memory or quality changes.
1. The launch overhead problem
GPU workloads are made of many kernel launches. For small or repeated operations, CPU-side launch overhead can become visible. LLM inference includes repeated decoding steps, so reducing launch overhead can improve latency. CUDA graphs let a runtime capture a sequence of GPU operations and replay it more efficiently when shapes and execution patterns are compatible.
2. What CUDA graphs capture
A CUDA graph represents a recorded set of GPU operations and dependencies. Instead of issuing every operation individually each time, the runtime can replay the captured graph. This can reduce overhead and improve consistency. The challenge is that graph capture prefers stable tensor shapes, memory addresses, and control flow. Highly dynamic serving can make capture more complicated.
3. Where LLM inference benefits
Decode loops often repeat similar operations token after token. If the runtime can stabilize shapes through batching or padding, graph replay can help. The benefit may show up as lower per-token latency or smoother p95 behavior. Prefill for long prompts may be less graph-friendly because prompt lengths vary widely, though runtime-specific techniques can still help.
4. Compatibility issues
CUDA graph support depends on the inference engine, model architecture, quantization, GPU, driver, and framework version. Some features may disable graph capture or force fallback paths. Developers should check runtime logs and benchmark both enabled and disabled modes. Silent fallback can make teams believe an optimization is active when it is not.
5. Shape management
Stable shapes are the central operational requirement. Serving systems may bucket requests, pad sequences, or use fixed batch sizes to make graphs reusable. Those choices can improve GPU efficiency but may waste some compute. The best configuration depends on traffic patterns. A public chat product and a nightly batch job usually need different settings.
6. Measuring impact
Measure time to first token, per-token decode latency, p95 latency, and CPU utilization. CUDA graphs primarily target launch overhead, so they should not be credited for changes caused by quantization, cache limits, or model switching. Use the same model, precision, context length, and runtime settings except for the graph option.
7. Failure modes
Common problems include capture errors, excessive padding, increased memory reservation, incompatibility with dynamic shapes, and confusing benchmark results. If graph capture increases memory enough to reduce concurrency, the net result may be negative. Treat it as a production tuning option, not a default assumption.
8. Practical recommendation
Use CUDA graphs after the basic deployment is stable. First choose the model, runtime, precision, and batching strategy. Then test graph capture on representative traffic. Keep rollback simple because graph-related issues can appear only under specific shapes or concurrency levels.
9. The capture-and-replay lifecycle
A CUDA graph is recorded once via stream capture, which traces the full directed graph of kernels and their dependencies for one decode step. On every subsequent step the runtime replays the captured graph instead of re-issuing and re-validating each kernel from the CPU. The launch overhead — normally a few microseconds per kernel, multiplied by dozens of kernels per token — is paid once and amortized across thousands of tokens. That is why the benefit shows up as lower, steadier per-token decode latency rather than higher peak throughput.
10. Why stable shapes matter
Graph replay assumes the same tensor shapes and memory addresses as capture, so serving stacks bucket requests into a small set of fixed batch sizes and pad to them; a captured graph exists per bucket. Prefill, where prompt lengths vary widely, is harder to graph and often uses piecewise or partial capture instead. The cost is memory: each captured graph reserves its own working set, so capturing many buckets can reduce the concurrency headroom you were trying to protect. Capture a few well-chosen batch sizes rather than every possible shape — a common compromise is to graph only the handful of batch sizes your scheduler actually produces under load, then let rare or oversized shapes fall back to normal eager execution without a captured graph.
11. Piecewise capture and the relationship to torch.compile
Full-graph capture is the textbook description, but it is not how most production stacks actually work, because a decode step contains operations that resist capture — dynamic KV cache indexing, custom attention kernels, and anything that branches on runtime values. Piecewise capture is the practical answer: the runtime captures the long stretches that are shape-stable, typically the feed-forward and projection layers, and leaves the attention path to execute eagerly. That recovers most of the launch-overhead saving without demanding that the entire step be static. This is also where CUDA graphs meet torch.compile, and the two are complementary rather than competing. torch.compile traces the model into an intermediate representation and fuses operations, reducing the number of kernels that exist at all; CUDA graphs reduce the CPU cost of launching whichever kernels remain. Applying compilation first and capture second usually gives more than either alone. The cost is startup time and memory. Each captured graph and each compiled shape variant is stored, so a runtime that buckets batch sizes into eight variants pays that overhead eight times, and warmup can add tens of seconds before the first request is served. On a long-lived server that is irrelevant. On a scale-to-zero deployment where cold starts are user-visible, it can easily outweigh the per-token gain, which is why the same configuration can be clearly right in one deployment and clearly wrong in another.
Implementation Checklist
- - Stabilize batch and sequence shapes first — graph capture needs them, and bucketing is how you get them.
- - Confirm capture succeeded in runtime logs rather than assuming the flag took effect.
- - Watch reserved memory after enabling capture; extra reservation can cost you more concurrency than it saves.
- - Measure per-token decode latency and p95, since launch overhead is what graphs actually address.
- - Keep the disable path one config change away — graph faults often surface only at specific shapes.
- - Stabilize decode shapes (fixed batch buckets, padding) before enabling capture.
- - Benchmark graphs on vs off with identical model, precision, and context.
- - Watch reserved memory — many captured graphs can shrink concurrency.
- - Verify the runtime did not silently disable capture for your config.
- - Apply CUDA graphs last, after model, precision, and batching are settled.
FAQ
Do CUDA graphs improve quality?
No. They are a performance optimization and should preserve model outputs.
Are they NVIDIA-specific?
CUDA graphs are part of NVIDIA CUDA; other platforms have different mechanisms.
Should beginners tune CUDA graphs first?
No. Start with model size, precision, runtime, and batching before advanced graph tuning.
Why did enabling graph capture reduce my concurrency?
Each captured graph reserves memory, and a runtime that buckets several batch sizes stores one per bucket. That reservation comes out of the same pool as the KV cache, so if you were already near the memory limit the lost cache capacity can outweigh the launch-overhead saving.
Do CUDA graphs help prefill or decode more?
Decode. The token-by-token decode loop repeats the same small kernels with stable shapes, which is ideal for graph replay. Prefill has variable prompt lengths and larger kernels, so launch overhead matters less and capture is harder.
Related Guides
Decision Resources
Sources and Methodology
This guide combines public model metadata with practical deployment heuristics used in InnoAI tools.
Continue Your Journey
Editorial Disclaimer
This guide is for informational and educational purposes only. Validate assumptions against your own workload, compliance requirements, and production environment before implementation.