Back to GPU Hub06 / Learning Path
Driver Stack
Runtime, driver, and system-level GPU software flow.
Main Sections
Track your progress
Sign in to save your progress across all 74 sections, take the final test, and earn a certificate.
Sub Topics
Topic 01
01
NVIDIA Driver Module Responsibilities
Theory
The NVIDIA driver is the control bridge between the operating system and GPU hardware.
It is responsible for isolation, scheduling, command submission, and fault handling so multiple applications can safely share the GPU.
Core responsibilities
- Resource management: allocate VRAM and protect process memory boundaries.
- Instruction submission: translate software requests into GPU command streams and queues.
- Context switching: multiplex GPU access across processes over time slices.
- Error handling: surface XID-class failures, watchdog events, and thermal throttling conditions.
Topic 02
02
Driver API vs Runtime API
Theory
CUDA exposes two programming layers: a high-level Runtime API and a low-level Driver API.
The Runtime API simplifies usage, while the Driver API offers explicit control over contexts and module loading.
Comparison table
| Aspect | Runtime API (cuda*) | Driver API (cu*) |
|---|
| Abstraction level | High-level convenience | Low-level explicit control |
| Context handling | Automatic | Manual |
| Ease of use | Easier | More verbose |
| Typical users | Most CUDA app developers | Framework/runtime/library developers |
| Relationship | Built on Driver API | Foundation layer |
Topic 03
03
NVML and nvidia-smi Metric Semantics
Theory
nvidia-smi is powered by NVML (NVIDIA Management Library), a system management interface for telemetry and controls.
Interpreting metrics correctly is essential because many counters represent activity windows, not absolute saturation.
Metric semantics
| Metric | What it means |
|---|
| GPU Utilization | At least one kernel was active during the sampling interval |
| Memory Utilization | Memory controller activity time, not just memory capacity usage |
| Power Draw | Instantaneous board power, critical for datacenter power envelopes |
| Memory Used | Allocated VRAM footprint, not bandwidth pressure |
Topic 04
04
CUPTI Role for Profiling
Theory
CUPTI (CUDA Profiling Tools Interface) provides profiling hooks and hardware counter access for performance analysis.
Tools such as Nsight Systems and Nsight Compute rely on CUPTI to expose timing, activity traces, and stall/counter data.
What CUPTI provides
- Tracing: kernel launch/complete timelines and runtime API activity.
- Hardware counters: cache misses, instruction throughput, memory transactions, and more.
- Root-cause signals for bottlenecks such as low bandwidth utilization or high stall percentages.
Topic 05
05
Full App-to-Hardware Software Stack
Theory
End-to-end GPU execution spans multiple software layers, from application code down to kernel drivers and silicon.
Understanding this stack helps isolate where latency, overhead, or incompatibility is introduced.
Stack view
| Layer | Role |
|---|
| Application layer | User code (for example Python + PyTorch) |
| Framework/library layer | cuBLAS, cuDNN, Triton, custom kernels |
| CUDA Runtime | High-level kernel launch and memory orchestration |
| CUDA Driver | System-facing device control and command submission |
| NVIDIA kernel module | OS kernel integration and low-level management |
| GPU hardware | Physical SMs, memory hierarchy, and execution pipelines |
