Back to GPU Hub05 / Learning Path
CUDA Programming
Kernel design, memory access, and runtime optimization.
Main Sections
Track your progress
Sign in to save your progress across all 74 sections, take the final test, and earn a certificate.
Sub Topics
Topic 01
01
CUDA Keywords and Memory Qualifiers
Theory
CUDA extends C++ with qualifiers that define execution location and memory placement.
These qualifiers tell the compiler whether code runs on CPU or GPU, and where variables are stored.
Execution space qualifiers
| Qualifier | Meaning |
|---|
| __global__ | Kernel function called on CPU and executed on GPU |
| __device__ | Function called and executed on GPU |
| __host__ | Standard CPU-only C++ function |
Memory qualifiers
| Qualifier | Memory behavior |
|---|
| __shared__ | Places variables in per-block shared memory on SM |
| __constant__ | Places read-only values in constant memory cache |
Topic 02
02
Kernel Launch Dimensions and Mapping
Theory
Kernels use execution configuration syntax: kernel<<<blocks, threads>>>(...).
Grid and block shapes can be 1D, 2D, or 3D via dim3, enabling natural mapping to tensors and images.
Built-in indexing variables
| Variable | Role |
|---|
| threadIdx | Thread index inside current block |
| blockIdx | Block index inside grid |
| blockDim | Block dimensions (threads per block) |
Global index formula (1D)
id = blockIdx.x * blockDim.x + threadIdx.x
Topic 03
03
cudaMalloc / cudaMemcpy / cudaFree Fundamentals
Theory
Host RAM and device VRAM are separate memory spaces, so allocation and transfer are explicit in CUDA runtime code.
Data movement over PCIe/NVLink is often a bottleneck, so transfers should be minimized and overlapped where possible.
Core APIs
| API | Purpose |
|---|
| cudaMalloc | Allocate bytes in GPU memory |
| cudaMemcpy | Copy data between host and device memory spaces |
| cudaFree | Release previously allocated GPU memory |
Copy directions
- HostToDevice: push input data from CPU to GPU.
- DeviceToHost: pull output data back from GPU to CPU.
Topic 04
04
Shared Memory Tiling and Synchronization
Theory
Tiling stages reusable chunks of global memory into shared memory so threads can reuse fast on-chip data.
This is foundational for high-performance kernels such as matrix multiplication and convolution.
Synchronization barrier
__syncthreads() is a block-wide barrier. It ensures all threads finish cooperative loads before any thread consumes the tile.
- Without synchronization, threads may read incomplete data (race condition).
- Correct placement of barriers is required for both correctness and performance.
Topic 05
05
Streams and CUDA Graphs
Theory
Default stream execution is ordered; streams enable concurrent kernels and copies when dependencies allow.
CUDA Graphs reduce CPU launch overhead by capturing repeated operation DAGs and replaying them efficiently.
Practical behavior
- Use multiple streams to overlap transfer for task B while task A computes.
- Use CUDA Graphs for repetitive multi-kernel workloads to lower launch overhead and jitter.
