How a warp actually executes

A GPU does not schedule individual threads. On NVIDIA hardware it schedules warps — groups of 32 threads that issue the same instruction in the same cycle. This is the SIMT (single instruction, multiple thread) model, and it is the single most important thing to understand about GPU performance, because almost every surprising result traces back to it. Thirty-two threads share one instruction pointer, so they move through your kernel together whether or not that is what your code implies. AMD hardware uses the same idea with a 64-thread wavefront (32 on RDNA), so the reasoning transfers.

When threads inside one warp hit a branch and disagree — some take the if, others the else — the hardware cannot run both at once. It executes one path with the disagreeing lanes masked off, then the other path with the mask inverted. Both paths cost full time, and the masked lanes contribute nothing. That is warp divergence. A two-way branch can halve throughput; a 32-way switch where every lane picks a different case can, in the worst case, cost 32× the time of a uniform warp while doing the same amount of useful work. Since Volta, independent thread scheduling lets warps reconverge more flexibly, but the cost of executing both sides remains.

The critical detail people miss: divergence only costs you within a warp. If threads 0–31 all take the true branch and threads 32–63 all take the false branch, those are two different warps and there is no penalty at all. This is why the fix for divergence is almost never removing the branch — it is reorganizing the data so that threads that will take the same path land in the same warp. Sorting or bucketing work by branch condition before launching the kernel routinely recovers most of the loss.

Occupancy and latency hiding

A GPU tolerates slow memory not by making it faster but by having enough other work queued to run while it waits. A read from global memory costs hundreds of cycles. Rather than stall, the streaming multiprocessor swaps to another resident warp and issues its instruction instead. Occupancy — the ratio of resident warps to the hardware maximum — measures how much of that alternative work is available.

Occupancy is capped by whichever per-SM resource runs out first: registers per thread, shared memory per block, or the block and warp limits themselves. A kernel using many registers per thread lets fewer warps stay resident, which reduces the pool available to hide latency. This is why register pressure shows up as a memory-latency problem even though nothing about the memory changed.

The common misconception is that higher occupancy is always better. It is not. Occupancy only needs to be high enough to cover the latency actually present; past that point, pushing it higher usually means cutting registers per thread, which forces spills to local memory and makes the kernel slower. Treat occupancy as a floor to clear, not a number to maximize. Measure with the occupancy estimator, then confirm against real kernel timings rather than trusting the ratio alone.

Frequently asked questions