GPU Parallelism in Graphics Workloads

Scope

This document explains how modern GPUs extract parallelism from graphics workloads across major vendors, with focus on:

1. How triangles using different shaders execute concurrently
2. How pixels from the same triangle execute together despite variable latency
3. How outputs from many parallel shader invocations are composited in correct visual order

The exact microarchitecture differs by vendor and generation, but the execution model is broadly similar.

1. Parallelism Layers in the Graphics Pipeline

GPU parallelism is extracted at several levels simultaneously:

1. Command level:

   • Multiple command buffers or queues can be in flight.
   • Graphics and compute may overlap if resources permit.

2. Draw/dispatch level:

   • Different draw calls can overlap in front-end work.
   • Pixel-heavy draws and vertex-heavy draws can overlap in different pipeline stages.

3. Primitive level:

   • Many triangles are transformed, culled, binned, and rasterized concurrently.

4. Fragment/pixel level:

   • Fragments are grouped into SIMD/SIMT execution quanta (warps/waves/subgroups).
   • Many groups run in parallel across many cores.

5. Instruction level:

   • Within each core, schedulers interleave independent instruction streams and switch to ready groups to hide latency.

A key concept: GPUs rely heavily on throughput and latency hiding, not single-thread low latency.

2. How Triangles with Different Shaders Run in Parallel

Practical reality

Within one draw call, all primitives normally use one pipeline state object (PSO), meaning one selected vertex/fragment shader pair (plus variants). So a single draw does not usually mix unrelated shaders per triangle.

Different shader programs typically appear across different draws, material batches, render passes, or pipeline state changes.

How parallel overlap still occurs

Even when shader programs differ, overlap occurs because:

1. Front-end and back-end decoupling:

   • While one draw is in raster/fragment stages, another may be in vertex or setup stages.

2. Deep hardware queues:

   • Work from multiple draws can coexist in internal queues (subject to hazards and resource limits).

3. Multi-engine scheduling:

   • Async compute can run alongside graphics where architecture and dependencies allow.

4. Fine-grained core scheduling:

   • Cores can host many resident groups from different kernels/shader programs (implementation-dependent), switching among ready groups.

Constraints:

• State changes (PSO, render target, blend/depth modes) create boundaries.
• Resource hazards (same UAV/image/buffer writes) may force synchronization.
• API ordering requirements (for example blending to same target) can serialize specific regions.

3. Pixel/Fragment Parallelism Inside a Triangle

After rasterization, covered samples/fragments are generated and grouped into execution quanta:

• NVIDIA: warps (typically 32 lanes)
• AMD: wavefronts/waves (typically 32 or 64 lanes depending on mode/generation)
• Intel Xe: SIMD subgroups across EUs
• Apple/ARM/Qualcomm: similar subgroup concepts under different names

Why fragments from one triangle run "together"

Fragments are usually issued in spatially coherent tiles/quads/subgroups. This improves:

1. Texture cache locality
2. Derivative computations for mip selection
3. Control-flow coherence

Variable-latency operations (texture fetch, memory access)

Texture fetch latency can vary due to cache hit/miss, compression state, format conversion, and memory contention. GPUs handle this by:

1. Massive multithreading:

   • Keep many warps/waves resident per core.

2. Hardware scheduling:

   • If one wave stalls on texture/memory, scheduler issues another ready wave.

3. Scoreboarding/dependency tracking:

   • Instructions execute when operands are ready.

4. Prefetch and caches:

   • Texture units and caches reduce average latency.

5. Occupancy tuning:

   • More resident waves improve hiding of long-latency operations, until register/shared-memory pressure limits occupancy.

Net effect: not every fragment finishes at the same time, but throughput remains high.

A 4-stage quad executor model

The idea of running one quad through a 4-stage, strictly in-order executor is plausible as a simulator or small-GPU design point.

One way to think about it is:

1. Treat the four fragments in a 2x2 quad as four logical lanes.
2. Advance one quad lane per cycle, or one stage per cycle, while keeping the other lanes' state resident.
3. Use SMT-like time slicing so the pipeline stays busy even though execution is strictly ordered within each lane.

That gives you a machine that is not out-of-order, but still has enough lane-level concurrency to model quad execution, derivatives, and masked control flow.

Why this is a good mental model

• Quads are already the natural unit for many fragment operations, especially derivatives and some interpolation behavior.
• A strict in-order machine is simpler to verify than a speculative one.
• The simulator can still expose realistic pressure points: live varying state, texture operands, predicates, and partial results.

What it means for register pressure

This model increases pressure on the architectural register file because each lane needs its own live state until the quad completes.

• Varyings, temporary values, and texture coordinates must be kept live across stage boundaries.
• If all four lanes of a quad are resident at once, you need roughly four copies of the lane-local state, plus any per-quad state.
• More in-flight quads improve throughput, but they also increase total live register demand and reduce occupancy.

In practice this means:

• Fewer resident quads when shaders use many temporaries, large varyings, or many texture coordinates.
• More spilling if the register allocator cannot keep the per-lane state on-chip.
• Tighter coupling between compiler decisions and hardware occupancy.

What it means for in-pipe memories and accumulators

The "in-pipe" state for such a machine is usually not just one set of pipeline registers. You need small lane-local or quad-local memories for:

• interpolated varyings
• texture coordinates and gradients
• predicate masks and control-flow state
• partial arithmetic results
• depth/coverage/sample information

If the design is strictly in order, these buffers act like stage-local holding registers and scoreboards:

• a lane cannot advance until the data for that stage is ready
• a later quad can occupy the freed stage while the earlier quad waits on memory
• accumulators may be lane-private so each lane can resume with the correct partial state

So the main trade-off is straightforward:

• More buffering and more lane-private state makes the executor more flexible and better at latency hiding.
• Less buffering makes the machine smaller and easier to reason about, but it stalls more often on texture or memory latency.

For a browser simulator or research GPU, this is a useful point on the design space because it is simple enough to implement, but still captures the real cost of varyings, per-lane state, and pipeline residency.

4. Divergence and Coherence

SIMD/SIMT groups execute best when lanes follow similar control flow.

• If an if/else splits lanes, hardware serializes paths with lane masks.
• Divergence reduces effective throughput.
• Texture and branch coherence within nearby pixels is therefore highly valuable.

Vendors optimize this differently (compiler heuristics, scheduling, cache layout), but the fundamental cost model is shared.

5. Vendor Architectures: How Parallelism Is Extracted

NVIDIA (GeForce/RTX/Data Center)

• Core unit: SM with warp schedulers.
• Execution model: many warps resident; scheduler selects ready warp each cycle.
• Latency hiding: high warp-level concurrency + fast context switching.
• Graphics + compute overlap: supported via concurrent engines and async compute (resource dependent).
• Modern additions: features like shader execution reordering (in ray tracing contexts) improve coherence in divergent workloads.

AMD (RDNA/CDNA lineage for graphics/compute emphasis)

• Core unit: CU/WGP with wavefront execution.
• Wave size: commonly wave32 or wave64 depending on mode and target.
• Parallel extraction: many waves in flight per CU/WGP; scheduler alternates ready waves.
• Front-end evolution (for graphics): geometry/primitive handling improvements and culling to reduce downstream pixel cost.
• Async compute overlap is a strong design theme where dependencies permit.

Intel (Xe family)

• Core unit: Xe cores with vector/SIMD execution across EUs.
• Uses thread/subgroup scheduling to hide memory and texture latency.
• Strong reliance on compiler and driver scheduling choices for occupancy, register pressure, and cache behavior.

Apple (AGX, tile-based deferred rendering style)

• Tile-based architecture emphasis:
  • Scene is binned into tiles.
  • Many operations happen in on-chip tile memory before final resolve.
• Benefit:
  • High bandwidth efficiency and strong locality for mobile power envelopes.
• Parallelism:
  • Per-tile parallel processing + SIMD subgroup execution within shader cores.

Mobile vendors (ARM Mali, Qualcomm Adreno, Imagination)

• Generally emphasize tile-based or tile-friendly rendering designs.
• Parallelism combines:
  • Tile-level independence
  • Subgroup SIMD execution
  • Latency hiding through many in-flight threads
• Strong focus on bandwidth minimization and power-efficient scheduling.

Important note: exact scheduling and cache details are often proprietary and vary per generation.

6. Correct Compositing and Draw Order Under Massive Parallelism

Parallel execution does not mean random final order. Correctness is enforced by pipeline rules and fixed-function tests.

Opaque geometry path (common case)

1. Rasterization generates fragments.
2. Per-fragment tests (scissor, stencil, depth) determine visibility.
3. Passing fragments write color/depth.

Depth testing means only nearest visible surfaces survive (for standard less/greater depth modes), independent of internal execution timing.

Transparent/blended geometry path

With blending enabled, order often matters because blending is not generally commutative:

• Typical pipeline: draw transparent objects back-to-front.
• Hardware ensures API ordering for writes to the same render target region, subject to defined synchronization and hazards.

Thus, even though many shader invocations run in parallel, output merger/ROP logic applies operations in a manner consistent with graphics API ordering guarantees.

Early-Z / Late-Z

• Early-Z can reject fragments before fragment shader execution when legal.
• Late-Z may be required when shader side effects or depth writes constrain reordering.

This affects performance, not final correctness.

Tile-based compositing nuance

In tile-based GPUs:

• Fragments for a tile accumulate in tile-local memory.
• Depth/stencil/blending resolve occurs with strong local ordering rules.
• Final tile resolve writes to system memory.

This reduces bandwidth while preserving API-visible output semantics.

7. How Different Shader Programs Are Composited Correctly

When different shaders contribute to the same frame:

1. CPU/driver submits ordered command streams.
2. Hardware may overlap execution internally.
3. Synchronization points, render pass boundaries, and barriers enforce visibility/order constraints.
4. Output merge stage plus depth/stencil/blend rules define final per-pixel result.

So internal scheduling is opportunistic, but externally visible results follow API ordering and attachment rules.

8. Typical Ordering Tools Used by APIs/Drivers

Across modern APIs (Metal, Vulkan, D3D12):

1. Render pass boundaries
2. Resource state transitions
3. Memory barriers
4. Subpass dependencies (where applicable)
5. Queue synchronization primitives (fences/semaphores/events)

These constrain or permit overlap while preserving correctness.

9. Performance Implications for Shader Authors

To maximize parallel efficiency:

1. Keep control flow coherent across neighboring pixels.
2. Minimize random memory access and exploit texture locality.
3. Balance register usage to maintain occupancy.
4. Reduce unnecessary state changes and shader permutations.
5. Separate opaque and transparent passes with intentional ordering.
6. Use depth pre-pass or early depth strategies when beneficial.

10. Summary

Modern GPUs extract parallelism from graphics workloads by combining:

1. Many in-flight draws/primitives/fragments
2. SIMD/SIMT subgroup execution
3. Fast hardware scheduling to hide texture/memory latency
4. API- and pipeline-level ordering rules for correct final composition

Triangles and pixels do execute in massive parallel fashion, but final frame correctness is maintained by depth/stencil/blend/output-merge semantics and explicit synchronization constraints.

11. Appendix: Concrete Frame Example with Ordering and Barriers

This example shows a typical frame with opaque geometry, transparent geometry, and post-processing.

Example frame plan

1. Depth pre-pass (optional)
2. Opaque pass (G-buffer or forward)
3. Lighting pass (if deferred)
4. Transparent pass
5. Post-process pass
6. UI/composite pass

Step-by-step correctness model

Step 1: Depth pre-pass

• Draw opaque geometry writing depth only.
• Parallelism: high triangle/fragment parallelism; early depth-friendly.
• Ordering requirement: none across unrelated objects; depth test defines visibility.

Step 2: Opaque color/G-buffer pass

• Draw opaque materials (different shaders across many draws).
• Parallelism:
  • Draws and fragments run massively in parallel.
  • Different materials can overlap in hardware queues.
• Correctness:
  • Depth test ensures nearest surface wins per pixel.
  • Blend usually disabled for opaque outputs, so order is mostly irrelevant for final color where depth is correct.

Barrier A (if deferred path)

• Ensure G-buffer/depth writes are visible before lighting reads.
• API shape: render-pass boundary, subpass dependency, or explicit memory/resource barrier.

Step 3: Lighting/full-screen pass

• Full-screen shader reads G-buffer and depth, writes lit color target.
• Parallelism: pixel groups run in parallel; texture latency hidden by warp/wave scheduling.
• Correctness: requires Barrier A visibility guarantees.

Step 4: Transparent pass

• Sort transparent draw items roughly back-to-front.
• Draw with blending enabled against lit color target.
• Parallelism:
  • Fragment execution still highly parallel.
  • But final blend accumulation for overlapping pixels follows API draw order semantics.
• Correctness:
  • Back-to-front ordering is needed for standard alpha blending.
  • Depth test often enabled with depth writes disabled.

Barrier B

• Ensure transparent pass completion before post-processing reads color target.

Step 5: Post-process

• Tone map, bloom composite, color grade, etc.
• May involve multiple compute/graphics passes with intermediate textures.
• Correctness:
  • Each pass that reads prior pass outputs needs explicit ordering/visibility barriers.

Step 6: UI/composite

• Draw UI on top of final scene target.
• Typically last to preserve visual layering.

What can overlap safely

1. Vertex/front-end work for later draws can overlap while earlier draws are rasterizing.
2. Async compute (for example culling, SSR prep, particles) may overlap graphics if resources are disjoint or synchronized correctly.
3. Independent post-process branches can run concurrently if they read immutable inputs and write distinct outputs.

What must be ordered

1. Any pass that reads a resource written by a prior pass must wait for visibility via barriers/dependencies.
2. Transparent blending to the same target region must respect draw ordering policy.
3. Final presentation must wait until all writes to the presentable image are complete.

Minimal mental model

• Execution order inside the GPU is flexible and aggressively parallel.
• Visibility order at API boundaries is explicit and enforced.
• Final pixel order correctness comes from depth/stencil/blend rules plus synchronization points.