lgoyal6 / decode-anatomy

decode‑anatomy.

"Decode is memory-bound" is the most repeated claim in LLM inference and almost nobody attaches a number to it. Memory-bound against which ceiling, on which card, at what batch size? The claim is usually made about a datacentre GPU in FP8 and then applied to whatever hardware is on hand.

Both ceilings measured on the card itself, then a real decode step decomposed against them. At batch 1 the answer is neither: the largest slice is not kernel work at all, and it disappears the moment you batch.

removed by a CUDA graph
38.3% at batch 1
the same, at batch 32
3.0% it batches away
compute-bound share
0.0% at every point measured
measured ceilings
711 GB/s, 93% of spec
Figure 1

The two ceilings, measured on the card

Every verdict further down is relative to these two lines, so they are measured on this card rather than read off a datasheet. The card does not reach its own spec sheet in either direction.

NVIDIA RTX A6000 32 GEMMs against the measured roofs
peak bandwidth
-
of quoted
-
peak bf16
-
of quoted
-
ridge point
-

Why measure the roofs at all

A roofline drawn from the datasheet puts the ridge point in the wrong place, and every verdict downstream moves with it. Measured here the ridge sits at 180.7 FLOP per byte. Anything to the left of it cannot be compute-bound however much arithmetic it does, which is what makes the compute-bound share in Figure 3 meaningful rather than assumed.

take away
Spec numbers are not ceilings.711 GB/s against a quoted 768.
what it sets up
Every share in Figure 3.Filed against these lines, not the datasheet.
Figure 2

How much of the step is just launching kernels

A CUDA graph replays the same work with the launches removed, so what it saves was never computation. Nearly 1,500 kernels go into one decode step.

Qwen2.5-7B-Instruct on an RTX A6000 eager against the same step captured as a graph
Batch
eager step
-
as a graph
-
removable
-
speedup
-
kernels per step
-

Why not just profile it

Because the profiler inflates the very thing being measured. Instrumenting launches makes launches expensive, and at batch 1 it runs 1.83x slower than clean wall time. So the headline comes from the graph capture, where two real executions are compared and neither is being watched.

take away
Launch overhead is a batch-1 problem.38.3% at batch 1, 3.0% at batch 32.
if you serve at batch 1
Capture a CUDA graph.1.62x, no kernel changes.
Figure 3

Three verdicts, and the one that is always zero

Every kernel gets filed against the measured ceilings: near the memory roof, near the compute roof, or far below both, which is latency-bound and not the same thing as memory-bound. Non-kernel time is what is left over.

share of the profiled step batch 1, 512 tokens of context
Batch
Context
memory-bound
-
compute-bound
-
latency-bound
-
non-kernel
-
throughput
-

Compute-bound is 0.0% at all 28 points. The compute ceiling is measured and never binds, which is the half of the folk claim that survives. The half that does not: calling everything else memory-bound hides that most of the step is below both ceilings, or outside the kernels.

These shares are profiled, so they carry the 1.83x inflation and the non-kernel slice reads larger than Figure 2 measures. Read the shape across batch and context, not the height.

the claim
"Decode is memory-bound."
what holds
Compute never binds.0.0% at every point.
what does not
Memory is not the biggest slice.35.6% memory against 59.4% outside the kernels.
Figure 4

What batching changes, and what it does not

Nine serving configurations over the same 200 prompts, greedy, one seed. Prefix caching and batching move bytes: up to 15 of 200 completions come back textually different. What is inside them almost never moves, and the score never moves at all.

Qwen2.5-7B-Instruct, 200 prompts, 512 max tokens completions that changed, out of 200
worst byte drift
-
worst answer drift
-
accuracy spread
-
one engine, repeated
-
earliest median divergence
-

Accuracy is 88.5% in all nine conditions. Not close to equal, equal. So byte-identical output is the wrong thing to gate a serving change on: it fails under exactly the settings you turn on for throughput, while the thing you actually care about is untouched.

The divergence is real rather than sampling noise. The same engine run five times over is byte-identical every time, so what moves the bytes is the batch a request lands in, not the decode.

gate on this
Answer equality.At worst 2 of 200 move.
not on this
Byte equality.Up to 15 of 200 move, and it costs you nothing.
Figure 5

Where it loses