ddp-crossover

Figure 1

Speedup, and the line it has to beat

Anything below the horizontal line means one GPU was faster. The bars behind the curve are the fraction of each step spent moving gradients rather than computing them.

2x Tesla T4, PCIe, no NVLink batch 256, throughput against a single GPU at the same width

parameters: -
one GPU: -
two GPUs: -
speedup: -
step spent on comms: -

Why it goes the wrong way

Gradient bytes grow with parameter count and so does arithmetic, so the ratio between them is roughly fixed. On a fast link that ratio is small enough not to matter. On this one, gradient sync costs about 5x the compute it buys at every width measured.

The models that appear to scale are the ones too small to be doing real work, where the second GPU is parallelizing launch overhead rather than arithmetic.

the usual advice: "Add a GPU, get about 2x."
on this link: 4.05 GB/sAbout a 175th of each GPU's own memory bandwidth.
so the rule flips: It pays below roughly 8M parameters.And costs you above.

Figure 2

Which mitigations actually move it

Four standard answers to "my DDP is slow", measured at every width rather than argued about. Each is shown at its own best setting, since the question is whether it can help at all.

Only gradient accumulation rescues it. At x8 it turns every losing width into a win, because syncing once per eight micro-batches divides communication by eight: the share of the step spent on comms falls from 65% to 15%. It is also the bluntest instrument here, multiplying the effective batch by the same eight, which changes the optimization problem rather than only its execution.

fp16 compression is necessary and insufficient. Halving the bytes on the wire buys about 1.3x and still lands below one GPU at the sizes where it matters.

bucket_cap_mb, the knob most often suggested, does nothing: across a hundredfold sweep, throughput at 537M parameters spans 0.3%. And FSDP is worse than plain DDP at every size, because it adds an all-gather of parameters on top of the gradient reduce-scatter, and all-gather runs at 1.73 GB/s here against all-reduce's 4.05.

the fix: Gradient accumulation, x81.80x at 8.5M params, but a bigger effective batch.
helps, not enough: fp16 gradientsAbout 1.3x, still under one GPU where it counts.
does nothing: bucket_cap_mb0.3% across a hundredfold sweep at 537M.
actively worse: FSDP33 to 35% below plain DDP at every size.

Figure 3

Where it loses

This is a statement about an interconnect, not about DDP. On NVLink or InfiniBand the ratio that drives all of this changes, and the usual advice is probably right again. 01_nccl_bench.py
Two GPUs is the smallest interesting case. Whether the crossover moves the same way at four or eight is not measured here. 02_ddp_sweep.py
One model family, one batch size. A wide MLP at batch 256, chosen because it makes the gradient-to-arithmetic ratio easy to vary cleanly, not because it is what anyone trains. 02_ddp_sweep.py
The small hardware is the point, and also the limit. A pair of T4s is the regime where interconnect cost is large enough to invert the assumption, which is what makes it visible and also what makes it specific. README.md