Parallelism Strategies and What They Cost the Network

A researcher tells you the run will use tensor parallelism of 8 and pipeline parallelism of 4. That sentence has already determined which GPUs must be adjacent, how much bandwidth you need, and what your step time floor will be.

---

The Problem at Scale

A parallelism plan arrives looking like a modeling decision and is in fact an infrastructure specification. Three integers, a tensor-parallel degree, a pipeline-parallel degree, and a data-parallel degree, multiply to the world size and simultaneously describe four different traffic patterns with four different message sizes, four different frequencies, and four different tolerances for latency. Read them correctly and you can size the fabric, constrain the placement, and predict the step time floor before a single GPU is allocated. Read them as "the researcher's business" and you will be handed a slow run and asked why your cluster is bad.

Take the plan in the scenario. Tensor parallel 8, pipeline parallel 4, and whatever data-parallel degree makes up the rest. On a 1,024-GPU allocation that is 8 times 4 times 32, so the data-parallel degree is 32. Those numbers say four things immediately.

Groups of eight GPUs will exchange activation tensors several times per transformer layer, in both directions, on every microbatch. That is hundreds of small collectives per step, and small collectives are latency-bound. Those eight GPUs have to be inside a single NVLink domain, which on current hardware means inside one node, and there is no configuration knob that makes it acceptable for them not to be.

Groups of four will pass activations stage to stage as point-to-point sends. The volume is small, the latency tolerance is moderate, and the real cost is not bandwidth at all but the pipeline bubble, which is pure idle time on every GPU in the group.

Thirty-two groups will allreduce the full gradient buffer once per step. That is the largest single transfer in the whole plan, it is bandwidth-bound rather than latency-bound, and it is the one dimension that can survive crossing an oversubscribed tier if it has to.

And nothing in the plan tells you any of this on its face. The plan is three integers. The traffic is what you have to derive.

The reason this matters more here than anywhere else in infrastructure work is that the derivation is cheap and the alternative is expensive. Getting the mapping wrong does not produce an error. It produces a run that is thirty or fifty percent slower than it should be, for six weeks, on ten thousand GPUs, and the slowdown is attributed to whatever else changed that week.

---

How It Works

Four dimensions, four traffic patterns

Data parallel replicates the model and splits the batch. Each replica computes gradients on its own microbatches, and at the end of the step every replica's gradients are averaged with an allreduce. One large collective per step, or in practice several, because frameworks bucket the gradients and fire each bucket's allreduce as soon as its parameters are ready, overlapping communication with the remaining backward pass.

The volume is fixed by the parameter count and is worth doing on paper. A 7-billion parameter model with bf16 gradients has a 14 GB gradient buffer. A ring allreduce moves 2(N-1)/N times the tensor size across each rank's links, so for any reasonably large ring that is close to 2 times 14, about 28 GB per rank per step. If your effective per-GPU scale-out bandwidth is 40 GB/s, that is 0.7 seconds of wire time. Against a two-second step, over a third of the step is communication unless it is overlapped. Substitute your own parameter count and link rate; the structure of the estimate does not change, and it is the number that tells you whether the plan is even feasible.

Tensor parallel splits individual layers across GPUs, so a single matrix multiply is computed in pieces and the pieces are recombined. That recombination is a collective inside the forward pass and again inside the backward pass, twice per transformer block in a typical arrangement. Messages are activation-sized rather than parameter-sized, which is to say small, and there are many of them. Small and frequent means latency-bound, and it means the collective is on the critical path with nothing to overlap it against.

Pipeline parallel splits the layer stack into stages and streams microbatches through it. The only traffic is a point-to-point send of one stage's output to the next stage's input, which is cheap. The cost is structural: at the start and end of every step, stages that have no microbatch to work on sit idle. With P stages and M microbatches, the classic bubble fraction is (P-1)/M. For P equals 4 and M equals 16, that is 3/16, about 19 percent of every GPU's time spent doing nothing. Raise M to 64 and the bubble falls to under 5 percent, which is why the microbatch count is a scheduling parameter you should care about.

Expert parallel, the MoE case, routes each token to a subset of experts that live on different GPUs. The pattern is an all-to-all: every rank sends a different slice to every other rank, twice per MoE layer, once to dispatch tokens and once to gather results. All-to-all is the most fabric-hostile collective there is. It cannot be reduced or trimmed by a smarter algorithm the way an allreduce can, and it lands directly on bisection bandwidth, so it is the one pattern where the oversubscription ratio between tiers shows up undisguised.

Sequence parallel is best understood as a companion to tensor parallel rather than a fifth dimension. It splits the operations that tensor parallel leaves replicated along the sequence axis, converting the tensor-parallel allreduce into a reduce-scatter plus an allgather. Same total volume, less activation memory, and the same absolute requirement to stay inside the NVLink domain.

Rank ordering is the placement

Frameworks map a flat global rank onto the parallelism coordinates in a fixed order, conventionally with tensor parallel varying fastest. With TP 8, PP 4, DP 32, global ranks 0 through 7 form one tensor-parallel group, ranks 8 through 15 the next, and so on. Ranks that are 8 apart are pipeline neighbors within the same data-parallel replica; ranks 256 apart are data-parallel peers.

Printing the mapping once, at startup, is the cheapest insurance in this lesson.

$ kubectl logs train-run-11-worker-0 | grep 'rank map' | head -5
rank map: global=0   node=gpu-node-0417 local=0  tp=0 pp=0 dp=0
rank map: global=7   node=gpu-node-0417 local=7  tp=7 pp=0 dp=0
rank map: global=8   node=gpu-node-0418 local=0  tp=0 pp=1 dp=0
rank map: global=255 node=gpu-node-0448 local=7  tp=7 pp=3 dp=7
rank map: global=256 node=gpu-node-0449 local=0  tp=0 pp=0 dp=8

Every tp group of eight shares a single node value. The moment a tp group spans two hostnames, you have the problem in the warning above, and you have it before step one rather than after week one.

That ordering is why local rank inside a node matters. If the launcher assigns ranks 0 through 7 to the eight GPUs of one node, the tensor-parallel group is exactly one NVLink domain and everything works. If a scheduler interleaves pods so that consecutive global ranks land on different nodes, the same configuration produces the disaster in the warning above, with no configuration change at all. The parallelism plan and the rank-to-node mapping have to be reasoned about together, because either one alone is meaningless.

---

Building and Operating It

The intake conversation with a researcher is short and should always cover the same four things: the three degrees, the microbatch count, whether the model has MoE layers, and the parameter count. From those you can compute the whole traffic profile.

$ torchrun --nnodes=128 --nproc-per-node=8 train.py \
    --tensor-model-parallel-size 8 \
    --pipeline-model-parallel-size 4 \
    --num-layers-per-virtual-pipeline-stage 2 \
    --micro-batch-size 1 --global-batch-size 2048

> world size 1024, data-parallel size 32
> using interleaved pipeline schedule, virtual stages 2

Then verify, on the running job, that each dimension landed where you expected. NCCL says exactly which transport each communicator chose, and that log line is the ground truth.

$ kubectl logs train-run-11-worker-0 | grep -E 'via (NVL|NET|P2P|SHM)' | sort -u
NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/direct pointer   # TP group, intra-node
NCCL INFO Channel 04/0 : 3[3] -> 11[3] via NET/IB/3            # DP group, rail 3
NCCL INFO Channel 08/0 : 3[3] -> 4[4] via NET/IB/3             # <-- TP crossing the node

That third line is the failure. Ranks 3 and 4 are in different tensor-parallel groups only if the degree is 4; with a degree of 8 they are in the same group and should never touch the network. One grep at job start catches a class of problem that otherwise costs the whole run.

Establish the bandwidth floor separately from the job, using nccl-tests, so you know what good looks like before anything is wrong.

$ mpirun -np 1024 -N 8 all_reduce_perf -b 512M -e 8G -f 2 -g 1
#       size         count    type  redop     time    algbw    busbw
#        (B)      (elements)                  (us)   (GB/s)   (GB/s)
   536870912       134217728   float   sum   6821.4    78.71   147.34
  2147483648       536870912   float   sum  27193.7    78.97   147.79
  8589934592      2147483648   float   sum 108530.2    79.15   148.13

Read busbw, not algbw. Algorithm bandwidth is message size divided by time, which understates a ring allreduce because each rank actually moves 2(N-1)/N times the buffer. Bus bandwidth applies that factor and is what you compare against the link rate. Here 147 GB/s of bus bandwidth on a node with eight 400 Gb/s NICs is roughly 37 percent of the 400 GB/s of aggregate NIC capacity, which for a large allreduce is a reasonable starting point on many fabrics and a number you should record and re-run weekly. The Fabric module goes deep on interpreting these results; the operational point here is that an unmeasured floor is not a floor.

For an MoE run, run the all-to-all benchmark too, because it is the pattern that exposes placement rather than link rate.

$ mpirun -np 512 -N 8 alltoall_perf -b 64M -e 1G -f 2 -g 1
#       size         count    type     time    algbw    busbw
#        (B)      (elements)            (us)   (GB/s)   (GB/s)
    67108864        16777216   float  3184.2    21.08    21.04
  1073741824       268435456   float 49871.6    21.53    21.49

# same benchmark, same 512 ranks, packed into one island
    67108864        16777216   float  1092.6    61.42    61.31

Same hardware, same job size, three times the throughput from placement alone. That gap is the oversubscription ratio made visible, and it is why an MoE plan and a fragmented cluster are a bad combination.

Finally, encode the constraint rather than hoping for it. A tensor-parallel group that must stay inside a node is expressed by making the pod the node: one pod, eight GPUs, eight local ranks. That removes the possibility of a scheduler splitting the group, because there is nothing left to split.

    - name: worker
      replicas: 128                # 128 pods x 8 GPUs = 1024, TP group = 1 pod
      template:
        spec:
          containers:
            - name: trainer
              resources:
                limits:
                  nvidia.com/gpu: 8      # whole node, no partial allocation

Pipeline and data-parallel constraints are looser and are expressed with topology keys rather than with pod shape, which is the subject of module 3.

---

Tradeoffs and Decision Framework

Dimension	Collective	Message size	Frequency	Required tier	What breaks if misplaced
Tensor parallel	Allreduce, or reduce-scatter plus allgather	Activation, small	Hundreds per step	NVLink, intra-node only	Step time doubles or worse, silently
Sequence parallel	Reduce-scatter and allgather	Activation, small	Hundreds per step	NVLink, intra-node only	Same as tensor parallel
Pipeline parallel	Point-to-point send and recv	Activation, small	Once per microbatch per stage	Same island, adjacent racks preferred	Bubble grows; latency adds to every microbatch
Expert parallel	All-to-all	Token-dependent, medium	Twice per MoE layer	Non-blocking island	Hits bisection directly; throughput collapses across tiers
Data parallel	Allreduce	Parameter-sized, large	Once per step, bucketed	Scale-out fabric; survives oversubscription	Communication stops hiding behind compute
Increasing microbatch count	None	None	None	None	Reduces bubble but raises activation memory

Four questions settle a plan. Does the tensor-parallel degree fit inside one node, since if it does not, nothing else you do will rescue the run? Does the whole job fit inside one island, and if not, which dimension is being pushed across the oversubscribed tier, because it must be the data-parallel one? Is the gradient allreduce volume small enough to hide behind the backward pass at the measured bus bandwidth, using the arithmetic above? And is the microbatch count high enough that the pipeline bubble is a rounding error rather than a line item?

The default: tensor parallel exactly equal to the GPUs per node and never more, pipeline parallel only as large as memory forces it to be, microbatches at least four times the pipeline depth, and every remaining GPU spent on data parallelism. Deviate when the model genuinely does not fit, not because a larger number looks more parallel.

---

Failure Modes and Common Mistakes

Setting the tensor-parallel degree above the GPUs per node. Latency-critical collectives move onto the NIC fabric, step time inflates dramatically, and nothing reports an error because the configuration is entirely legal.

Assuming consecutive global ranks are physically adjacent. The framework's group construction depends on it, and a launcher or scheduler that interleaves ranks across nodes silently destroys the tensor-parallel group.

Ordering local ranks by PCI enumeration instead of NVML index. The two agree on some hardware and not on others, so the bug appears only when you change generations and looks like a fabric problem.

Reading algorithm bandwidth as if it were bus bandwidth. Algorithm bandwidth ignores the 2(N-1)/N factor a ring actually moves, so it understates the fabric by roughly half and makes a healthy allreduce look broken.

Choosing a microbatch count without computing the bubble. Four stages with eight microbatches wastes about 27 percent of every GPU in the pipeline group, which is a larger loss than most fabric problems you will ever chase.

Placing an MoE run across islands. All-to-all cannot be reorganized to avoid the oversubscribed tier the way an allreduce can, so it converts the core's oversubscription ratio directly into step time.

Treating the parallelism plan as the researcher's private business. It determines placement, bandwidth, and failure blast radius, so it belongs in the intake conversation alongside GPU count and duration.

---