Why Training Infrastructure Is Not Inference Infrastructure

You have run inference platforms for years. Your first training cluster looks like the same hardware and the same Kubernetes, so you bring the same instincts, and every one of them is wrong.

---

The Problem at Scale

An inference cluster is a population. Requests arrive independently, the load balancer spreads them across whatever replicas are healthy, and losing one replica costs you that replica's share of capacity and nothing else. Every reflex you have as a platform engineer is built on that property. Horizontal autoscaling assumes capacity is fungible. Rolling updates assume replicas are interchangeable. A PodDisruptionBudget assumes that "sixty percent available" is a meaningful and useful state. Liveness probes assume a sick instance can be removed without consulting anything else in the system.

A training run has none of those properties. It is a single program whose address space happens to be spread across thousands of GPUs, and it advances in lockstep. Every rank executes the same step at the same time, and every step ends in a collective: an allreduce over gradients, an all-to-all over expert activations, a point-to-point handoff into the next pipeline stage. A collective is a barrier. It does not complete until every participating rank has arrived. There is no partial allreduce, no quorum, no degraded mode that averages over the ranks that showed up on time.

That single fact rewrites the operational model in three places.

Capacity is allocated once, not adjusted continuously. The world size is fixed when the job launches. It is baked into the parallelism plan, the optimizer sharding, the global batch size, and the learning rate schedule the researcher tuned against that batch size. You cannot add sixty-four GPUs at 2am because the queue looks busy, and you cannot remove them because another team filed a ticket. The allocation is the job.

The unit of availability is the job, not the replica. With 8,192 GPUs running and one dead, you do not have 99.99 percent of your throughput. You have zero. The run is stopped, and it stays stopped until something replaces the missing rank and the whole process group is rebuilt. Availability for this workload is binary in a way that no serving system ever is.

Recovery is measured in lost work, not in lost seconds. A serving pod that crashes is replaced in the time it takes to pull an image and pass a readiness probe. A training job that dies restarts from its last checkpoint, which means every step computed since that checkpoint is discarded and recomputed. If you checkpoint every thirty minutes and die twenty-nine minutes in, you burn the wall clock of 8,192 GPUs twice over the same twenty-nine minutes. That is 8,192 times 29 divided by 60, roughly 3,960 GPU-hours, deleted. Substitute your own fleet size and interval; the shape of the arithmetic does not change.

This course assumes the unit of work is a job that occupies ten thousand GPUs for six weeks, cannot degrade gracefully, and dies if any one participant dies. The serving side is covered elsewhere: the NVIDIA device plugin, the GPU operator, MIG, and DCGM setup belong to the Production GPU Infrastructure course, and everything about request-time behavior belongs to the LLM Inference on Kubernetes course. This course stops at the trained checkpoint.

---

How It Works

The step is a distributed barrier

A training step on a data-parallel job runs forward, runs backward, and reduces gradients across every rank before the optimizer can apply an update. Modern frameworks overlap that reduction with the backward pass: gradients are bucketed, and each bucket's allreduce is launched as soon as its parameters have gradients, so communication hides behind the remaining compute. That overlap is why a well-tuned job spends most of its step doing math even though it moves gigabytes per step.

It also means the barrier is not at the end of the step. It is smeared across the second half of the backward pass, and every rank must keep pace with every other rank through all of it.

$ torchrun \
    --nnodes=128 --nproc-per-node=8 \
    --rdzv-backend=c10d --rdzv-endpoint=rdzv-0.train-run-7.svc:29500 \
    --rdzv-id=train-run-7 \
    train.py --global-batch-size 2048 --seq-len 8192

[rank0]: Rendezvous complete: world_size=1024, local_rank=0
[rank0]: Initialized ProcessGroupNCCL, timeout=0:10:00
[rank0]: step 1     loss 11.402  step_time 0.612s
[rank0]: step 2     loss 11.108  step_time 0.418s
[rank0]: step 3     loss 10.947  step_time 0.417s

The consequence that surprises people coming from serving is statistical. Job step time is the maximum over ranks, not the mean, and maxima behave very differently from means as you add participants. Suppose each rank independently has a one percent chance in any given step of taking an extra 100 ms because of a page fault, a data loader hiccup, or a clock dip. On one rank, ninety-nine percent of steps are clean. On 1,024 ranks, the probability that no rank hiccups is 0.99 to the power of 1,024, which is roughly 0.00003. Essentially every step at that scale is a slow step, because you are sampling the tail 1,024 times per step. This is why tail-latency thinking, which serving engineers already have, transfers directly, and why average-case thinking does not.

Failure is silence, not an error

NCCL has no failure detector. When a rank dies, its peers do not receive a reset or an error. They post their next collective, wait for a participant that will never arrive, and block. The only thing that eventually breaks the deadlock is a timeout, and in PyTorch that is the ProcessGroupNCCL watchdog, which defaults to ten minutes.

So the failure signature of a dead rank is ten minutes of nothing, followed by an abort storm.

$ kubectl logs train-run-7-worker-43 -c trainer | tail -6
[rank344]:[E ProcessGroupNCCL.cpp:563] [Rank 344] Watchdog caught collective
  operation timeout: WorkNCCL(SeqNum=88214, OpType=ALLREDUCE,
  NumelIn=268435456, NumelOut=268435456, Timeout(ms)=600000)
[rank344]:[E ProcessGroupNCCL.cpp:1182] To avoid data inconsistency, we are
  taking the entire process down.

Read that carefully, because it is the single most misread log line in distributed training. Rank 344 is not broken. Rank 344 is a healthy rank complaining that somebody else never showed up. The loudest logs in a collective failure come from innocent ranks; the guilty rank is usually the one that stopped logging. The first move in triage is not to read the errors, it is to find the rank whose log went quiet earliest and look at the node underneath it.

$ for p in $(kubectl get pods -n train -l job-id=train-run-7 -o name); do
    echo "$(kubectl logs -n train $p --tail=1 | cut -c1-24) $p"
  done | sort | head -3

2026-03-11T03:41:07.118Z pod/train-run-7-worker-100
2026-03-11T03:51:19.402Z pod/train-run-7-worker-43
2026-03-11T03:51:19.407Z pod/train-run-7-worker-44

Worker 100 stopped ten minutes before everyone else, which is exactly the watchdog interval. It is the cause; the other 1,023 are witnesses.

---

Building and Operating It

Start by deleting things. The training pod spec should have no liveness probe, no HPA, no rolling update strategy, and no PodDisruptionBudget that pretends partial availability is useful. What it should have is a restart policy that fails fast to the job controller, a termination grace period long enough for a checkpoint flush, and a shared-memory allocation big enough for the data loader.

spec:
  restartPolicy: Never          # a rank restart in place is never correct
  terminationGracePeriodSeconds: 300   # long enough to flush a shard write
  containers:
    - name: trainer
      resources:
        limits:
          nvidia.com/gpu: 8
          rdma/ib: 8
      env:
        - name: NCCL_ASYNC_ERROR_HANDLING
          value: "1"
        - name: TORCH_NCCL_DUMP_ON_TIMEOUT
          value: "1"
      volumeMounts:
        - { name: dshm, mountPath: /dev/shm }

restartPolicy: Never is the important line. Restarting a single rank container in place is never the right answer, because the surviving ranks are already in a process group that the restarted rank cannot join. You want the failure to propagate up to whatever owns the gang, so the entire job is torn down and rescheduled together. TORCH_NCCL_DUMP_ON_TIMEOUT is worth turning on everywhere: it writes a per-rank record of the last collectives each rank issued, which is how you find out that rank 344 was waiting on sequence number 88214 while rank 800 never got past 88213.

The second change is to node maintenance. On a serving cluster, draining a node is routine and safe. On a training cluster it is an outage.

$ kubectl drain gpu-node-0417 --ignore-daemonsets --delete-emptydir-data
node/gpu-node-0417 cordoned
evicting pod train/train-run-7-worker-43
pod/train-run-7-worker-43 evicted
node/gpu-node-0417 drained

Four lines, and you just deleted every step since the last checkpoint on 1,024 GPUs. The habit that replaces it is cordon-first: mark the node unschedulable so nothing new lands on it, record it in whatever holds your replacement queue, and evict only at a checkpoint boundary. That requires knowing which nodes currently hold a gang, which is a reason to label them at admission time rather than trying to reconstruct it under pressure.

The third change is alerting. Pod restart counts, replica availability, and request error rates are the wrong signals here, and a training job will happily satisfy all of them while producing nothing. The signals that matter are step time and time since the last checkpoint, both emitted by the training process itself.

# Step time has regressed more than 20 percent against the last hour
(
  avg_over_time(training_step_seconds{job_id="train-run-7"}[10m])
  /
  avg_over_time(training_step_seconds{job_id="train-run-7"}[1h] offset 1h)
) > 1.2

# The job is alive but has not completed a step in five minutes: a stall
time() - training_last_step_timestamp_seconds{job_id="train-run-7"} > 300

That second query is the one that pages. It fires at five minutes, which is half the watchdog timeout, so a human is looking at logs while the ranks are still blocked and the evidence is still live rather than after everything has aborted and the pods have been garbage collected.

---

Tradeoffs and Decision Framework

Serving instinct	What it does to a training job	What to do instead
Horizontal autoscaling on utilization	Nothing useful; world size is fixed at launch and cannot change mid-run	Fixed gang allocation, sized by the parallelism plan
Liveness probe restarts a stuck container	Turns a stall into a certain job kill and hides the real culprit	No liveness probe; watch step time at the job level
Rolling update, one replica at a time	There is no such thing as a partially updated job	Drain to a checkpoint, restart the whole gang on the new image
PodDisruptionBudget with maxUnavailable	Grants permission to delete a rank, which is never acceptable mid-step	Checkpoint-boundary eviction, gated on the run itself
Drain the node on a hardware warning	Kills the run for a fault that may not be affecting it	Cordon, queue the node for replacement, evict at a checkpoint
Alert on pod restarts and error rate	Silent on the failure that matters: a job that is running and stalled	Alert on step time regression and time since last step

Three questions settle almost every decision on a training cluster. First, does this action stop the job, and if it does, how many GPU-hours of recomputation does it cost at the current checkpoint interval? Second, is the fault I am reacting to actually degrading the run right now, or is it a warning about future risk that can wait for the next boundary? Third, if I do nothing for one more checkpoint interval, does the situation get worse, or does it just stay the same?

The default: never take an action that stops a running job unless the job is already stopped, or the fault is actively degrading step time. Warnings, correctable errors, pending firmware updates, and capacity rebalancing all wait for a checkpoint boundary. Uncorrectable errors, thermal throttling, and a link that has dropped to a degraded width do not wait, because they are costing you throughput every step.

---

Failure Modes and Common Mistakes

Treating the loudest rank as the broken one. Every healthy rank logs a watchdog timeout, so the error log is dominated by innocent parties. Sort worker logs by last-write timestamp and investigate the one that went quiet first.

Trusting GPU utilization as a liveness signal. A GPU blocked in a NCCL kernel reports near 100 percent utilization, so a completely stalled job looks like a perfectly healthy one on the dashboard everyone checks first.

Leaving the watchdog timeout at ten minutes on a large fleet. Ten minutes of 8,192 idle GPUs per incident is over 1,300 GPU-hours. Shorten it to match your straggler tolerance, and page well before it fires.

Applying the serving drain runbook. Node maintenance that is routine and invisible on a serving cluster is a full job restart here, and it will be attributed to whatever the researcher changed that day rather than to the drain.

Setting restartPolicy to OnFailure on training pods. A rank that restarts in place cannot rejoin the existing process group, so you get a container that comes up healthy inside a job that is already dead, which delays diagnosis by however long it takes someone to notice.

Sizing capacity by average utilization. A fleet that averages seventy percent utilization can still be unable to place a 512-GPU job, because the free GPUs are scattered across racks in ones and twos. Placement, not the aggregate, decides whether the job runs.

Assuming the checkpoint interval is a storage decision. It is a failure-rate decision, and it sets the expected cost of every incident on this list. Module 5 derives the interval properly, but the operational consequence starts here.

---