Anatomy of a Ten Thousand GPU Cluster

Someone hands you a diagram of a new cluster: 1,250 nodes, eight GPUs each, two network tiers. Before you schedule anything on it you need to know which failures take out eight GPUs and which take out eight hundred.

The Problem at Scale

The diagram you were handed describes one hierarchy: the network. A real cluster has at least four, and they do not line up.

There is the network hierarchy, which decides how fast two GPUs can talk. There is the power hierarchy, which decides which machines go dark together. There is the cooling hierarchy, which decides which machines throttle together. And there is the management hierarchy, the boot servers, the image registry, the storage mounts, and the Kubernetes control plane, which decides which machines fail to come back together. A node belongs to one of each, and the boundaries cross. Two nodes on the same power distribution unit can sit on different network leaves. Two nodes under the same leaf switch can draw from different PDUs and different cooling loops.

That misalignment is the whole subject of this lesson, because every scheduling and reliability decision you make later depends on knowing, for each shared component in the building, exactly how many GPUs go away when it fails. Call that number the component's blast radius. A cluster you cannot describe in terms of blast radii is a cluster you cannot place a job on safely, and you will not find out until a job that should have survived a single failure did not.

The scale is what makes this unavoidable. At 1,250 nodes and eight GPUs each you have 10,000 GPUs, but you also have roughly 10,000 optical transceivers on the scale-out fabric alone, a few hundred switches, several hundred PDUs, and a cooling plant with its own single points of failure. Components with a mean time between failures measured in decades still fail constantly when you own tens of thousands of them. The arithmetic is unforgiving: 10,000 transceivers with a five-year MTBF produce, on average, about 10,000 divided by (5 times 365), roughly five and a half failures a day. Most of those are on links that are not carrying your job right now. Some are not.

The instinct you bring from general Kubernetes work makes this worse rather than better. Spreading replicas across failure domains is correct for a stateless service, because it converts a correlated failure into a partial one. For a training job it converts a fast job into a slow one and buys nothing, because a partial failure and a total failure have the same consequence: the job stops. You do not want your ranks spread across failure domains. You want them packed as tightly as the fabric allows, and you want to know exactly what that packing costs you when the domain fails.

How It Works

The tiers, and what each one costs

Inside a node, eight GPUs are connected by NVLink through NVSwitch, giving every pair full bandwidth without going near the PCIe bus or a NIC. Current-generation parts land in the high hundreds of GB/s per GPU, bidirectional. Outside the node, each GPU is served by a scale-out NIC in the 200 to 400 Gb/s class, which is 25 to 50 GB/s. The intra-node and inter-node fabrics are roughly an order of magnitude apart in bandwidth, and that ratio is the single most important number in the building. It is why tensor parallelism belongs inside a node and data parallelism belongs across nodes, which the next lessons build on directly.

The same command that maps GPU-to-GPU links also maps GPU-to-NIC affinity, and on a rail-optimized node the pairing is one to one.

$ nvidia-smi topo -m
        GPU0  GPU1  GPU2  GPU3  NIC0  NIC1  NIC2  NIC3  CPU Affinity  NUMA
GPU0     X   NV18  NV18  NV18   PIX   SYS   SYS   SYS   0-51          0
GPU1   NV18    X   NV18  NV18   SYS   PIX   SYS   SYS   0-51          0
GPU2   NV18  NV18    X   NV18   SYS   SYS   PIX   SYS   0-51          0
GPU3   NV18  NV18  NV18    X    SYS   SYS   SYS   PIX   0-51          0

Legend: PIX = same PCIe switch, SYS = traverses the host bridge

Every GPU has exactly one NIC at PIX, on its own PCIe switch. That NIC is the GPU's rail. If a job's traffic leaves GPU 2 through NIC 0 because something in the placement or the NCCL configuration is wrong, it crosses the host bridge on the way out and lands on the wrong leaf on the way in, which costs bandwidth twice. The Production GPU Infrastructure course covers reading this matrix in depth; what matters here is that the rail assignment is a physical property of the node, not a routing choice.

Above the node, the organizing idea in modern training clusters is the rail. In a rail-optimized topology, the eight NICs in a node do not share a switch. NIC 0 on every node in the island connects to leaf switch 0, NIC 1 to leaf switch 1, and so on. The result is that GPU 3 on any node can reach GPU 3 on any other node in the island with exactly one switch hop, never touching the spine. Because collectives in a well-placed job are organized so that rank i on each node talks to rank i on the others, most of the traffic never leaves its rail.

Racks are then a physical grouping that cuts across rails: a rack holds some number of nodes, typically four to eight for air-cooled designs and more for liquid-cooled ones, and each of those nodes has a NIC on every rail. An island (some vendors say pod, or scalable unit) is the set of racks that share a spine group and is non-blocking internally. Between islands you traverse a higher tier, which is where oversubscription lives.

Oversubscription, and the ratio you actually get

Oversubscription at a switch tier is uplink capacity divided into downlink capacity, and it is decided by how the ports on each switch are split. Take a 64-port leaf. Wire 32 ports down to nodes and 32 up to the spine and you have a 1:1, non-blocking tier. Wire 48 down and 16 up and you have 48 divided by 16, a 3:1 tier: if every attached node tries to send at line rate simultaneously, each gets a third of it.

The number in the design document is a floor, not a promise, and this is the part that catches people. The oversubscription your job experiences is the one you create by placing it badly, not the one printed on the diagram. A 512-GPU job placed entirely inside one island rides a 1:1 fabric. The same job spread over four islands puts three quarters of its allreduce traffic across the 3:1 core, and since a ring allreduce moves 2(N-1)/N times the tensor size per rank on both the reduce-scatter and allgather halves, that traffic is not incidental. It is the step. The job does not fail; it just runs at a fraction of the throughput and nobody can explain why, because the hardware is identical to the run last week that was fast.

Read the actual link state rather than the diagram. Every port has a negotiated width and rate, and links that have silently degraded from 4x to 1x are common enough that checking is part of the job.

$ ibstat mlx5_3
CA 'mlx5_3'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 1183
                Link layer: InfiniBand

$ ibdiagnet --pc --get_phy_info -o /tmp/ibdiag
-I- Scanning the fabric
-I- Discovered 1264 nodes, 4128 ports
-W- Link width degraded: switch lid 214 port 17 -> 1x (expected 4x)
-W- Symbol error counter above threshold on 3 ports
-I- Fabric summary written to /tmp/ibdiag/ibdiagnet2.db_csv

A single link that renegotiated to 1x is a quarter of the bandwidth on that path. It will not fail any health check, it will not raise an alert on any dashboard built for servers, and it will slow one rank, which slows the job. Isolating that link across thousands of ports is the subject of a later lesson in the Fabric module; recognizing that the fabric has a per-link health state, separate from up and down, starts here.

Building and Operating It

You cannot schedule against a topology that only exists in a PDF. The operational task is to get the four hierarchies into node labels, and to keep them accurate as hardware moves.

Start from the fabric itself. ibnetdiscover walks the subnet and gives you the switch each host port is attached to, which is the ground truth for rail and island membership.

$ ibnetdiscover | grep -A1 'gpu-node-0417 HCA-3'
Ca      1 "H-b8599f0300f4e2c1"          # "gpu-node-0417 mlx5_3"
[1](b8599f0300f4e2c1)  "S-b8599f0300a1b204"[17]  # lid 1183 lmc 0 "leaf-i02-r03" lid 214 4xNDR

That one line tells you the node is on leaf leaf-i02-r03, which by the naming scheme is island 02, rail 03. Turn that into labels and the scheduler can finally reason about it.

$ kubectl label node gpu-node-0417 \
    topology.train/island=i02 \
    topology.train/rack=r14 \
    topology.train/power-domain=pdu-r14-a \
    topology.train/cooling-loop=cdu-3
node/gpu-node-0417 labeled

$ kubectl get nodes -L topology.train/island,topology.train/rack,topology.train/power-domain
NAME             STATUS   ISLAND   RACK   POWER-DOMAIN
gpu-node-0417    Ready    i02      r14    pdu-r14-a
gpu-node-0418    Ready    i02      r14    pdu-r14-a
gpu-node-0419    Ready    i02      r14    pdu-r14-b

Note the third node. Same rack, same island, different PDU. If you had assumed rack equals power domain you would have got that wrong, and the assumption would have been invisible until a PDU tripped and took a set of nodes that made no sense against your model.

The label set is worth arguing about once and then freezing, because everything downstream keys off it: the topology constraints in module 3, the hot spare pools in module 4, the power budgets in module 7. The minimum viable set is island, rack, power domain, and cooling loop. Rail does not need a node label, because every node is on every rail; what you need per rail is a health state, which lives in your monitoring system rather than in the API server.

Then do the blast radius audit. For every shared component, write down how many GPUs it takes with it, and check the answer against the labels rather than against the drawing.

$ kubectl get nodes -l topology.train/power-domain=pdu-r14-a \
    -o jsonpath='{range .items[*]}{.status.capacity.nvidia\.com/gpu}{"\n"}{end}' \
  | paste -sd+ | bc
32

Thirty-two GPUs per PDU, four nodes. If your smallest production job is 64 GPUs, then no PDU fault can ever be survivable in place, and every one is a job restart. That is a fact worth knowing before an electrician asks whether they can work on a panel this afternoon.

Finally, keep the management hierarchy in view. The boot servers, the container registry, the shared filesystem mounts, and the rendezvous endpoint are shared by every node in the cluster. Their blast radius is 10,000 GPUs, and they are the cheapest components in the building. A registry that cannot serve 1,250 simultaneous image pulls turns every full-cluster restart into a thundering herd, which is a failure mode that only ever appears when you are already recovering from something else.

Tradeoffs and Decision Framework

Four questions decide placement policy for a given job. How many GPUs does it need relative to an island, since a job that fits in one island should never be allowed to leave one? What is the largest single blast radius it will be exposed to, and is that acceptable at the current checkpoint interval? Are the failure domains it touches independent, or does one PDU or one cooling loop cover most of the allocation? And is the fleet's free capacity actually placeable, or is it fragmented into pieces too small for the job even though the aggregate looks fine?

The default: pack every job into the smallest number of islands that can hold it, prefer contiguous racks within the island, and accept the concentrated power and cooling blast radius that packing creates. You buy back the risk with checkpoint frequency and standby capacity, both of which are cheaper than paying an oversubscription tax on every step for six weeks.

Failure Modes and Common Mistakes

Assuming rack equals failure domain. Racks usually contain more than one power domain and, in rail-optimized designs, no shared network component at all. Derive the domains from the actual wiring and label them separately.

Using pod anti-affinity to spread a training job. It is the correct reflex for a stateless service and the wrong one here: it increases the number of independent failure domains the job depends on while pushing collectives onto slower tiers.

Trusting the oversubscription ratio in the design document. That number describes the fabric at full load with perfect placement. The ratio your job experiences is set by where its ranks landed.

Treating a leaf switch as a redundant component that needs no monitoring. Losing a rail degrades every job in the island without making anything unhealthy, so it is invisible to node-level and pod-level alerting.

Deriving topology labels from hostnames. Naming conventions drift the moment hardware is moved or a rack is repurposed, and the resulting bad placement is nearly impossible to diagnose because every component reports healthy.

Ignoring link width and error counters. A port that renegotiated from 4x to 1x is up, passes every check, and slows one rank enough to slow the entire job. Poll ibdiagnet or perfquery on a schedule and alert on the delta.

Forgetting that the management plane has the largest blast radius of all. Image pulls, mounts, and rendezvous are shared by every node, and they only fail under the load of a full-cluster restart, which is precisely when you need them.