What Actually Limits a GPU Cluster

Someone asks how many GPUs you can put in one cluster. The published node limit is not the answer, and neither is the hardware. The real ceiling is somewhere else entirely.

---

The Problem at Scale

The published Kubernetes scalability envelope talks about thousands of nodes and a hundred thousand-odd pods. Every one of those numbers was measured on a conformance cluster running synthetic pods with no device plugin, no gang scheduler, no topology constraints, no operators, and no workload that creates a thousand pods in the same second. Your cluster has all of those. The published limits describe a cluster that does not resemble yours, so treating them as a budget is how teams end up surprised at a third of the number.

A GPU fleet also has an unusual object profile, and the shape of that profile is what decides which ceiling you hit first. Ten thousand GPUs is roughly 1,250 nodes at eight GPUs each, which is a small cluster by node count. Those nodes run very few pods: a training node often runs exactly one workload pod plus a handful of DaemonSets. So on the two dimensions that scalability documents are written against, node count and pod count, you look comfortable. Then you hit a wall anyway, because the dimensions that actually bind are ones nobody publishes a number for.

The two workload classes push on different ones, which is the part worth internalizing early.

Training stresses the scheduler and the burst. A large run creates hundreds of pods in one instant, every one of them carrying a gang membership and a topology constraint, and every one of them requiring the scheduler to reason about a set rather than a pod. The steady-state object count is trivial. The instantaneous decision load is not, and a full-fleet restart after an incident is the worst version of it.

Serving stresses churn and address space. Replicas are created and destroyed continuously, each one taking a pod IP, generating EndpointSlice writes, updating a Deployment status, and producing readiness transitions that fan out to every watcher. The instantaneous decision load is small. The sustained write rate into etcd is not.

Both of them land on a control plane that also has to serve every operator you installed. Each operator maintains informers, and an informer is a watch plus a full initial list. The API server load of a GPU fleet scales with nodes multiplied by controllers, not with nodes, and a GPU cluster accumulates controllers faster than any other kind: a device plugin, a GPU operator, a DCGM exporter, a network device plugin, a gang scheduler, a quota controller, a CSI driver, a policy engine, and whatever the research platform team shipped last quarter.

---

How It Works

The ceilings, in the order they usually arrive

The ordering is not universal, but it is common, and the first entry is the one people never count as a limit. A fleet can be at sixty percent allocation and still be unable to place anything large, because the sixty percent that is free is distributed as single GPUs across four hundred nodes. Capacity that cannot be placed is not capacity, and the number worth tracking is the largest allocation the fleet could satisfy right now, not the sum of what is unused.

Why the arithmetic is different on a GPU fleet

Work the numbers rather than trusting intuition, because each one substitutes cleanly for your own fleet.

Take the pod address space first, because it is the ceiling that produces the sharpest failure. Most CNI configurations allocate a fixed pod CIDR per node, and /24 is a very common default because it gives 254 usable addresses.

$ kubectl get node gpu-h100-0141 -o jsonpath='{.spec.podCIDR}{"\n"}'
10.244.7.0/24

$ kubectl cluster-info dump 2>/dev/null | grep -m1 cluster-cidr
    "--cluster-cidr=10.244.0.0/16",

A /16 split into /24 blocks yields 256 blocks, so this cluster holds 256 nodes. At eight GPUs per node that is 2,048 GPUs, and the fleet plan says ten thousand. The 257th node joins the cluster, never gets a CIDR, and stays NotReady with a CNI error in a log nobody reads. Nothing about that failure mentions GPUs, and re-numbering a live cluster is not a maintenance window, it is a migration.

Now the etcd side. A GPU node object is unusually large, because it carries the device plugin's resource entries, a long list of topology and capability labels, the GPU operator's annotations, and a list of every image on the node.

$ kubectl get node gpu-h100-0141 -o json | wc -c
41207

$ kubectl get --raw /metrics | grep -E '^etcd_db_total_size_in_bytes' 
etcd_db_total_size_in_bytes{endpoint="https://10.0.1.11:2379"} 3.284373504e+09

Forty kilobytes per node times 1,250 nodes is about 50 MB of node objects, which is not alarming on its own. The problem is that every kubelet status update rewrites the whole object, and each rewrite is a new etcd revision retained until compaction. etcd grows from write rate, not from object count, and node status is the highest-frequency large write in a GPU cluster. Add a serving tier churning pods and EndpointSlices and the write rate is the number to watch. The etcd Operations course owns the tuning; what you own here is knowing which of your objects are big and which are hot.

Then the scheduler. A default bind against a permissive pod is inexpensive, and published throughput figures are measured that way. Every constraint you add moves the cost, and gang scheduling changes the shape of the work entirely: instead of one decision per pod, the scheduler simulates an entire set against a snapshot and discards the simulation if the set does not fit.

# Time to make one scheduling decision, and the depth behind it
histogram_quantile(0.99,
  rate(scheduler_scheduling_attempt_duration_seconds_bucket[5m]))

scheduler_pending_pods{queue="unschedulable"}

If a p99 decision takes 200 ms, the scheduler makes about five decisions per second, and a 1,250-pod full-fleet restart takes four minutes of pure scheduling before the first container starts. That is survivable. Let the same constraint set push p99 to two seconds and it is forty minutes, during which the fleet is idle and everyone assumes something is broken.

---

Building and Operating It

You cannot manage a ceiling you have not measured, and every number in this lesson is available from the cluster you already have. Build a single dashboard whose only job is to answer how close each ceiling is.

Start with the one nobody instruments: placeable capacity.

# Nodes with no GPU currently claimed by any pod: the only nodes a
# whole-node gang can actually land on.
$ kubectl get pods -A -o json | jq -r '
    [.items[] | select(.spec.nodeName != null)
     | {n: .spec.nodeName,
        g: ([.spec.containers[].resources.limits["nvidia.com/gpu"] // "0"
             | tonumber] | add)}]
    | group_by(.n)[] | select(([.[].g] | add) > 0) | .[0].n' | sort -u > /tmp/busy

$ kubectl get nodes -l fleet/class=training -o name | sed 's|node/||' \
  | grep -vxFf /tmp/busy | wc -l
188

One hundred and eighty-eight fully free nodes out of 1,250 means the largest gang you can currently admit is 1,504 GPUs, whatever the aggregate free count says. Publish that next to your utilization number; the two together tell a story that neither tells alone.

Then the control plane. Three queries cover the load that matters.

# Request rate by verb: LIST is the expensive one and the one operators generate
sum by (verb, resource) (rate(apiserver_request_total{verb=~"LIST|WATCH"}[5m]))

# Open watches, which is roughly informers times the objects they follow
sum by (resource) (apiserver_longrunning_requests{verb="WATCH"})

# etcd write pressure, the driver of database growth
sum(rate(etcd_mvcc_put_total[5m]))

The second one is the audit. If the watch count on pods is an order of magnitude above the number of components you think are watching pods, something is creating informers per object or restarting in a loop, and you have found a control plane problem that has nothing to do with scale.

Address space needs a policy rather than a dashboard, because it cannot be fixed reactively. Size the cluster CIDR for the fleet you will have in three years, not the one you are deploying, and if your CNI supports multiple pod CIDRs per node or a smaller per-node block, use it: GPU nodes run few pods, so allocating a /24 to a node that will host twelve pods wastes 95 percent of the block.

$ kubectl get pods --all-namespaces --field-selector spec.nodeName=gpu-h100-0141 \
    --no-headers | wc -l
11

Eleven pods on a node holding 254 addresses. Cutting the per-node block to a /27 multiplies your node ceiling by eight without touching anything else.

Two of these ceilings are inherited rather than designed, and it is worth saying so plainly. Power and cooling cap how much of the fleet can draw full load at once, and that envelope was fixed when the building was fitted out. You do not negotiate it with a manifest; you plan capacity against it, which lesson 5.4 does. Fabric capacity is the same: you inherit an interconnect and design the Kubernetes layer above it, which is module 2.

---

Tradeoffs and Decision Framework

Ceiling	How it announces itself	How to measure it	How to raise it
Placeable capacity	Large jobs queue while utilization looks fine	Largest currently admissible allocation	Defragmentation policy, reservation, whole-node requests
Scheduler decision rate	Slow drain after a restart, not an error	p99 attempt duration times pending depth	Fewer constraints, coarser topology tiers, scheduler sharding
Pod address space	Nodes join and stay NotReady	Cluster CIDR divided by per-node block size	Smaller per-node block, or a second cluster
API server load	Latency rises for everything at once	LIST and WATCH rate, longrunning watch count	Scope informers, fewer operator replicas, more API servers
etcd size and history	Slow writes, then a maintenance mode surprise	Database size and MVCC put rate	Compaction and defrag policy, fewer hot large objects
Serving object churn	EndpointSlice and status write storms	Write rate attributable to the serving namespaces	Batch scale steps, cap replica churn, split the namespace
Power and cooling	Facilities calls you	Rack draw against rack budget	Nothing in Kubernetes. Plan against it

Four questions decide whether you have outgrown a cluster. Which of these numbers is closest to its ceiling right now, and how fast is it moving, because the answer is almost never the same as the one people assume? Is the constraint raisable with configuration, or is it structural like the pod CIDR and the power envelope? Does the growth come from more nodes, more churn, or more controllers, since those three have different fixes and only one of them is about size? And what is the blast radius you are willing to accept, because at some point the honest answer is a second cluster rather than a bigger one, which is where lesson 9.2 picks this up.

The default: measure all seven, review them monthly, size the pod CIDR and the etcd disk for three years out because both are painful to change later, and treat the largest placeable allocation as a first-class capacity metric alongside utilization. Component-level tuning for the API server and etcd belongs to the Kubernetes Performance Optimization and etcd Operations courses; the fleet-scale symptoms and sizing belong to module 9.

---

Failure Modes and Common Mistakes

Planning capacity against the published node limit. That figure was measured without a device plugin, a gang scheduler, topology constraints, or your operator set, none of which are optional on a GPU fleet.

Treating aggregate free GPUs as available capacity. A fleet that is forty percent free and fully fragmented cannot admit a large gang or a sharded replica, and the aggregate number will report health right through the incident.

Sizing the cluster CIDR for the initial deployment. It is the one ceiling with no graceful workaround, and hitting it converts an expansion into a cluster migration.

Adding operator replicas for availability without scoping their informers. Each replica is another full cache of every object it watches, and three replicas of a cluster-wide pod watcher cost three times the fanout.

Assuming etcd grows with object count. It grows with write rate and retained history, and node status updates on a GPU fleet are large and constant even when nothing is being deployed.

Tuning the scheduler by raising its throughput setting. Under gang and topology constraints the expensive part is simulation over a snapshot, and asking for more attempts per second does not make a snapshot smaller.

Ignoring the correlation between the two workload classes. A full-fleet training restart and a serving scale event stress different subsystems, and the incident that takes the control plane down is usually both at once.

Believing the ceiling is a GPU ceiling. By the time GPUs are the constraint you have already passed the control plane, the scheduler, the address space, and probably the power budget.

---