Reading a Fleet's Resource Profile

The dashboard says the fleet is at 94% GPU utilization. That number is compatible with a fleet doing excellent work and with one doing almost nothing useful, and you cannot tell which from the dashboard.

---

The Problem at Scale

The utilization figure answers exactly one question: over the last sampling window, was at least one kernel resident on the device. That is the entire definition. It does not know how much of the die that kernel occupied, whether any tensor pipes were involved, or whether the kernel was doing arithmetic at all.

Two consequences follow, and they point in opposite directions, which is why the number is not merely imprecise but actively misleading.

It reads high when nothing is happening. A GPU spinning inside a blocked collective, waiting on a rank that died eight minutes ago, reports 100 percent, because the wait is a resident kernel polling a flag. A serving replica holding a model in memory and answering four requests a minute reports whatever its idle loop happens to look like, and on many runtimes that is not zero.

And its denominator is wrong. Utilization is measured per device, over devices that a pod has claimed. It has nothing to say about the GPUs that are cordoned for a driver rollout, sitting in a repair queue, stranded because a pod requested three of eight, or free but too fragmented to place anything. The fraction of your fleet that never reaches the utilization metric at all is usually larger than the inefficiency the metric is capable of showing you.

Put numbers on it, and substitute your own. Ten thousand installed GPUs. Nine thousand six hundred on nodes that are Ready and schedulable, the rest cordoned, draining, or awaiting parts. Eight thousand nine hundred claimed by a pod. Ninety-four percent of those reporting busy. That last number is the one on the wall, and it describes 8,366 GPUs out of 10,000 installed, which is 84 percent before you have asked a single question about whether the work was useful. Then ask that question and the honest figure moves again, in the same direction.

This is not a monitoring inconvenience. It is why platform teams and research teams talk past each other. The platform dashboard says the fleet is full, so the answer to "we need more GPUs" is "you have them all and they are busy." Both parties are reading real numbers. Only one of them is denominated in work.

---

How It Works

The allocation ladder

Each rung is a separate measurement with a separate remedy, and you can read all of them off the cluster you already have.

$ kubectl get nodes -l fleet/gpu=true -o json | jq -r '
    [.items[] | {
      ready: ([.status.conditions[] | select(.type=="Ready" and .status=="True")] | length),
      sched: (if .spec.unschedulable then 0 else 1 end),
      cap:   (.status.capacity["nvidia.com/gpu"] | tonumber)}]
    | {installed: ([.[].cap] | add),
       schedulable: ([.[] | select(.ready==1 and .sched==1) | .cap] | add)}'
{
  "installed": 10000,
  "schedulable": 9600
}

Four hundred GPUs are on nodes that are up but unavailable, and that number belongs to whoever owns node lifecycle, not to whoever owns training efficiency. The gap between installed and schedulable is the rung most fleets never instrument, and it is the one that grows silently: a cordon applied during an incident and never removed, a node that failed a health check in March, a pool held back from a driver rollout.

The next rung is allocation, and the interesting part of it is not the total but the shape.

$ kubectl get pods -A -o json | jq '[.items[]
    | select(.status.phase=="Running")
    | .spec.containers[].resources.limits["nvidia.com/gpu"] // "0" | tonumber]
    | add'
8900

$ kubectl get pods -A -o json | jq -r '[.items[]
    | .spec.containers[].resources.limits["nvidia.com/gpu"] // empty]
    | group_by(.) | map({req: .[0], count: length})'
[{"req":"1","count":312},{"req":"2","count":48},{"req":"8","count":1043}]

Three hundred and twelve single-GPU pods is not a problem by itself. Three hundred and twelve single-GPU pods scattered across eight-GPU nodes is 312 nodes that can no longer host anything requiring a whole node, which is why the fleet has free GPUs and cannot admit a training gang or rebuild a sharded replica. Lesson 4.3 turns that into an admission rule; here it is a line item in the profile.

Then the busy rung, where the metric everyone quotes finally applies, and where a second reading immediately contradicts it.

$ nvidia-smi --query-gpu=index,utilization.gpu,power.draw,clocks.sm \
             --format=csv,noheader
0, 100 %, 703.44 W, 1980 MHz
1, 100 %, 698.11 W, 1980 MHz
2, 100 %, 121.87 W, 345 MHz
3, 100 %, 699.02 W, 1980 MHz

All four report 100 percent. GPU 2 is drawing a sixth of the power at a sixth of the clock, which is what a device blocked in a collective looks like: busy by the metric, idle by physics. Power draw is a better liveness signal for an accelerator than utilization is, and it costs nothing to collect. The same divergence shows up in the DCGM profiling fields, where graphics-engine activity and streaming multiprocessor activity disagree completely.

$ dcgmi dmon -e 1001,1002,1004 -c 3
# GPU  GRACT  SMACT  TENSO
    0  0.998  0.971  0.514
    1  0.997  0.968  0.511
    2  1.000  0.031  0.000

The honest measure for each class

The top rung has no single metric, because the two workload classes produce different things and must be scored differently.

For training, the measure is model FLOPs utilization: arithmetic the model actually required, divided by what the hardware can do. The numerator comes from the model rather than the machine. For a dense transformer, forward and backward together cost approximately six times the parameter count times the tokens processed, so a 7-billion-parameter model at 8.39 million tokens per step is about 3.5e17 FLOPs per step. Divide by a measured step time of 1.8 seconds for 1.96e17 FLOP/s achieved, then divide by 512 GPUs times the per-device dense peak. Substitute your own peak; the arithmetic is what matters, and the mechanism is that every gap between this number and one has a name.

Goodput is the companion and it is the honest one: useful training time divided by wall clock, counting restarts, requeues, and work recomputed after a failure. A run at 45 percent MFU that restarts twice a day has worse goodput than one at 38 percent that never does, and only goodput sees it.

For serving, the symmetric measure is tokens served per provisioned GPU-hour. The word that carries the weight is provisioned. Dividing by busy GPU-hours flatters you: it silently excludes the replicas you are paying for and not using, which is precisely the inefficiency you are trying to see. And unlike training, the target is not as close to one as possible, because a serving tier that is fully consumed has no headroom and cannot absorb a spike. The serving question is not whether headroom exists but whether anything is running in it, which is the second serving number worth tracking: the fraction of provisioned capacity that was warm and idle when demand arrived, and what was occupying it.

---

Building and Operating It

Build the ladder as recording rules so that every rung has the same shape and the same time resolution, and so nobody can quote one rung without the others being one click away.

record: fleet:gpus_installed
expr: sum(kube_node_status_capacity{resource="nvidia_com_gpu"})

record: fleet:gpus_schedulable
expr: sum(kube_node_status_allocatable{resource="nvidia_com_gpu"}
        unless on (node) kube_node_spec_unschedulable == 1)

record: fleet:gpus_allocated
expr: sum(kube_pod_resource_limit{resource="nvidia_com_gpu"}
        * on (pod) group_left kube_pod_status_phase{phase="Running"})

record: fleet:gpu_hours_useful_training
expr: sum(training_world_size * training:mfu)

One detail in those rules is worth defending, because it is the difference between a profile that survives an argument and one that does not. Integrate over time rather than reading instantaneous values. A fleet is not a state, it is a flow of GPU-hours, and a spot reading taken at 10am on a Tuesday will show a different allocation shape than one taken at 3am on a Sunday. Every rung should be reported as GPU-hours accumulated over a window, which makes the rungs additive, makes the losses comparable, and makes the whole profile denominated in the same unit as the invoice.

Then do the thing that actually changes behavior, which is a monthly reconciliation that accounts for one hundred percent of the fleet's GPU-hours into named buckets, each with an owner. Not a dashboard, a table.

$ ./fleet-accounting.sh --month 2026-07
installed GPU-hours                 7,440,000   100.0%
  unschedulable: repair queue         148,800     2.0%   platform-hw
  unschedulable: driver rollout        44,640     0.6%   platform-sw
  free, placeable                     223,200     3.0%   capacity
  free, stranded by fragmentation     186,000     2.5%   platform-sched
  allocated: serving, serving traffic 2,232,000   30.0%  inference
  allocated: serving, warm headroom     744,000   10.0%  inference
  allocated: training, stepping       3,348,000   45.0%  research
  allocated: training, lost to restarts 513,360    6.9%  platform-rel

Nothing in that table is a metric you can buy. It is an accounting exercise, and its value is that every line has a name next to it and no line can hide inside "utilization." The two lines people find uncomfortable are the ones that matter most: warm headroom, which is a deliberate purchase and should be justified or backfilled, and work lost to restarts, which is the reliability budget expressed in the only currency anyone cares about. Module 6 is what shrinks that last line and module 5 is what monetizes the one above it.

The instrumentation depth, the alerting, and the fleet-wide view across heterogeneous hardware are module 8's subject. What belongs here is the reading habit: never quote a rung without its neighbours, and never let a single fleet-wide efficiency number stand in for two workload classes that are scored differently.

---

Tradeoffs and Decision Framework

Measure	What it honestly answers	How it misleads	Who should read it
GPU utilization	Is a kernel resident on this device	Saturates near 100 percent for healthy, sick, and stalled alike	Nobody making a decision; a presence check at most
Power draw and SM clocks	Is the device doing electrical work	Says nothing about whether the work is useful	On-call, as the cheapest true liveness signal
Allocated over schedulable	Is the fleet claimed	Hides fragmentation behind an aggregate	Capacity planning
Largest placeable allocation	Can I still run the work I have	Volatile; needs a trend, not a spot value	Scheduling and admission policy
MFU	How much of the machine a training run is getting	Model-specific; flattered by short sequences	Research and platform, per run, against its own baseline
Goodput	Useful work over wall clock, restarts included	Nothing. It is the honest number	Everyone, and it is the case for reliability work
Tokens per provisioned GPU-hour	What serving produced for what it cost	Not comparable across model sizes	Inference and finance, per model

Four questions get you from a complaint to a cause. Which rung of the ladder moved, because a drop in useful work with allocation unchanged is a completely different investigation from allocation falling with the fleet unchanged? Is the run slower than its own baseline or slower than someone's expectation, since only the first is a defect? Are clocks and power where they were last week, which is the check that costs thirty seconds and eliminates an entire class of cause? And is the headroom you are paying for occupied by anything, because that is the largest single lever on this table and it is a policy decision rather than a measurement.

The default: publish the five-rung ladder with an owner per rung, score training with MFU and goodput against each run's own baseline, score serving with tokens per provisioned GPU-hour, reconcile the fleet's GPU-hours monthly, and keep utilization off any dashboard an executive can see.

---

Failure Modes and Common Mistakes

Building a capacity argument on utilization. It saturates near 100 percent regardless of efficiency, so a fleet delivering a third of its possible output will report health right through the conversation.

Measuring efficiency against busy GPU-hours instead of provisioned ones. That denominator excludes exactly the waste you are looking for, and it makes an over-provisioned serving tier look excellent.

Forgetting the rungs above allocation. Cordoned nodes, repair queues, and held-back pools are frequently a larger loss than any inefficiency inside the running workload, and nothing in the standard GPU dashboards shows them.

Using the sparse peak FLOPs figure in the MFU denominator. It halves every result and converts healthy runs into investigations.

Comparing MFU or tokens per GPU-hour across different models. Both move substantially with model shape, sequence length, and batch size, so the only fair comparison is a workload against itself.

Treating serving headroom as waste to be eliminated. It is insurance you deliberately bought, and the correct response is to occupy it with preemptible work, not to delete it and discover the spike.

Ignoring clocks and power. A rack left with a low power cap or a node quietly throttling produces a uniform permanent slowdown that no utilization or error metric will ever reveal.

Reporting one fleet-wide efficiency number. Averaging a training measure and a serving measure produces a figure that describes neither class and cannot be acted on by either team.

---