Reading a Training Run's Resource Profile

A researcher says the run is slow. The GPUs show 95% utilization on every dashboard you have. Both statements are true, and the utilization number is telling you almost nothing.

---

The Problem at Scale

The number in the Utilization column of nvidia-smi answers one question: over the last sampling window, was at least one kernel resident on the device? That is the whole definition. It does not know how many of the streaming multiprocessors that kernel occupied, how many warps were resident on each, whether any tensor cores were involved, or whether the kernel was doing arithmetic at all.

Which means a GPU running a single-block kernel that uses one percent of the die reports 100 percent. A GPU spinning inside a blocked NCCL collective, waiting for a rank that died eight minutes ago, also reports 100 percent, because the wait is implemented as a resident kernel polling a flag. On a training workload the utilization metric saturates near 100 percent for a perfectly tuned run, a badly tuned run, and a completely stalled run alike, which makes it not merely imprecise but actively misleading.

$ nvidia-smi --query-gpu=index,utilization.gpu,power.draw,clocks.sm \
             --format=csv,noheader
0, 100 %, 703.44 W, 1980 MHz
1, 100 %, 698.11 W, 1980 MHz
2, 100 %, 121.87 W, 345 MHz
3, 100 %, 699.02 W, 1980 MHz

All four GPUs report 100 percent utilization. GPU 2 is drawing a sixth of the power of its neighbors at a sixth of the clock, which is what a GPU blocked in a collective looks like: busy by the utilization metric, idle by physics. Power draw is a better liveness signal for a training GPU than utilization is, and it costs nothing to collect.

This is not a monitoring inconvenience. It is the reason platform teams and research teams talk past each other. The platform team's dashboards say the fleet is fully utilized, so the answer to "we need more GPUs" is "you already have them all and they are busy." The research team's step time says otherwise. Both are reading real numbers. Only one of them is denominated in work.

What you need is a metric whose denominator is the hardware's capability and whose numerator is arithmetic the model actually required. That metric is model FLOPs utilization, and everything in this lesson is either how to compute it, how to decompose the gap between it and 100 percent, or what to do with the answer.

---

How It Works

The utilization ladder

DCGM exposes a ladder of increasingly honest signals, and knowing where each one sits saves a lot of arguing. DCGM_FI_PROF_GR_ENGINE_ACTIVE is the graphics engine active fraction, essentially the same claim nvidia-smi makes. DCGM_FI_PROF_SM_ACTIVE is the fraction of streaming multiprocessors with at least one resident warp, which finally distinguishes a one-block kernel from a full-device one. DCGM_FI_PROF_SM_OCCUPANCY is resident warps against the maximum. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is the fraction of cycles the tensor pipes were issuing, which is the closest thing to a hardware-level answer to "is this thing doing the math it was bought for."

$ dcgmi dmon -e 1001,1002,1003,1004,1005 -c 5
# GPU  GRACT  SMACT  SMOCC  TENSO  DRAMA
    0  0.998  0.971  0.412  0.514  0.331
    1  0.997  0.968  0.409  0.511  0.329
    2  1.000  0.031  0.008  0.000  0.002
    3  0.996  0.970  0.413  0.512  0.330

GPU 2 again. Graphics engine active at 1.000, SM active at 0.031, tensor pipes at zero. That is a GPU running a spin-wait kernel and nothing else, and the two leftmost columns disagree so completely that any dashboard showing only the first one is worse than no dashboard. On the healthy GPUs, tensor pipe activity around 0.5 is normal and good for transformer training; it is not fifty percent efficiency, because the step also contains normalization, activation functions, optimizer arithmetic, and communication, none of which touch the tensor pipes.

Model FLOPs utilization

MFU is achieved model FLOPs per second divided by the fleet's peak FLOPs per second. Both halves need care.

The numerator comes from the model, not from the hardware. For a dense transformer, the forward and backward passes together cost approximately 6 times the parameter count times the number of tokens processed. If the run uses full activation recomputation, add roughly another 2 times parameters times tokens for the recomputed forward, which gives 8PT; that extra work is real and it is a good reason to report hardware FLOPs utilization alongside MFU when recomputation is on.

Work an example. A 7-billion-parameter model, a global batch of 2,048 sequences of 4,096 tokens, so 8.39 million tokens per step. Model FLOPs per step is 6 times 7e9 times 8.39e6, about 3.5e17. If the measured step time is 1.8 seconds, achieved throughput is 3.5e17 divided by 1.8, about 1.96e17 FLOP/s.

The denominator is 512 GPUs times the per-device peak. Call that peak F and substitute the published dense bf16 number for your part. If F is around 1e15 FLOP/s, the fleet peak is 5.12e17, and MFU is 1.96e17 divided by 5.12e17, about 38 percent.

Where the wall clock actually goes

MFU tells you how much of the machine you are getting. The step time decomposition tells you where the rest went, and it has four buckets: compute, exposed communication, data wait, and idle.

Exposed communication is the subtle one. Frameworks overlap gradient allreduce with the backward pass, so total collective time is not the same as time lost. What costs you is the portion that could not hide behind compute. The practical way to measure it is by subtraction: run the same model and batch on a single node where the only collectives are intra-node, record the step time, then scale out and attribute the difference.

$ nsys profile -t cuda,nvtx --stats=true -o step python train.py --max-steps 20
...
 Time(%)  Total Time(ns)  Instances    Name
 -------  --------------  ---------  -------------------------------------
    41.2      742,118,004      1,920  ampere_bf16_gemm_128x256_ldg8_stages
    18.7      336,904,551        320  ncclDevKernel_AllReduce_Sum_bf16_RING
    11.3      203,551,890      1,280  ncclDevKernel_SendRecv
     9.8      176,442,101      3,840  layer_norm_fwd_kernel
     6.1      109,884,332        640  elementwise_kernel

Nineteen percent of device time inside an allreduce kernel does not mean nineteen percent lost, because most of it overlaps. But if ncclDevKernel_AllReduce climbs from 19 percent to 45 percent when you double the world size, the overlap has stopped working and the bucket sizes or the schedule need attention.

---

Building and Operating It

Establish the baseline before you need it. Run the same model and per-GPU batch at one node, eight nodes, sixty-four nodes, and full scale, and record step time and MFU at each point. Scaling efficiency is throughput at N divided by N times throughput at one, and it is the number that separates "the fabric is slow" from "this model never went faster than that."

Then make the training process itself emit the metrics, because nothing outside the process can compute them. Three gauges are enough to run a fleet: step time, tokens per second, and MFU, all labeled by job and by rank.

# MFU, computed in the recording rule so the constant lives in one place
record: training:mfu
expr: >
  (training_model_flops_per_step{} / training_step_seconds{})
  /
  (training_world_size{} * 1e15)

# Straggler signal: the slowest rank against the median rank
record: training:step_skew
expr: >
  max by (job_id) (training_step_seconds)
  /
  quantile by (job_id) (0.5, training_step_seconds)

That second rule is the one people forget to build. Averaging step time across ranks is close to useless, because a synchronous job's ranks all report nearly the same step time by construction: they are all waiting for the slowest one. What tells you something is the ratio of the maximum to the median, and more precisely which rank is consistently at the maximum. A skew of 1.02 is a healthy job. A skew of 1.3 that is always the same rank is a sick node, and module 4 is about finding it.

Alert on the derivative, not the level, because the absolute MFU of a run depends on the model and nobody can tell you in advance what it should be.

# MFU has dropped more than 10 percent against this run's own first hour
(
  avg_over_time(training:mfu{job_id="train-run-11"}[15m])
  /
  quantile_over_time(0.9, training:mfu{job_id="train-run-11"}[1h] offset 5h)
) < 0.9

Two more practical notes. First, do not compute MFU from a rolling average of step time that includes checkpoint steps, evaluation steps, or the first few warmup steps; they are real wall clock and belong in goodput, but mixing them into MFU produces a number that oscillates for reasons unrelated to efficiency. Report both, and define goodput explicitly as useful training time divided by wall clock, including restarts and lost work.

Second, sample DCGM profiling fields at a coarse interval. The profiling metrics use the same hardware counters a profiler does, and scraping them aggressively on every GPU has a measurable cost. Ten to thirty seconds is plenty for fleet monitoring; use nsys or the framework profiler when you need per-kernel resolution, on one job at a time.

---

Tradeoffs and Decision Framework

Metric	What it actually measures	What it misses	Use it for
nvidia-smi utilization	Whether any kernel is resident	Width, occupancy, arithmetic, and whether the kernel is a spin-wait	A presence check, nothing more
SM_ACTIVE	Fraction of SMs with a resident warp	Whether those warps do useful math	Distinguishing a stalled GPU from a working one
PIPE_TENSOR_ACTIVE	Tensor pipe issue rate	Everything the model does outside matmuls	Detecting clock limits and kernel regressions
Step time	Wall clock per optimizer step	Nothing, but it is model-specific and has no absolute scale	Regression detection within one run
MFU	Useful model arithmetic against dense peak	Recomputation overhead, and it flatters short sequences	Comparing efficiency across hardware and plans
Goodput	Useful training time over total wall clock	Nothing. It is the honest number	Capacity planning and the case for reliability work
Power draw per GPU	Actual electrical work being done	Which part of the step is drawing it	The cheapest true liveness signal you have

Four questions get you from a complaint to a cause. Is the run slower than its own baseline, or slower than someone's expectation, because only the first is a defect? Did MFU drop while step time held, which means the batch or sequence length changed rather than the infrastructure? Is the max-to-median step skew flat, which points at a global cause such as the fabric or the data pipeline, or is it concentrated on specific ranks, which points at hardware? And are the clocks and power draw where they were last week, which is the check that costs thirty seconds and eliminates an entire class of cause?

The default: record MFU, step time, max-to-median skew, and per-GPU power for every run, alert on the derivative of MFU against that run's own first hour, and never put GPU utilization on a training dashboard where an executive can see it.

---

Failure Modes and Common Mistakes

Building capacity arguments on GPU utilization. It saturates near 100 percent regardless of efficiency, so a fleet at 99 percent utilization can be delivering a third of its possible throughput and the metric will never say so.

Using the sparse peak FLOPs figure in the MFU denominator. It halves every result and turns healthy runs into investigations. Pin the dense constant in one place and have every dashboard read it from there.

Averaging step time across ranks. In a synchronous job every rank waits for the slowest, so the average is nearly the maximum by construction and hides the one rank that is causing it. Track max over median.

Comparing MFU across models as if it were a hardware score. Sequence length, batch size, and recomputation all move it substantially, so the only fair comparison is a run against itself or against an identical configuration.

Forgetting to include restarts and lost work in the efficiency story. A run at 45 percent MFU that restarts twice a day has far worse goodput than one at 38 percent that never does, and only goodput captures that.

Scraping DCGM profiling fields every second on every GPU. These read the same counters a profiler uses, and aggressive collection costs real throughput on the workload you are trying to measure.

Ignoring clocks and power. A rack left with a low power cap, or a node quietly thermal throttling, produces a permanent uniform slowdown that no utilization or error metric will ever reveal.

---