Training and Inference on One Fleet

You own one GPU fleet. Half of it runs jobs that occupy hundreds of nodes for six weeks and die if any rank dies. The other half runs replicas that must answer in 800 milliseconds and scale with traffic. Both are your problem.

---

The Problem at Scale

The comfortable answer used to be to build two things: a training cluster with a gang scheduler, fat nodes, a fast fabric, and a maintenance policy that treats every eviction as an outage; and a serving cluster with autoscaling, rolling updates, and a load balancer in front. That answer is gone for a budget reason rather than a technical one. The accelerators are the same parts, they were bought once, they are the most expensive line item the company has, and nobody is going to sign off on running two of them at sixty percent each. So you own one fleet, and the two things running on it disagree about nearly every default in Kubernetes.

State each one as a resource shape rather than as machine learning, because that is all the platform needs to know. A training job is a single program whose address space happens to be spread across hundreds of nodes, advancing in lockstep, allocated once at launch and released once at the end. Its capacity is indivisible: 512 GPUs or nothing. Its failure is total: one rank dies and the run stops. Its time constant is weeks. It will wait in a queue for hours without complaint, because waiting costs nothing and starting badly costs six weeks.

A serving replica is one instance of a function that answers requests, one of many, created and destroyed continuously as demand moves. Its capacity is divisible: sixty replicas or fifty-nine, and fifty-nine is a slightly worse Tuesday rather than a failure. Its failure is partial and self-healing. Its time constant is the time budget of a single request. It cannot wait in a queue at all, because a replica that becomes ready ten minutes after the traffic arrived did not help.

Now look at what happens when you tune the platform for one of them. A platform built for training taints its GPU nodes, routes everything through a gang scheduler, disables the horizontal autoscaler because a world size cannot change mid-run, and treats kubectl drain as a change-managed event. Put a serving deployment on it and every scale-up waits behind a scheduler session, and the pod that should have replaced a crashed replica in forty seconds is still Pending.

Run it the other way and the damage is worse. A platform built for serving bin-packs aggressively, drains nodes on a rolling schedule, sets a PodDisruptionBudget that permits ten percent unavailability, and puts a liveness probe on everything. Each of those is correct for replicas and each silently destroys a training run: the bin-packer strands GPUs so no large gang fits, the drain kills a rank, the disruption budget grants permission to delete one, and the liveness probe restarts a process that is blocked in a collective and cannot rejoin its peers.

---

How It Works

Four axes, and both classes sit at opposite ends of all of them

The classes differ on more than one dimension, but four of them carry almost all of the consequence, and naming them gives you a way to evaluate any mechanism you are about to install.

The unit of scheduling. For training the unit is the gang: a set of pods that must be admitted together or not at all, which lesson 1.3 develops. For serving it is the replica, which is usually one pod and increasingly is several, which lesson 1.4 develops. In neither case is the unit a pod, and the default scheduler only knows about pods.

The unit of failure. A training job's availability is binary. A serving tier's availability is a fraction, and that fraction is what the error budget is written against. It is why the same node fault produces a page in one class and a metric blip in the other.

The direction of the capacity change. Training capacity is set once and does not move; serving capacity moves constantly, in both directions, driven by something outside your control. The platform has to support both on the same node pool without one starving the other.

The cost of waiting. This is the axis that surprises people, and it is the one that makes coexistence possible. A training job in a queue is behaving correctly and costing nothing. A serving replica in a queue is an incident in progress. That asymmetry is not a problem to be solved; it is the lever. It is the reason training work can be made to absorb the fleet's idle capacity and yield it back, which module 5 turns into a utilization strategy.

Where the platform is genuinely shared

It is tempting to conclude that the classes have nothing in common and should simply be partitioned. That is wrong, and the shared surfaces are exactly where the design work is.

The node is shared. Both classes consume nvidia.com/gpu from the same device plugin and depend on the same driver and firmware version, so when you upgrade a driver you upgrade it for both.

The scheduler is shared, or at least the nodes it decides over are. Two schedulers with independent caches deciding about the same GPUs is a known way to produce a kubelet admission failure after both believed they had won, so you either partition by node taint or run one scheduler that understands both admission models.

The image and weight distribution path is shared, and this is the one that surprises teams. A training image and a model weight set are both multi-gigabyte artifacts pulled by many nodes at once, so a full-fleet training restart landing on the registry at the same time as a serving scale-up is a real correlated failure. Module 5 is largely about it.

The quota and priority system is shared, because both classes are charged against the same finite GPU count and someone has to decide who yields. That is a policy question rather than a scheduling one, and module 4 is where it lives.

The observability stack is shared and the metrics are not. GPU utilization means something different for each class, and the dashboard that satisfies a finance review satisfies neither. Lesson 1.5 takes that apart.

---

Building and Operating It

The practical shape is one cluster, one device plugin, and a deliberate separation of the two classes at three layers: node pools, priority, and queues. Start with labels that say which class a node is currently serving, because everything else keys off them.

$ kubectl get nodes -L fleet/class,nvidia.com/gpu.product --no-headers | head -4
gpu-h100-0141   Ready   training   NVIDIA-H100-80GB-HBM3
gpu-h100-0142   Ready   training   NVIDIA-H100-80GB-HBM3
gpu-h100-0410   Ready   serving    NVIDIA-H100-80GB-HBM3
gpu-h100-0411   Ready   serving    NVIDIA-H100-80GB-HBM3

$ kubectl get nodes -l fleet/class=serving -o json \
  | jq '[.items[].status.allocatable["nvidia.com/gpu"] | tonumber] | add'
512

Note that fleet/class is a label, not a permanent property of the hardware. The boundary between the two pools is the most important number you tune, and it has to move on a weekly timescale rather than a purchase-order timescale.

Priority is where the yielding rule gets written down. Three classes are enough, and the numbers matter less than the ordering and the preemptionPolicy.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: serving-production
value: 100000
preemptionPolicy: PreemptLowerPriority
description: "Latency-facing replicas. May preempt anything below."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-committed
value: 50000
preemptionPolicy: Never          # a committed run does not preempt serving
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-backfill
value: 1000
preemptionPolicy: Never
description: "Checkpoint-tolerant work that lives in serving headroom."

That third class is what pays for the whole arrangement. Serving must hold headroom to absorb a spike, headroom is idle GPUs by definition, and training-backfill occupies them until the spike arrives. Module 5 covers how much headroom and how to make the yield clean; the point here is that the two classes are complementary, not merely competing, once the policy exploits the asymmetry in the cost of waiting.

The queues make the same statement to the admission layer.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: serving
spec:
  cohort: gpu-fleet
  preemption:
    reclaimWithinCohort: Any       # take back borrowed capacity immediately
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: h100-sxm
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 512
              borrowingLimit: 0    # serving never borrows; it must always fit

borrowingLimit: 0 on the serving queue is deliberate and slightly counterintuitive. Letting serving borrow from training looks generous, and it means that on the day you need to reclaim, the reclaim path is in the critical path of an incident. Size serving to its peak, let training borrow the difference, and keep the reclaim direction one-way.

Then audit for the defaults nobody meant to apply. Every PodDisruptionBudget, HPA, and liveness probe attached to a gang workload is a serving assumption that made it across the boundary.

$ kubectl get pdb -A -o json | jq -r '.items[]
    | select(.spec.selector.matchLabels["fleet/class"]=="training")
    | [.metadata.namespace, .metadata.name, .spec.maxUnavailable] | @tsv'
train   research-frontier-pdb   1

$ kubectl get hpa -n train
NAME              REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS
train-run-9-hpa   Job/train-run-9        98%/70%   64        128       64

Both of those objects were copied from a serving template and both are actively dangerous. The disruption budget tells the eviction API that deleting one rank is acceptable, and the autoscaler is trying to change a world size that was fixed at rendezvous. Reject them at admission rather than finding them during an incident.

Finally, alert per class, because the two have almost no signals in common. A serving tier is healthy when its latency percentiles and ready replica count are where you expect them. A training job is healthy when it is completing steps, and one that is Running and not stepping is the pathology no stock alert catches.

# Serving: capacity that is actually ready, not merely scheduled
sum by (model) (kube_deployment_status_replicas_available{namespace="serve"})
  / sum by (model) (kube_deployment_spec_replicas{namespace="serve"}) < 0.9

# Training: the job is up and producing nothing
time() - training_last_step_timestamp_seconds{job_id!=""} > 300

---

Tradeoffs and Decision Framework

Platform decision	What training wants	What serving wants	Workable resolution
Admission	Atomic over the whole gang, minutes are fine	Immediate, per replica, seconds matter	Both, routed by workload class at the queue
Node packing	Whole nodes, tightly grouped	Whatever fits, spread for availability	Whole-node requests for training, bin-pack the serving pool only
Eviction	Never, except at a checkpoint boundary	Routinely, that is how rollouts work	Priority-gated eviction plus a checkpoint-aware grace period
Target utilization	As close to 100 percent as possible	Deliberate headroom for spikes	Serving headroom filled with preemptible training
Autoscaling	Meaningless during a run	The primary control loop	HPA and KEDA scoped to the serving namespaces only
Disruption budget	Zero unavailable, always	A percentage, so rollouts can proceed	No PDB on gang workloads; the gang object is the budget
Failure response	Stop, diagnose, restart the whole job	Replace the instance and move on	Different runbooks, keyed off the priority class

Four questions settle most arguments here. Which class is this mechanism written for, and what does it do to the other one at 3am? Is the capacity under discussion divisible, because if it is not, averages and percentages are the wrong vocabulary entirely? Does this workload pay for waiting or pay for latency, since that answer alone sets its queue, its priority, and its eviction policy? And is the headroom serving requires being used by anything, because if it is not, you are paying full price for insurance.

The default for a shared fleet: one cluster, one device plugin, two node pool labels whose boundary you move weekly, three priority classes with a one-way preemption direction from serving to training, and separate queues, alerts, and runbooks per class. Everything after this in the course is a refinement of that shape.

---

Failure Modes and Common Mistakes

Copying a serving pod template onto a training workload. The liveness probe, the PodDisruptionBudget, the rolling update strategy, and the HPA are all individually correct for replicas and all individually fatal to a gang.

Letting a cluster-wide controller treat every pod as interchangeable. Deschedulers, bin-packers, spot reclaim handlers, and node auto-repair all default to the assumption that a pod can be moved, and each of them will eventually delete a rank.

Sizing the serving pool for average traffic and borrowing for peaks. Reclaim then sits in the critical path of every incident. Size serving for peak and let training borrow the trough instead.

Running two schedulers over the same GPU nodes. Independent caches produce independent assumptions about the same devices, and the loser finds out at kubelet admission after the gang was already declared satisfiable.

Applying one utilization target to the whole fleet. A fleet held at 95 percent cannot absorb a traffic spike or admit a large job promptly, and a fleet held at 60 percent is burning money. The correct target is per class, and the gap between them is the backfill tier.

Treating the training and serving pools as a permanent physical split. If the boundary can only move through a procurement cycle, it will be wrong for most of the year, and the wrong half will be idle while the other half queues.

Assuming the shared surfaces are only the nodes. The registry, the weight distribution path, the quota system, and the control plane are shared too, and they fail under the correlated load of a full-fleet training restart landing on top of a serving scale-up.

---