The Job Is the Unit, Not the Pod

You submit a 512 pod training job to a cluster with 400 free GPUs. Kubernetes cheerfully schedules 400 pods, which sit idle waiting for peers that will never arrive, holding the resources that would have let a smaller job run.

---

The Problem at Scale

The default Kubernetes scheduler is a greedy, per-pod binder. It takes one pod off the queue, finds the best node for that pod in isolation, binds it, and moves on. That design is exactly right for the workloads Kubernetes was built for, where each pod is independently useful the moment it starts. It is exactly wrong for a workload where a pod is worth nothing until all of its peers are also running.

Watch what that produces.

$ kubectl get pods -n train -l job-id=train-run-9 --no-headers | awk '{print $3}' | sort | uniq -c
    400 Running
    112 Pending

$ kubectl describe pod train-run-9-worker-455 | tail -3
Events:
  Type     Reason            Age    Message
  Warning  FailedScheduling  4m12s  0/1250 nodes are available: 1250 Insufficient nvidia.com/gpu

Four hundred pods are Running. Every one of them has completed its container start, allocated its CUDA context, and blocked in the rendezvous barrier waiting for a world size of 512 that will never assemble. They will wait there forever, because nothing in Kubernetes has an opinion about how long a pod should stay useless. Meanwhile they hold 3,200 GPUs.

This is worse than an outage, because an outage is visible. The cluster reports 400 pods healthy, GPU allocation near capacity, and no errors anywhere. Your utilization dashboard, if it is built on allocated GPUs rather than useful work, shows a busy cluster. The system is doing precisely what you told it to do, at full cost, producing nothing.

Now add a second job. Job A needs 512 GPUs, job B needs 512 GPUs, and the cluster has 800 free. The scheduler interleaves them: A gets 400, B gets 400, neither can start, and neither will ever release what it holds, because a Pending pod does not cause a Running pod to be evicted. That is a classical resource deadlock, and it is stable. It does not resolve at 3am, it does not resolve when load drops, and it does not resolve when the researcher who submitted job B goes home. It resolves when a human deletes something.

Priority and preemption make it worse rather than better, because they also operate per pod. A high-priority job preempts thirteen pods scattered across thirteen different low-priority jobs. Each of those thirteen jobs dies, because losing one rank kills a gang. The high-priority job collects thirteen freed GPUs, which is not enough to run, and waits. You have destroyed thirteen running jobs to gain nothing.

---

How It Works

All-or-nothing admission

Gang scheduling replaces the per-pod decision with a decision over a set. The set carries a minimum member count, and the scheduler will not bind any pod in the set until it has confirmed that it can bind all of them. If it cannot, it binds none, and the resources stay free for a job that does fit.

In Volcano that set is a PodGroup, and the field that carries the semantics is minMember.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: train-run-9
  namespace: train
spec:
  minMember: 64                 # 64 pods, 8 GPUs each, world size 512
  minResources:
    nvidia.com/gpu: "512"
  queue: research-frontier
  priorityClassName: production-training

minMember is the count that must be schedulable together. minResources is the aggregate the group needs, and it is what the queueing logic reserves. Setting minMember lower than the pod count is legal and is how elastic training is expressed, but for a synchronous run with a fixed world size the two should be identical. A minMember of 60 on a 64-pod job means Volcano will happily start the job with 60 pods, and your rendezvous will then block waiting for the other four.

It is worth being explicit about how far this is from a Deployment, because the two objects look superficially similar and mean opposite things. A Deployment's controller maintains a replica count as an invariant: if a pod dies, create another, forever, independently of what the others are doing. A PodGroup describes a set whose members are only meaningful together, and whose correct response to losing a member is to tear the set down. Replica count is a target in one and a contract in the other. The moment you find yourself reasoning about a training job in terms of "how many replicas are up," you have imported the wrong abstraction, and every conclusion after that will be off by the difference between a population and a program.

The same inversion applies to the probes. A readiness probe on a service means "route traffic here," which is meaningless when there is no traffic and every peer address was fixed at rendezvous. A liveness probe means "this instance is unhealthy, replace it," which for a rank is a decision to kill the job. The only probe that earns its place on a training pod is a startup probe that gates on the process reaching rendezvous, and even that is usually better expressed as a job-level timeout than as a per-pod check.

The state machine is visible, which matters more than it sounds.

$ kubectl get podgroup -n train
NAME           STATUS     MINMEMBER   RUNNING   AGE
train-run-9    Inqueue    64          0         6m
train-run-12   Running    32          32        4h21m

$ kubectl describe podgroup train-run-9 -n train | tail -4
Events:
  Type     Reason         Age   Message
  Warning  Unschedulable  5m    2/64 tasks in gang unschedulable: pod group is
                                not ready, 62 minAvailable; insufficient
                                nvidia.com/gpu on 1250 nodes

Inqueue with zero running is the state you want to see when capacity is short. Nothing is held, nothing is wasted, and the reason is written down. Compare that against 400 pods in Running telling you nothing.

Admission is not placement

Kueue solves an adjacent problem and it is important not to confuse the two, because teams routinely install Kueue, believe they have gang scheduling, and then hit the original failure anyway.

Kueue works at the Job level. It suspends the Job object before any pod is created, evaluates it against a ClusterQueue's quota, and unsuspends it only when the quota is available. That gives you queueing, hierarchical quota, borrowing between teams, and a clean answer to "whose capacity is this." What it does not give you is a guarantee that the pods will fit on the nodes. Kueue admits against quota; the kube-scheduler still places against nodes, one pod at a time. If the cluster has 512 GPUs of free quota but they are fragmented into ones and twos across 300 racks, Kueue unsuspends the Job, the scheduler binds what it can, and you are back to a half-started gang.

The mitigation is waitForPodsReady, which gives an admitted workload a deadline to get all of its pods running and evicts and requeues it if it misses. That converts a silent hang into a bounded retry, which is a real improvement, but notice what it is: a timeout compensating for the absence of atomic placement, not atomic placement.

---

Building and Operating It

The practical shape is a Volcano Job whose tasks map to your parallelism plan, with restart policy expressed at the job level rather than the pod level.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: train-run-9
spec:
  minAvailable: 64
  schedulerName: volcano
  queue: research-frontier
  policies:
    - event: PodFailed
      action: RestartJob        # one rank dies, the whole gang restarts
    - event: PodEvicted
      action: RestartJob
  tasks:
    - name: worker
      replicas: 64
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: trainer
              resources:
                limits:
                  nvidia.com/gpu: 8

RestartJob on PodFailed is the line that encodes the mental model from the first lesson. A single rank failure is a job failure, so make the controller do the whole teardown and rebuild rather than leaving a zombie gang for a human to find. The alternative, restarting the failed pod alone, produces a container that comes up healthy and cannot join anything.

On the quota side, a ClusterQueue per team with a shared cohort is what lets an idle team's capacity be borrowed instead of stranded.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-frontier
spec:
  cohort: gpu-fleet             # teams in one cohort can borrow from each other
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: h100-sxm
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 4096
              borrowingLimit: 2048

nominalQuota is what the team owns, borrowingLimit is how much more it may take from idle peers, and reclaimWithinCohort is what happens when the owner comes back. Module 3 goes through the fair-share arithmetic and the reclaim policy in detail; what matters here is that the queue is where a training job waits, and waiting in a queue holding nothing is the correct behavior.

Then turn on the safety net and set the deadline honestly.

apiVersion: config.kueue.x-k8s.io/v1beta1
kind: Configuration
waitForPodsReady:
  enable: true
  timeout: 10m                  # longer than image pull plus rendezvous
  requeuingStrategy:
    timestamp: Eviction
    backoffLimitCount: 3

Ten minutes is not arbitrary. It needs to cover the slowest realistic image pull for a multi-gigabyte training image across however many nodes pull simultaneously, plus process group initialization, which itself takes tens of seconds at large world sizes. Set it to five minutes and you will evict healthy jobs during a registry slowdown, which converts a transient problem into a requeue loop.

The alert that war story ends with is the cheapest thing in this lesson. Every training job should emit a step counter, and any job that has been Running for longer than its expected startup without emitting step 1 should page. That single signal catches partial gangs, rendezvous failures, wedged data loaders, and a NIC that came up without RDMA, all of which look identical from outside the pod.

---

Tradeoffs and Decision Framework

Mechanism	Guarantees	Does not guarantee	Use when
Default kube-scheduler	Fast, per-pod, no extra components	Anything about sets; partial binding is the norm	Never, for synchronous training
kube-scheduler coscheduling plugin	Simple gang admission via PodGroup	Rich queueing, quota, borrowing, topology awareness	Small clusters, one team, minimal operational surface
Volcano	Gang scheduling, queues, priorities, topology plugins	Native integration with upstream Job quota semantics	You need placement control and gang semantics together
Kueue	Quota, cohorts, borrowing, reclaim, suspend-based admission	Atomic placement onto nodes by itself	Multiple teams sharing a fleet, quota is the hard problem
Kueue plus a gang scheduler	Quota and atomic placement	Simplicity; you now run two controllers	Large shared research fleets, which is most of this course
Static partitioning, one team per node pool	Total isolation, trivial to reason about	Any utilization at all; idle capacity is stranded	Only when isolation is a compliance requirement

Four questions settle the choice. Is the hard problem contention between teams, or fragmentation within a team, because the first is a quota problem and the second is a placement problem? Does any job in the fleet need more GPUs than the largest contiguous free block you routinely have, since that is what makes atomic placement mandatory rather than nice? Can a job tolerate a smaller world size, which decides whether minMember may sit below the replica count? And who is on call for the scheduler itself, because Volcano and Kueue are both control-plane components with their own failure modes.

The default for a shared research fleet: Kueue for admission and quota, a gang scheduler for placement, minMember equal to the replica count, and a job-level alert on time-to-first-step. Reach for elasticity only after the failure budget arithmetic in module 4 says it pays for itself.

---

Failure Modes and Common Mistakes

Setting minMember below the replica count on a synchronous job. The gang starts, the rendezvous blocks waiting for the missing ranks, and you get the exact failure that gang scheduling was installed to prevent, with the scheduler reporting success.

Assuming quota admission implies placement. Kueue unsuspends against quota, not against node fit. Without a gang scheduler underneath, a fragmented cluster still produces half-bound jobs.

Leaving waitForPodsReady disabled. It is off by default, and it is the only thing that bounds how long an admitted-but-unplaceable job holds capacity.

Using per-pod preemption on gang workloads. Preempting scattered pods kills several jobs and frees a set of GPUs too fragmented to run the preemptor, which is the worst possible outcome on both sides.

Forgetting the scheduler routing label. A job that omits schedulerName: volcano or the Kueue queue label silently falls through to the default scheduler, in a cluster everyone believes is protected.

Alerting on Pending pods instead of on Running jobs that are not stepping. Pending is the healthy state under contention. A Running job with no completed step is the pathology, and it is invisible to every stock alert.

Setting the pod-ready timeout shorter than a cold image pull. During a registry slowdown, every admitted job is evicted and requeued, which multiplies the load on the registry and turns a slow start into a cluster-wide requeue storm.

---