The Replica Is the Unit, Not the Pod

A model too large for one node turns a single serving replica into four pods that only mean anything together. Every assumption your Deployment made just stopped being true.

---

The Problem at Scale

The previous lesson took apart the training side: the job is the unit, a partially placed gang is worth nothing, and the default scheduler's unit of decision is smaller than your unit of work. The serving side has the same disease and almost nobody expects it, because serving is where the pod-as-unit assumption came from in the first place.

A Deployment makes three promises, and all three are load-bearing everywhere else in Kubernetes. Pods are interchangeable, so any one can replace any other. Each pod is independently useful, so a pod that passes its readiness probe can serve a request. The replica count is a target that a controller drives toward one pod at a time, so scaling, rolling, and repairing are all the same operation applied repeatedly.

Now put a model on the fleet whose weights do not fit on one node. Eight accelerators at 80 GB each gives a node roughly 640 GB of device memory, and after the runtime and the key-value cache reservation you have meaningfully less than that for parameters. A model that needs a terabyte and a half of weights is therefore a three or four node workload, and the pods on those nodes are not three or four replicas. They are one replica, sharded, and the shards are useless apart. Rank zero holds the first slice of every layer and cannot produce a token without the others.

Every one of the three promises is now false, and the failures are specific.

Interchangeability is gone: worker pod 2 of group 5 cannot substitute for worker pod 2 of group 6, because each group has already formed a communication topology and loaded its own slice of weights. Independent usefulness is gone: twelve of your sixteen pods can serve nothing at all, ever, and if they land in a Service's endpoints they will accept connections that hang. And the one-at-a-time control loop is now actively wrong: deleting a single worker pod does not degrade a replica by a quarter, it takes the replica to zero, and the controller helpfully replaces exactly that one pod, which comes up healthy and joins nothing.

$ kubectl get pods -n serve -l app=llm-xl --no-headers | wc -l
16

$ kubectl get endpointslice -n serve -l kubernetes.io/service-name=llm-xl \
    -o jsonpath='{range .items[*].endpoints[*]}{.targetRef.name}{" "}{.conditions.ready}{"\n"}{end}' \
  | head -4
llm-xl-0        true
llm-xl-0-1      true
llm-xl-0-2      true
llm-xl-0-3      true

Four groups of four pods, and all sixteen are in the endpoint list as ready. Twelve of them cannot answer a request. Three quarters of the traffic this Service receives is going somewhere it will never come back from, and every component involved reports success.

---

How It Works

The group needs an identity, not a count

The first thing a multi-pod replica needs is a name for the group and a stable position within it, because the pods have to find each other before they can do anything. A Deployment supplies neither. A StatefulSet supplies ordinal identity and stable DNS, which is closer, but it still treats the set as one flat population: it has no concept of "pods 4 through 7 are one thing," so its rolling update, its scaling, and its disruption behavior are all still per pod.

LeaderWorkerSet is the object that closes the gap. You declare a group size and a replica count, and the controller creates that many groups, each with one leader and size-minus-one workers, each group carrying a group index label and a per-group headless Service so the workers can resolve the leader.

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: llm-xl
  namespace: serve
spec:
  replicas: 4                       # four replicas...
  leaderWorkerTemplate:
    size: 4                         # ...of four pods each. 16 pods, 4 units.
    restartPolicy: RecreateGroupOnHostFailure
    leaderTemplate:
      spec:
        containers:
          - name: server
            resources:
              limits:
                nvidia.com/gpu: 8
    workerTemplate:
      spec:
        containers:
          - name: shard
            resources:
              limits:
                nvidia.com/gpu: 8

restartPolicy: RecreateGroupOnHostFailure is the line that encodes the mental model, and it is the exact serving counterpart of the RestartJob policy a training gang sets on pod failure. Losing one member means the group is dead, so tear the group down and rebuild it rather than replacing a member into a topology that has already moved on. The alternative produces a pod that starts, reports healthy, and belongs to nothing.

Readiness is a property of the group

This is where the serving case diverges sharply from training, and it is the part most teams get wrong on the first attempt. A training gang has no readiness problem because nobody is sending it traffic. A serving replica does, and getting it wrong means routing requests into a hole.

Only the leader should appear in the Service's endpoints, because the leader is the only pod that speaks the request protocol. That part is easy: label the Service selector so it matches leaders only. The hard part is what the leader's readiness probe checks. A naive probe on the leader's HTTP port passes as soon as the process binds the socket, which happens long before the workers have connected and loaded their shards. The leader is then marked Ready, endpoints are updated, traffic arrives, and every request blocks waiting on a shard group that is still pulling weights.

The probe has to interrogate the group, not the process.

readinessProbe:
  httpGet:
    path: /health/ready             # must return 200 only when every shard
    port: 8000                      # has registered and loaded its slice
  periodSeconds: 5
  failureThreshold: 2
startupProbe:
  httpGet:
    path: /health/ready
    port: 8000
  periodSeconds: 10
  failureThreshold: 120             # weight load can take many minutes

The startup probe with a long failure threshold and the readiness probe with a short one are doing different jobs. The startup probe tolerates a cold start measured in minutes without letting the readiness threshold declare a healthy pod dead. The readiness probe then reacts quickly once serving has begun, so a group that loses a shard leaves the endpoint list in seconds rather than continuing to accept requests it cannot answer. Getting these backwards produces the two classic failures: a replica killed halfway through loading weights, or a broken replica that keeps taking traffic.

---

Building and Operating It

Operate on groups. Every command, dashboard, and alert that counts pods is telling you something that is not capacity.

$ kubectl get lws -n serve
NAME     READY   UPDATED   AVAILABLE   AGE
llm-xl   3/4     4         3           6h12m

$ kubectl get pods -n serve -L leaderworkerset.sigs.k8s.io/group-index \
    --no-headers | awk '{print $NF, $3}' | sort | uniq -c
      4 0 Running
      4 1 Running
      4 2 Running
      3 3 Running
      1 3 Pending

Three of four replicas available, and group 3 is one pod short. Notice that fifteen of sixteen pods are Running, which is 94 percent by the metric your existing dashboard uses and 75 percent of the capacity you actually have. Report ready groups over desired groups, and never report pod counts to anyone making a capacity decision.

# Serving capacity, counted correctly
sum by (model) (leaderworkerset_status_ready_replicas)
  / sum by (model) (leaderworkerset_spec_replicas) < 1

# The pathology: a leader that is Ready while its group is incomplete
count by (group) (kube_pod_status_phase{phase="Running", group!=""}) != 4
  and on (group) leaderworkerset_leader_ready == 1

Rollouts need the same correction. A rolling update expressed in pods will take one worker out of a live group, which does not update that group, it breaks it. The rollout configuration has to count in groups.

spec:
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdateConfiguration:
      maxSurge: 1                   # one extra whole group during the roll
      maxUnavailable: 0             # never take a serving group down first

maxSurge: 1 with maxUnavailable: 0 means you need one replica's worth of spare GPUs to roll at all, which on a four-pod replica is thirty-two GPUs held aside during every deploy. That is a real cost and it is the honest one: the alternative is losing a quarter of your capacity for the duration of each group's cold start, and on a model whose weights take minutes to load that window is long enough to matter. Lesson 5.1 is about shortening it.

Then the disruption path, per group rather than per deployment.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-xl-group-3
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      leaderworkerset.sigs.k8s.io/name: llm-xl
      leaderworkerset.sigs.k8s.io/group-index: "3"

Scaling deserves one explicit warning, because it is where the group abstraction most often leaks back out. An autoscaler pointed at this workload must change the replica count, which the controller expands into whole groups. Point one at the pod count instead and it will add pods that belong to no group and subtract pods from groups that are serving. Anything you attach to a multi-pod replica has to speak in replicas, and that includes the HPA, KEDA, and any home-grown scaler someone wrote against pod metrics two years ago.

Two boundaries are worth stating so you know where to look next. The mechanics of sharding a model across hosts, the inference engine, and the key-value cache belong to the Production LLM Inference on Kubernetes course; this lesson is only about what the grouping does to the platform. And the admission side, where a scale-up that cannot place a whole group must fail cleanly instead of stranding GPUs, is lesson 3.3.

---

Tradeoffs and Decision Framework

Mechanism	Expresses	Does not express	Use when
Deployment	A pod population with a target count	Grouping, group readiness, group replacement	The replica genuinely fits in one pod
StatefulSet	Stable identity and ordered DNS	That a subset of ordinals is one unit	You need identity but the pods are independent
LeaderWorkerSet	Group size, group index, group rollout, group restart	Quota and cross-workload admission ordering	Any replica that spans more than one pod
Volcano Job for serving	Gang admission you already run for training	Endpoints, readiness gating, rolling updates	You want one admission path and will build the serving parts
Manual grouping with labels	Whatever you implement	Everything, until you implement it	Never, past a proof of concept
Fit the model on one node	All of the above problems disappear	The model you were asked to serve	Quantization or a smaller variant meets the quality bar

Four questions settle the design. Does the model actually need more than one node, because the cheapest fix for everything in this lesson is a variant that fits in one and the quality tradeoff is often smaller than the operational one? Is every controller that can delete a pod in this namespace aware of groups, since one that is not will eventually cost you a whole replica? Are your capacity alerts denominated in groups, because a pod-denominated alert understates the loss by exactly the group size? And do you have somewhere for a rebuilt group to land, given that a group needs whole nodes and the pool may not have them free.

The default: use LeaderWorkerSet the moment a replica exceeds one pod, gate the endpoint on a leader probe that verifies every shard, count rollouts and disruption budgets in groups, and hold one spare group of capacity in the pool so a rebuild is never blocked.

---

Failure Modes and Common Mistakes

Putting every pod of the group in the Service selector. Workers accept connections they cannot answer, and a fraction of traffic equal to the worker share of the group disappears into hanging requests.

Probing the leader's socket instead of the group's state. The leader binds its port before the shards have loaded, gets marked Ready, and receives traffic during a cold start that can run for minutes.

Replacing a single failed pod instead of rebuilding the group. The new pod starts healthy and joins nothing, so the group stays broken while every object in the cluster reports the desired state.

Writing PodDisruptionBudgets in pod percentages. Two evictions spread across two groups remove two whole replicas, which the percentage was never meant to permit.

Rolling updates counted in pods. Taking one worker out of a live group does not update it, it destroys it, and the rollout then proceeds to do the same to every other group.

Alerting on pod readiness. Fifteen of sixteen pods Running is 94 percent by pod count and 75 percent by capacity, and the gap is exactly the size of your incident.

Leaving no room for a rebuilt group. A group needs whole nodes simultaneously, so a pool with scattered free GPUs can be unable to reconstruct a replica it just lost.

Assuming this is only a serving problem. It is the same all-or-nothing structure as a training gang, arriving on a different timescale and with the additional requirement that a partial unit must never receive traffic.

---