The Pod That Cannot Move

The node is cordoned and the pod will not reschedule. There is capacity in the cluster, the image pulls fine, and the pod stays Pending because its data is in a zone the scheduler is not allowed to leave.

---

The Problem at Scale

Kubernetes is built around a premise: a pod is disposable. If one dies, schedule another somewhere with room, and because the replacement is identical the system is healed. Every convenience in the platform, from rolling updates to node autoscaling to spot instances, rests on that premise.

A stateful pod breaks it, and it breaks it in a specific place. The pod is not disposable because it is attached to data, and that data is in one place.

The chain is worth making explicit, because each link removes a degree of freedom the scheduler had:

The pod is bound to a claim. Not to storage generally, to one specific PersistentVolumeClaim holding its data.

The claim is bound to a volume. A real disk that already exists somewhere.

The volume exists in a topology. A zone, or a single node for local storage. It cannot be in two places and, for most storage types, it cannot move.

Therefore the pod can only run where the volume is. Not a preference the scheduler weighs against others. A hard constraint it cannot violate.

---

How It Works

The zone was chosen by an event nobody remembers

Zone binding usually happens by accident, and it is worth understanding because it explains a lot of later inflexibility.

When a volume is provisioned dynamically it has to be created somewhere. If provisioning happens the moment the claim is created, before any pod exists to need it, the storage system picks a zone with no knowledge of where the workload will eventually run. That choice is then permanent, and every pod that ever uses that claim is pinned to it.

volumeBindingMode is the setting that decides this, and its default is not the one most platforms want. Immediate binding provisions as soon as the claim exists. WaitForFirstConsumer defers provisioning until a pod is actually scheduled, so the volume is created where the workload landed rather than the workload being dragged to where the volume landed.

The failure this prevents is a large one: a workload whose replicas are all in a single zone, giving you no zone redundancy at all, because the first claim was created in that zone and everything since has followed it.

The node failure that does not heal

This is the behaviour that surprises people most, and it is not a bug.

When a node stops responding, the control plane cannot distinguish a node that has died from a node that is merely unreachable. For a stateless workload that ambiguity is cheap: schedule a replacement, and if the original comes back it is redundant.

For a stateful workload the same assumption is dangerous. If the old pod is still running and still writing to that volume, starting a second pod against the same volume risks two writers on one filesystem, which is how data gets corrupted rather than merely duplicated.

So Kubernetes does the safe thing and refuses to guess. StatefulSet pods on an unreachable node enter an unknown state and stay there, not for a timeout, but until something establishes what actually happened. The pod does not reschedule, the volume is not released, and the workload stays down.

The resolution requires someone or something to assert that the old pod is genuinely gone, which is a decision with consequences and therefore not one the scheduler makes on its own.

What this removes

Once you internalise the constraint, a set of otherwise puzzling behaviours all become the same behaviour:

Node drains stall. A drain evicts pods and waits for them to become ready elsewhere. A stateful pod whose volume is on the draining node has nowhere to become ready.

Cluster autoscaling gets stuck. A node holding a bound volume cannot be removed, so it lingers underutilised.

Spot instances become dangerous. Reclaim is exactly the ambiguous node loss described above.

Zone rebalancing does not exist. Nothing will move your workload to fix an imbalance, because nothing can move the data.

Rolling updates are slower and riskier. Each replacement must wait for the volume to detach and reattach, which is a real operation with its own failure modes.

---

Building and Operating It

Find out where your volumes actually are, since most teams have never looked.

# The zone every bound volume lives in. Concentration here is a
# redundancy problem that no pod-level anti-affinity can fix.
kubectl get pv -o custom-columns=\
NAME:.metadata.name,\
CLAIM:.spec.claimRef.name,\
ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values

Check the binding mode on the classes teams are actually using:

kubectl get storageclass -o custom-columns=\
NAME:.metadata.name,\
BINDING:.volumeBindingMode,\
DEFAULT:.metadata.annotations."storageclass\.kubernetes\.io/is-default-class"

Immediate on the default class is worth changing before it produces a single zone workload nobody intended.

Diagnose a pending stateful pod by asking the scheduler why:

kubectl describe pod <pod> | tail -20
# "node(s) had volume node affinity conflict" is the constraint in
# this lesson: the volume is somewhere the pod is not allowed to be.

---

Tradeoffs and Decision Framework

Storage choice	Mobility	Durability	Performance	Cost of the constraint
Network attached, zonal	Within one zone	Survives node loss	Good	Pinned to a zone permanently
Network attached, regional	Across zones	Survives zone loss	Lower, replicated writes	Expensive, and still not free to move
Local disk	None, pinned to a node	Dies with the node	Best available	Node loss is data loss
Ephemeral	None	Dies with the pod	Best	Not persistence

Two questions shape every stateful deployment. Where can this workload legally run, which the volume already decided, and what happens when that place becomes unavailable, which is a design decision rather than something the platform handles.

Default: WaitForFirstConsumer on every StorageClass so volumes follow workloads rather than the reverse, zonal network storage unless the workload genuinely needs local disk performance, and an explicit decision about node loss recorded before it happens.

---

Failure Modes and Common Mistakes

Immediate binding producing a single zone deployment. Every replica in one zone because the first claim landed there, and pod anti-affinity cannot fix a storage decision.

Force deleting stuck StatefulSet pods. Overrides the protection against two writers on one volume, which is the failure it exists to prevent.

Expecting node failure to self heal. For stateful workloads it deliberately does not, and waiting for it to is waiting for something that will not happen.

Local storage without accepting node loss is data loss. The performance is real and so is the consequence.

Cluster autoscaler configured without stateful awareness. Nodes holding bound volumes cannot be reclaimed, so the fleet drifts underutilised.

Assuming anti-affinity gives zone redundancy. It spreads pods and does nothing about where their data already is.