What a StatefulSet Actually Guarantees

The database went down and the StatefulSet did exactly what it promised. It kept the names stable, kept the volumes attached, and had no opinion whatsoever about which replica was the primary.

---

The Problem at Scale

The name is doing damage. A StatefulSet sounds like the Kubernetes answer to running stateful software, and it is not. It is a controller that solves three specific problems, all of them about identity and ordering, and none of them about the data.

Here is the whole list of what it guarantees:

Stable network identity. Pod zero is always name-0, with a DNS name that survives restarts and rescheduling.

Stable storage. Each pod gets its own claim, and that claim follows the ordinal rather than the pod instance. name-0 reattaches to the volume name-0 had before.

Ordered operations. Pods are created in order and terminated in reverse, and by default each step waits for the previous one to be ready.

That is the complete set. Everything else people expect is application behaviour that something else must provide.

---

How It Works

What people assume it handles

Each of these is a real operational need, each is commonly assumed to be covered, and none of it is:

Failover. Nothing elects a primary. If pod zero is your primary and it dies, the StatefulSet restarts pod zero and waits. It will not promote pod one, because it does not know what a primary is.

Replication health. Whether replicas are caught up is invisible to the controller. A replica lagging by an hour is Ready as far as Kubernetes is concerned, provided its probe passes.

Membership. Adding a replica creates a pod. Whether the cluster inside those pods knows about the new member, and whether it has been given data, is the application's problem.

Backups. No relationship to the controller at all.

Version upgrades. Changing the image rolls the pods. Whether the on disk format needs migrating, and in what order relative to the rolling, is not modelled.

Quorum. Scaling from three to two is a pod count change. That it may have just destroyed a majority is not something the controller can know.

Where the ordering guarantee helps and where it hurts

Ordered rolling is genuinely useful for stateful software. Replacing every replica simultaneously would destroy the cluster, and the default sequential behaviour prevents that.

It is also the source of a specific stall. Because each step waits for readiness, a pod that never becomes ready halts the rollout indefinitely, and the pods behind it are never touched. That is usually correct, since continuing to roll into a broken cluster is worse, and it means a bad configuration change leaves you with a partially rolled StatefulSet and a controller waiting patiently.

Worth knowing: podManagementPolicy: Parallel disables the ordering for creation and scaling, which is right for workloads whose members are genuinely independent and wrong for most databases.

The readiness probe is doing more than it looks

For a stateless workload a readiness probe answers whether this pod can serve traffic. For a stateful workload it is also the gate on the rolling update, which gives it a second job it is rarely designed for.

That produces a decision most teams make by accident. A probe that reports ready as soon as the process starts lets a rollout proceed past a replica that is still recovering or still syncing, so the next replica goes down while the previous one is not actually usable. A probe that reports ready only when fully caught up is more correct and can stall a rollout for a long time on a replica with a lot to replay.

Neither is wrong in general. The point is that the probe is the mechanism controlling how fast your database is allowed to be taken apart, and it deserves to be designed rather than copied.

Volumes outlive the pods, and the StatefulSet

This one is a safety feature that reads as a bug until you understand it.

Deleting a StatefulSet does not delete its PersistentVolumeClaims. Scaling from five replicas to three does not delete the claims for pods three and four. The data is deliberately retained, so scaling back up reattaches the original volumes.

The consequence is that orphaned claims accumulate quietly, still provisioned, still costing money, invisible on any dashboard showing running workloads. And a claim retained from an old deployment will be reattached by a new one with the same name, which surprises people who expected a fresh start.

---

Building and Operating It

Check what the controller thinks against what the application thinks, since they can disagree completely.

# Kubernetes view: all three ready
kubectl get statefulset postgres
kubectl get pods -l app=postgres

# Application view: who is primary, and are the replicas caught up?
# There is no Kubernetes command for this, which is the lesson.
kubectl exec postgres-0 -- <the database's own status query>

The absence of a kubectl answer to the second question is the point. If nothing in your stack answers it, nothing is managing failover.

Find the orphaned claims:

# Claims with no pod using them. Retained on purpose, and forgotten.
kubectl get pvc -o json | jq -r '
  .items[] | select(.status.phase=="Bound") |
  "\(.metadata.namespace)/\(.metadata.name)"'

Set the ordering policy deliberately rather than inheriting it:

spec:
  # OrderedReady is the default and is correct for most databases.
  # Parallel is for members that are genuinely independent.
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      # Hold the rollout after the first pod so a bad change does not
      # walk through every replica while you are watching it.
      partition: 2

---

Tradeoffs and Decision Framework

Need	StatefulSet provides	Who actually provides it
Stable names	Yes	The controller
Stable volumes per ordinal	Yes	The controller
Ordered create, delete, update	Yes	The controller
Primary election and failover	No	An operator, or the database itself
Replication lag awareness	No	The database, surfaced by you
Cluster membership	No	An operator, or an init procedure
Backups	No	A separate system, Module 5
Safe version upgrades	No	An operator, or a documented procedure
Quorum awareness on scale	No	Nothing, unless you build it

Two questions decide whether a bare StatefulSet is enough. Does this workload need coordinated failover, because if it does then something else has to provide it. And is the readiness probe telling the truth about readiness, since it is gating both traffic and the rollout.

Default: a bare StatefulSet only for workloads whose members are independent or whose clustering is entirely self managed, an operator for anything with a primary, and an explicit owner named for failover in either case.

---

Failure Modes and Common Mistakes

Expecting automatic failover. The most consequential assumption in this course, and it surfaces during the first primary loss.

A readiness probe that lies. Ready before the replica is usable lets a rollout take down the next one while the previous is not serving.

A rollout stalled on an unready pod, read as a bug. It is the ordering guarantee working, and continuing would be worse.

Scaling down through quorum. Three to two is a pod count change to the controller and a majority to the cluster inside.

Orphaned claims accumulating. Retained by design, invisible in workload views, and still billed.

Parallel management on a database. Removes the ordering that stops every member being replaced at once.