The Failure Modes With No Stateless Equivalent

Restarting a stateless pod fixes it. Restarting this one brought back a replica holding data from before the incident, and it rejoined the cluster and started serving it.

---

The Problem at Scale

Stateless failures are, at bottom, all the same failure: something is not serving. The pod is down, or slow, or erroring, and the response is to replace it or route around it. The blast radius is bounded by the request in flight.

Stateful systems have a second category, and it is the one that produces the incidents people still talk about years later. Failures where the system continues serving and the data is wrong.

These have three properties that make them qualitatively harder.

They are silent. No error rate moves, no probe fails, and the system reports itself healthy.

They are not fixed by restarting. The damage is on disk. Replacing the process replaces nothing.

They can be permanent. A stateless outage ends when service resumes. Lost or corrupted data does not come back because the pods came back.

---

How It Works

Split brain

Two members both believe they are the primary and both accept writes.

The usual cause is a network partition. Each side can see itself and not the other, and each concludes the other is dead. If the failover mechanism promotes without confirming it holds a majority, both sides now have a primary.

The damage is done during the partition and discovered afterwards. Two divergent write histories exist, both durable, both legitimate from their side's perspective. Healing the partition does not merge them, because there is no correct merge: two customers were assigned the same identifier, two balances were decremented from the same starting point.

Recovery means choosing which history to keep and discarding the other, which is a business decision rather than a technical one, and someone has to make it with incomplete information.

The prevention is quorum. A member only accepts writes while it can see a majority, so a minority partition refuses to serve rather than serving wrongly. That is why an even member count is a poor choice and why scaling a three member cluster down to two is more dangerous than it looks.

The returning stale replica

The scenario in this lesson's opener, and the one that catches teams who did everything else right.

A replica is partitioned or shut down. The cluster carries on without it. Later it comes back, and if nothing checks how far behind it is before readmitting it, it rejoins holding an old view of the world and begins serving reads from it.

Users see data disappear and reappear depending on which replica answered. Nothing is down. Every probe passes.

The prevention is that rejoining is a decision, not an event. A returning member has to establish how far behind it is and catch up before serving, and a member that is too far behind should be rebuilt from scratch rather than allowed to catch up incrementally.

Corruption replicates faithfully

Replication is designed to propagate writes. It does not distinguish a write you meant from a write you did not.

A bad migration, an application bug, an accidental delete: whatever it does to the primary is dutifully applied to every replica within milliseconds. Replicas are not a defence against logical corruption. They are an amplifier for it.

This is the single strongest argument for backups in a replicated system, and it is the argument that gets waved away by the observation that the data exists in three places. It exists in three places, identically wrong.

What defends against this is a copy from before the event, which means backups with history, which is Module 5.

Silent data corruption

Rarer, and worth knowing exists. Bit rot on disk, a firmware bug, a filesystem issue: data changes without any component reporting an error. Reads return successfully with wrong content.

Checksums are the defence, at the storage layer, the filesystem layer or the database layer. Whether your stack has them is a question worth having answered.

---

Building and Operating It

Detect the failures that pass probes, since nothing in the default stack will.

# Split brain: how many members think they are primary?
# Anything other than exactly one is an incident.
for p in $(kubectl get pods -l app=postgres -o name); do
  echo "$p: $(kubectl exec $p -- <role query>)"
done

# Replication lag per replica. The single most valuable stateful
# metric, and it is application level so nothing emits it for you.
max by (pod) (db_replication_lag_seconds)

# More than one primary. Alert immediately, at any duration.
count(db_role == 1) != 1

Gate readiness on being caught up rather than on being alive:

readinessProbe:
  exec:
    # A replica that is running but hours behind should not serve reads.
    # This is the check that prevents the returning stale replica.
    command: ["/bin/sh", "-c", "<lag check, fail if beyond threshold>"]
  periodSeconds: 10

Keep quorum out of reach of routine operations:

# Odd member count so a majority is always well defined
replicas: 3
---
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2      # a majority of three

---

Tradeoffs and Decision Framework

Failure	Detected by	Prevented by	Recoverable?
Split brain	Multiple members claiming primary	Quorum, odd member counts	Only by discarding one history
Stale replica readmitted	Replication lag monitoring	Lag gated readiness, rebuild past a threshold	Yes, remove and rebuild it
Logical corruption	Application level validation	Nothing replication offers	Only from a backup predating it
Silent corruption	Checksums	Checksums at some layer	From a backup, if detected

Three questions establish your exposure. Can two members serve writes simultaneously, which is a quorum question and has a yes or no answer. Does anything check replication lag before a replica serves reads, since that is the returning stale replica. And do you have a copy from before a logical error, because replicas will not be one.

Default: odd member counts with quorum enforced, readiness gated on replication lag rather than liveness, an alert on more than one primary at any duration, and backups justified as the only defence against corruption that replicates.

---

Failure Modes and Common Mistakes

Treating replicas as backups. They replicate corruption faithfully and within milliseconds.

Even member counts. A majority is ambiguous, which is exactly what split brain needs.

Readiness gated on liveness. A replica that is running and hours behind passes and serves stale reads.

Availability monitoring as correctness monitoring. Every failure in this lesson keeps the service up.

Scaling to two replicas to save cost. Quorum in a two member cluster is both members, so any single failure stops writes.

Assuming a healed partition merges. It does not. Two histories exist and one has to be chosen.