Why Databases Resist Orchestration

Kubernetes is built to replace a failing pod with an identical one. A database replica is not identical to its peers, it holds different data, and replacing it is a data operation rather than a scheduling one.

---

The Problem at Scale

The previous two lessons described symptoms: pods that cannot move, guarantees that stop short of failover. This lesson is the cause underneath both, and once you see it the rest of the course stops being a list of gotchas.

Kubernetes is a control loop. It compares desired state against observed state and takes action to close the gap. That model is extraordinarily effective and it rests on four assumptions about workloads:

Replicas are interchangeable. Any pod can serve any request, so replacing one with another is a no-op from outside.

Restarting is a valid repair. If a pod is unhealthy, kill it. The replacement starts clean and the problem is gone.

State lives elsewhere. The pod is a stateless function over a request. Whatever needs to persist is somebody else's concern.

Convergence is safe. Moving toward desired state is always an improvement, so acting quickly is better than acting slowly.

A database violates all four, and every awkward interaction in this course is one of those violations surfacing.

---

How It Works

Replicas are not interchangeable

Three pods behind a Deployment are three copies of the same function. Three pods in a database cluster are one primary and two replicas, each holding a slightly different amount of history, one of which is authoritative.

The consequences are structural rather than incidental.

Load balancing to any pod is wrong. Writes go to one specific pod. Reads may go to others, and only if the application can tolerate lag, which is the Module 4 material.

Pod ordinals carry meaning. In a Deployment the pod names are noise. Here postgres-0 may be the primary, or may have been at some point, and the ordinal is load bearing in ways nothing enforces.

Scaling is not symmetric. Adding a replica means provisioning a volume and then copying the entire dataset onto it before it is useful, which can take hours. Removing one may destroy quorum.

Restarting is not a repair

For a stateless pod, restarting clears the accumulated bad state and costs a few seconds. It is the universal first response and it is usually right.

For a database, a restart is a shutdown, and the quality of that shutdown determines whether you lose anything. A clean shutdown flushes buffers, completes in flight transactions and closes the write log properly. A SIGKILL after the grace period does none of that, and recovery on the next start means replaying the log to work out what was actually committed.

This is why terminationGracePeriodSeconds matters far more here than elsewhere. A default of thirty seconds on a database with a large buffer pool is a promise you cannot keep, and the kill that follows is the ungraceful path.

The instinct to restart a misbehaving pod is the single most dangerous reflex to bring from stateless operations, because it is correct nearly everywhere else.

State does not live elsewhere

The assumption that persistence is somebody else's problem is what makes stateless orchestration simple, and here you are that somebody else.

Everything that would normally be handled by the managed database your application talks to is now yours: durability, replication, consistency during failover, backups, recovery, capacity. Kubernetes provides scheduling and lifecycle. It provides nothing for any of that.

Convergence is not always safe

This is the subtlest one and it is where operators earn their place.

The control loop is optimised for speed. Something is not as desired, so act. For most workloads acting quickly is strictly better.

For a database, several correct actions require waiting rather than acting:

Promoting a replica is only safe once you know how far behind it is. Removing a member is only safe once you know quorum survives. Restarting is only safe once buffers are flushed. Reattaching a volume is only safe once you know the previous holder is genuinely gone, which is lesson one.

Each of those is a decision with a data consequence, and a control loop that converges eagerly will make it before it has the information. Operators are, in large part, the machinery for delaying convergence until it is safe.

---

Building and Operating It

Set the grace period from the workload rather than the default.

spec:
  # Long enough for a clean shutdown: flush buffers, finish in flight
  # transactions, close the write log. Thirty seconds is the default
  # and is a promise most databases cannot keep.
  terminationGracePeriodSeconds: 300
  containers:
    - name: postgres
      lifecycle:
        preStop:
          exec:
            # Ask the database to shut down properly rather than
            # relying on the signal reaching the right process.
            command: ["/bin/sh", "-c", "<the database's own shutdown command>"]

Protect against the disruptions that would take quorum:

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  # Voluntary disruptions only: drains and evictions. It does nothing
  # about node failure, which is the involuntary case from lesson one.
  minAvailable: 2
  selector:
    matchLabels: { app: postgres }

Make the roles visible, since Kubernetes will not show them:

# Label pods with their current role so kubectl output means something
kubectl get pods -l app=postgres -L role
# NAME         READY  STATUS   ROLE
# postgres-0   1/1    Running  replica
# postgres-1   1/1    Running  primary     <- not the ordinal you expect
# postgres-2   1/1    Running  replica

That last point matters more than it looks. Without role labels, every kubectl view of a database cluster shows three identical pods, and the person on call has no way to see which one matters.

---

Tradeoffs and Decision Framework

Kubernetes assumption	True for stateless	For a database	What has to fill the gap
Replicas interchangeable	Yes	No, one is authoritative	Role aware routing
Restart repairs	Yes	No, it is a shutdown	Grace periods, preStop, and restraint
State lives elsewhere	Yes	No, you are elsewhere	Backups, replication, recovery
Fast convergence is safe	Yes	No, some actions must wait	An operator, or a human

Two questions decide how much machinery you need. Does anything in your stack know which pod is authoritative, because if not, every automated action is being taken blind. And how long does a clean shutdown take, since that number is the grace period and the default is almost certainly wrong.

Default: grace period sized from a measured clean shutdown, a preStop hook that asks the database to stop properly, a disruption budget that protects quorum, and role labels so the cluster is legible in ordinary tooling.

---

Failure Modes and Common Mistakes

Restarting a database pod as a first response. Correct nearly everywhere else and a shutdown here.

The default grace period. Thirty seconds is not enough to flush a large buffer pool, so the kill is ungraceful and the next start is a recovery.

No disruption budget. A routine node drain evicts enough replicas to lose quorum, and nothing objected.

Load balancing writes across replicas. The Service selects all pods and only one accepts writes.

No role visibility. Three identical looking pods, and the person on call cannot tell which is the primary.

Assuming a disruption budget covers node failure. It governs voluntary evictions only, and the involuntary case is lesson one.