The Recovery Decision Tree

It is 2 AM, the cluster is down, and you need to know within five minutes whether you are recovering it or rebuilding it. Four questions get you there.

---

The Problem

The previous lesson established what cannot be regenerated. This one turns that into a triage procedure you can run under pressure, because knowing the model and applying it at 2 AM are different skills.

The four questions are: is there a surviving control plane peer, did the etcd data survive, did the CA private keys survive, and do you have a backup you have actually validated. Answer those in order and every incident in this course lands on one of three verdicts. None of the four requires a working API server. All four are answerable from a shell on a node in under five minutes.

The reason to insist on this discipline is that the alternative is what most teams actually do, which is to start typing. Someone restarts the kubelet. Someone else restarts containerd. A third person runs kubeadm reset on the node that will not come up, and the incident becomes unrecoverable at that moment, forty minutes in, because that node held the only copy of ca.key. The overwhelming majority of permanent losses in Kubernetes incidents happen after the initial failure, committed by the responder. Module 1's last lesson is entirely about that; the tree exists so that you know what you are protecting before you have a chance to destroy it.

There is a second reason, and it is about time budget rather than safety. The three verdicts have wildly different shapes. A RECOVERABLE incident is a procedure with a known end: you follow the steps, the cluster comes back whole, and the postmortem is about detection. A RECOVERABLE WITH LOSS incident has a communication problem attached, because someone outside the incident channel needs to be told that every service account token is about to be invalidated or that four hours of writes are gone. An UNRECOVERABLE verdict is not an incident at all, it is a project, and the sooner you say so the more of the dying cluster you can salvage while access still works. Choosing the wrong shape costs hours. Running a repair procedure for ninety minutes on a cluster that needed a rebuild is the single most common way a four-hour outage becomes a twelve-hour one.

---

How It Works

Before the four questions, spend thirty seconds on a prior one that reroutes the whole tree: are you locked out, or have you lost something? They look identical from a terminal and they have nothing in common. A lockout means every byte of state is intact and you simply cannot authenticate or authorize; the fix is Module 4 and the verdict is always RECOVERABLE. A loss means something is gone. The error text tells you which you are in:

$ kubectl get nodes
The connection to the server api.example.internal:6443 was refused - did you
specify the right host or port?

$ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "kubernetes-admin"
cannot list resource "nodes" in API group "" at the cluster scope

$ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority

A refused connection means no process is listening: the control plane is down, and the tree applies. A Forbidden means the API server is running, TLS succeeded, and it authenticated you, so nothing is lost and you have an RBAC problem. An x509 error means the process is up and the trust relationship is broken, which is a PKI problem and usually an expiry, covered by the Production Kubernetes Operations course. Only the first of those three sends you down the tree.

Q1: is there a surviving control plane peer

Ask this first because a healthy peer is a complete answer to the next two questions at once. Every control plane node in a kubeadm cluster carries its own full copy of the PKI, including all three CA private keys and the service account signing key, and its own full etcd member holding the entire keyspace. One surviving peer means you have lost nothing irreplaceable, whatever happened to the other nodes.

$ ssh cp-node-02 'sudo crictl ps --name kube-apiserver'
CONTAINER      IMAGE          CREATED       STATE     NAME
a3f81c7b29de   4a2f7b18cc90   6 days ago    Running   kube-apiserver

$ ssh cp-node-02 'sudo ls /etc/kubernetes/pki/ca.key'
/etc/kubernetes/pki/ca.key

If a peer is alive, the remaining question is whether etcd still has quorum, which is separate from whether nodes are up. Three members tolerate one failure; five tolerate two. Losing the majority stops writes even though the surviving member holds every byte of the data:

$ ETCDCTL_API=3 etcdctl \
    --endpoints=https://10.0.0.11:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    member list --write-out=table
+------------------+---------+------------+-------------------------+
|        ID        | STATUS  |    NAME    |       CLIENT ADDRS      |
+------------------+---------+------------+-------------------------+
| 8e9f2a1c4d6b3057 | started | cp-node-01 | https://10.0.0.10:2379  |
| 1b47c0e5a9f36d82 | started | cp-node-02 | https://10.0.0.11:2379  |
| c25d8f30b6e14a79 | started | cp-node-03 | https://10.0.0.12:2379  |
+------------------+---------+------------+-------------------------+

Read that carefully: member list shows the configured membership, not liveness. A member that has been dead for a week still appears as started, because the field describes what the cluster believes about the member rather than whether it responded just now. Liveness comes from endpoint status with --cluster, which actually contacts each endpoint. Confusing the two is how people conclude their quorum is fine while the cluster is refusing writes.

Q2: did the etcd data survive

The useful form of this question is not "does the directory exist" but "will etcd load it". A data directory can be entirely present and entirely unusable, most often because the write-ahead log was truncated by a power loss or a filesystem that lied about a flush.

$ ls /var/lib/etcd/member/
snap  wal

$ du -sh /var/lib/etcd
1.4G    /var/lib/etcd

$ journalctl -u kubelet --since -30m | grep -i etcd | tail -3
Jun 12 02:07:44 cp-node-01 kubelet[1184]: E0612 02:07:44.913 pod etcd-cp-node-01
  container etcd exited with 1: walpb: crc mismatch

Both a snap and a wal subdirectory with a plausible size is a good sign. A crc mismatch, panic: freepages, or database file is corrupted in the container logs means treat the answer as no and move to Q4. The dangerous case is the middle one, where etcd starts, serves reads, and is missing recent revisions; that is why the next lesson insists you check application-visible reality rather than trusting a green control plane.

Q3: did the CA private keys survive

Four files decide this. Three CA private keys and the service account signing key, in the layout the previous lesson walked through:

$ ls -l /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/sa.key \
        /etc/kubernetes/pki/front-proxy-ca.key /etc/kubernetes/pki/etcd/ca.key
-rw------- 1 root root 1679 Jun  6 09:12 /etc/kubernetes/pki/ca.key
-rw------- 1 root root 1675 Jun  6 09:12 /etc/kubernetes/pki/etcd/ca.key
-rw------- 1 root root 1675 Jun  6 09:12 /etc/kubernetes/pki/front-proxy-ca.key
-rw------- 1 root root 1679 Jun  6 09:12 /etc/kubernetes/pki/sa.key

Data plus keys is RECOVERABLE, and Module 2 covers the procedure. Data without keys is RECOVERABLE WITH LOSS: your objects all come back, the cluster's identity does not, and the cost is that every node rejoins and every service account token is invalidated at the moment you swap the signing key.

Q4: is there a backup you have validated

This question is only worth asking in its strict form, because the weak form has a near-100 percent false positive rate. A backup job that reports success is not a validated backup. A file in object storage with the right name is not a validated backup. A snapshot whose header you have read is:

$ etcdutl snapshot status /mnt/backups/etcd-2026-06-12-0100.db --write-out=table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 3f1ac207 |  8471192 |      14038 |     1.3 GB |
+----------+----------+------------+------------+

Two things in that output matter and neither is the hash. TOTAL KEYS should be within a plausible range of your cluster's object count; a snapshot reporting a few hundred keys for a cluster running thousands of pods captured something other than what you think. TOTAL SIZE should be near your live database size. On etcd 3.5 and later this subcommand moved to etcdutl, and etcdctl snapshot status still works but prints a deprecation warning, which is worth knowing before you burn ten minutes at 2 AM deciding whether the warning is the reason your restore is failing. It is not.

---

Doing It Safely

The tree is read-only by design, and preserving that property is the whole discipline. Three specific traps convert a diagnostic into a mutation.

Starting etcd to see whether it works is a write. etcd replays and may compact its write-ahead log on startup, so an attempt to "just check" a suspect data directory can destroy the evidence you would have used for a forensic recovery. Copy the directory first, with the process stopped, then experiment on the copy:

$ sudo systemctl stop kubelet
$ sudo cp -a /var/lib/etcd /mnt/backups/etcd-datadir-cp-node-01-preinspect
$ sudo du -sh /mnt/backups/etcd-datadir-cp-node-01-preinspect
1.4G    /mnt/backups/etcd-datadir-cp-node-01-preinspect

Restarting the kubelet on a control plane node is a write. It re-reads the manifest directory and restarts static pods, which is normally what you want and is occasionally the action that takes a limping API server away from you before you have read the kubeadm-config ConfigMap out of it. Read what you need from a live API first; there is no rule that says diagnosis has to happen before extraction.

The tree does not tell you to fix anything. Its output is a verdict and a module number. Resist the pull to start the repair while you are still three questions in, because the last question is frequently the one that changes the plan.

---

Tradeoffs and Decision Framework

Each combination of answers has exactly one verdict. The table is the tree flattened, which is easier to scan when you already have your four answers.

Peer alive	etcd data	CA keys	Backup	Verdict and where to go
Yes, quorum intact	n/a	n/a	n/a	RECOVERABLE. Replace the dead node, Module 2
Yes, quorum lost	On a survivor	Yes	Any	RECOVERABLE WITH LOSS. Force a new cluster or restore, Module 3
No	Survived	Survived	Any	RECOVERABLE. Rebuild the control plane around the data, Module 2
No	Survived	Lost	No PKI backup	RECOVERABLE WITH LOSS. New CAs, every node rejoins, Module 2
No	Lost	Survived	Snapshot exists	RECOVERABLE WITH LOSS. Restore, lose writes since the snapshot, Module 3
No	Lost	Lost	Nothing validated	Rebuild. Salvage while you still have access, Module 10

Three questions settle the ones the table cannot. Is the encryption at rest key in your possession, because if it is not, the Secrets in an otherwise perfect restore are permanently unreadable and the verdict on those objects is UNRECOVERABLE regardless of everything else. Is the workload state in Git and continuously reconciled, because that changes the cost of a rebuild from weeks to days without changing any verdict above. And is anyone actually affected right now, which the next lesson answers and which decides how much risk you are entitled to take in the repair.

Default: when two paths lead to the same verdict, take the one that preserves more evidence. Forcing a new etcd cluster from a survivor and restoring from a snapshot both land on RECOVERABLE WITH LOSS, but the survivor holds every write up to the failure while the snapshot is by definition older, and the survivor's data directory still exists after a failed restore attempt whereas a restore performed in place does not. Prefer the reversible path even when it is slower, because the irreversible one has no second attempt.

---

Common Mistakes

Starting the repair before finishing the tree. The fourth question changes the plan often enough that answering three and acting is a coin flip. All four take five minutes together.

Treating etcdctl member list as a health check. It reports configured membership, not liveness, and a member that died last week still shows as started. Use endpoint status --cluster when you need to know who is actually answering.

Skipping Q0 and treating a lockout as a disaster. A Forbidden error means the API server is up, TLS worked, and it knows who you are, so nothing is lost. Responders who miss this have restored etcd to fix an RBAC binding they deleted, which is a catastrophic answer to a five-minute problem.

Counting a backup you have never read. The failure mode is not a missing file, it is a file containing a snapshot of the wrong cluster, an empty database, or ciphertext you cannot decrypt. Read the header before you count it as a yes.

Forgetting that a healthy peer is a copy of the PKI. Teams with no PKI backup and three control plane nodes routinely have three copies of everything irreplaceable and do not realize it, then rebuild from scratch after losing one node.

Running the tree on the broken node only. The questions are about the cluster, not the host. ca.key on any control plane node answers Q3, and a PKI archive in object storage answers it too.

Letting the verdict drift during the incident. Once you have a verdict, changing it requires new evidence, not new anxiety. A written verdict in the incident channel is what makes that visible when a third responder joins at hour two with a fresh theory.

Refusing to say the word rebuild. An UNRECOVERABLE verdict declared at minute twenty leaves you hours of API access to extract manifests, Secrets, and data. Declared at hour six, it usually does not.

---