All posts
Kubernetes Operations

Recover or Rebuild? Four Questions Decide Any Kubernetes Disaster

Only two things in a Kubernetes cluster cannot be regenerated: the CA private keys and the etcd data. Four read-only checks tell you which you lost, and the verdict, within five minutes.

By Sharon Sahadevan··12 min read

It is 2 AM. kubectl times out, every dashboard is red, and the alerting pipeline that runs inside the cluster has stopped alerting. You are looking at fifteen symptoms with no way to rank them, and the one question that decides the next twelve hours is whether you are recovering this cluster or rebuilding it. Four read-only checks answer that in under five minutes, and none of them needs a working API server.

The instinct is to fix the loudest thing. The loudest thing is almost always the API server, and the API server is one of the most disposable components in the system: a stateless process started from a YAML file that kubeadm rewrites in one command.

QUICK ANSWER

Two things in a Kubernetes cluster cannot be regenerated from anything else: the private keys of the certificate authorities, and the data in etcd. Everything else is derivable from those two plus a ClusterConfiguration that fits on one screen.

So every incident lands on one of three verdicts. Recoverable: you end with the cluster you started with, and downtime is the only cost. Recoverable with loss: the cluster comes back, but you permanently give up something specific, usually a window of writes or the cluster's identity. Unrecoverable: no procedure returns it, and the remaining work is salvage and rebuild.

Four questions, asked in order, tell you which: is a control plane peer alive, did the etcd data survive, did the CA private keys survive, and do you have a backup you have actually read.

What is actually irreplaceable#

Sort everything in the cluster into three classes. The class decides the verdict.

Derivable, identity preserving. Regenerating it produces something the rest of the cluster accepts with no other change. Every leaf certificate is here: the API server serving certificate, its client certificates for etcd and the kubelet, the etcd server and peer certificates, the credentials inside admin.conf. So are the static pod manifests, the kubelet configuration, bootstrap tokens and the Node objects themselves.

Reconstructible, identity breaking. You can create a replacement and it works, but everything that trusted the original stops trusting it. The cluster CA, the etcd CA, the front proxy CA and the service account signing keypair live here once the originals are gone.

Irreplaceable. No procedure returns it: the etcd data with no snapshot behind it, the encryption at rest key, and the contents of persistent volumes.

The second class is where intuition fails, and it is the expensive one. Derivable does not mean free. You can always generate a certificate authority. What you cannot generate is the certificate authority, the one whose public certificate is already embedded in the kubeconfig on every node and in every projected token mount. A new one gives you a working control plane that no existing participant trusts. Every node rejoins, and every service account token is invalidated the moment the signing key changes.

The material that decides all of this is small. On a kubeadm control plane:

$ du -sh /etc/kubernetes/pki /var/lib/etcd /etc/kubernetes/manifests
28K     /etc/kubernetes/pki
1.4G    /var/lib/etcd
16K     /etc/kubernetes/manifests

Of that 28 KB of PKI, roughly 6 KB is irreplaceable: ca.key, etcd/ca.key, front-proxy-ca.key and sa.key. The 1.4 GB of etcd is irreplaceable in a different sense, because it is content rather than identity. The 16 KB of manifests is worth nothing, and an hour spent hand-repairing kube-apiserver.yaml instead of running kubeadm init phase control-plane all is an hour spent on the number that does not matter.

sa.key is the one people miss. It is not a CA and signs no certificates. It signs service account tokens, and it is not derived from the cluster CA, so a backup that captures ca.key and nothing else leaves every token in the cluster unrecoverable.

First, measure whether anyone is affected#

Before the four questions, spend a minute outside the cluster. A control plane outage is usually not a traffic outage. Containers belong to the runtime, the kubelet does not treat an unreachable API server as an instruction to delete anything, kube-proxy's rules are already in the kernel, and CoreDNS answers from its informer cache. What stops is change: scheduling, scaling, endpoint updates, token refresh.

Your dashboards cannot tell the difference, because they run in the cluster and read from the API server. A curl from outside can:

$ curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
    https://checkout.example.internal/healthz
200 0.087

A 200 at normal latency means you have hours, and hours is what a careful recovery costs. It is the difference between a responder who restores a four-hour-old snapshot in the first fifteen minutes and one who spends those fifteen minutes copying the etcd data directory somewhere safe.

Q0: lost, or locked out?#

One check reroutes everything. The error text tells you whether something is gone or you simply cannot get in:

$ kubectl get nodes
The connection to the server api.example.internal:6443 was refused - did you
specify the right host or port?

$ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "kubernetes-admin"
cannot list resource "nodes" in API group "" at the cluster scope

$ kubectl get nodes
Unable to connect to the server: tls: failed to verify certificate: x509:
certificate signed by unknown authority

A refused connection means nothing is listening, and the four questions apply. Forbidden means the API server is up, TLS worked and it knows who you are: nothing is lost, you have an RBAC or webhook lockout, and the verdict is always recoverable. An x509 error is a broken trust relationship, usually an expired certificate. Responders who skip this check have restored etcd to fix an RBAC binding they deleted.

Q1: is a control plane peer alive?#

Ask this first, because a healthy peer answers the next two questions at once. Every control plane node in a kubeadm cluster carries a full copy of the PKI, including all three CA private keys and sa.key, and a full etcd member holding the entire keyspace.

$ ssh cp-node-02 'sudo crictl ps --name kube-apiserver'
CONTAINER      IMAGE          CREATED       STATE     NAME
a3f81c7b29de   4a2f7b18cc90   6 days ago    Running   kube-apiserver

$ ssh cp-node-02 'sudo ls /etc/kubernetes/pki/ca.key'
/etc/kubernetes/pki/ca.key

Then check quorum, which is separate from whether nodes are up. Do not use member list for this. It reports configured membership, and a member that has been dead for a week still shows started. Use a command that actually contacts every endpoint:

$ ETCDCTL_API=3 etcdctl \
    --endpoints=https://10.0.0.11:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    endpoint health --cluster
https://10.0.0.11:2379 is healthy: successfully committed proposal: took = 9.312ms
https://10.0.0.12:2379 is healthy: successfully committed proposal: took = 11.04ms
https://10.0.0.10:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

Two of three answering means quorum is intact. Three members tolerate one failure; five tolerate two. Peer alive with quorum is recoverable: remove the dead member, join a replacement, lose nothing.

Q2: did the etcd data survive?#

The useful form of the question is not whether the directory exists but whether etcd will load it.

$ ls /var/lib/etcd/member/
snap  wal

$ journalctl -u kubelet --since -30m | grep -i etcd | tail -1
Jun 12 02:07:44 cp-node-01 kubelet[1184]: E0612 02:07:44.913 pod etcd-cp-node-01
  container etcd exited with 1: walpb: crc mismatch

Both snap and wal at a plausible size is a good sign. A crc mismatch, panic: freepages or database file is corrupted in the container logs means treat the answer as no.

Q3: did the CA private keys survive?#

Four files decide it:

$ sudo ls -l /etc/kubernetes/pki/{ca,sa,front-proxy-ca}.key /etc/kubernetes/pki/etcd/ca.key
-rw------- 1 root root 1679 Jun  6 09:12 /etc/kubernetes/pki/ca.key
-rw------- 1 root root 1675 Jun  6 09:12 /etc/kubernetes/pki/etcd/ca.key
-rw------- 1 root root 1675 Jun  6 09:12 /etc/kubernetes/pki/front-proxy-ca.key
-rw------- 1 root root 1679 Jun  6 09:12 /etc/kubernetes/pki/sa.key

If you are reading them from a backup instead, check that each key actually belongs to its certificate. A backup holding ca.crt and no ca.key can verify signatures and sign nothing, and a key and certificate from different generations of the cluster fail only at the moment the API server refuses to start. Compare the public keys:

$ openssl x509 -in ca.crt -noout -pubkey | sha256sum
4f1c8a2e93b7d605e1a4c7f28d3b9016a5c2e8f47b91d306c5a8e2f74b1d9c30  -

$ openssl pkey -in ca.key -pubout | sha256sum
4f1c8a2e93b7d605e1a4c7f28d3b9016a5c2e8f47b91d306c5a8e2f74b1d9c30  -

Matching digests mean the pair belongs together. Data plus keys is recoverable. Data without keys is recoverable with loss: your objects all come back and the cluster's identity does not.

WARNING

Never answer Q3 by running kubeadm init phase certs ca. If a CA is missing it generates a new one. During an incident where a disk is half mounted, "missing" can mean the real CA is intact under a mount you did not notice, and you have just created a new trust root on top of it. Answer the question with ls and openssl. Reading never generates.

Q4: is there a backup you have actually read?#

The weak form of this question is almost always yes and almost never useful. A backup job that reports success is not a validated backup. A snapshot whose header you have read is:

$ etcdutl snapshot status /mnt/backups/etcd-2026-06-12-0100.db --write-out=table
+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 3f1ac207 |  8471192 |      14038 |     1.3 GB |   3.6.0 |
+----------+----------+------------+------------+---------+

TOTAL KEYS should be in a plausible range for your object count, and TOTAL SIZE near your live database size. A few hundred keys for a cluster running thousands of pods is a snapshot of something other than what you think. Use etcdutl here: etcdctl snapshot status was removed in etcd v3.6, and kubeadm ships etcd 3.7.3 for Kubernetes 1.35 to 1.37.

One more check belongs with this question. If encryption at rest is enabled, the etcd data is only as recoverable as the key that decrypts it, and that key lives in a file on the control plane nodes, not in etcd:

$ grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml

Restore a perfect snapshot without that file and every object comes back except the Secrets, which stay ciphertext permanently. That part of the verdict is unrecoverable regardless of everything else.

The verdict, and the fix in order of cost#

Each combination of answers has exactly one verdict. Cheapest first:

Peer aliveetcd dataCA keysBackupVerdict and fix
Yes, quorum intactn/an/an/aRecoverable. Remove the dead member, join a replacement
NoSurvivedSurvivedAnyRecoverable. Rebuild the control plane around the data
Yes, quorum lostOn a survivorYesAnyRecoverable with loss. Force a new cluster from the survivor, or restore
NoLostSurvivedSnapshot existsRecoverable with loss. Restore, lose every write since the snapshot
NoSurvivedLostNo PKI backupRecoverable with loss. New CAs, every node rejoins, every token invalidated
NoLostLostNothing validatedRebuild. Salvage while you still have access

When two paths reach the same verdict, take the one that preserves more evidence. Forcing a new etcd cluster from a survivor and restoring a snapshot both land on recoverable with loss, but the survivor holds every write up to the failure, and its data directory still exists if the attempt goes wrong. A restore performed in place has no second attempt.

And keep the triage read-only until you have all four answers. Starting etcd "to see whether it works" is a write, because it replays its write-ahead log on startup. Copy the data directory first, with the process stopped, and experiment on the copy:

$ sudo systemctl stop kubelet
$ sudo cp -a /var/lib/etcd /mnt/backups/etcd-datadir-cp-node-01-preinspect
WAR STORY

A team lost one control plane node in a three-node cluster to a failed disk controller. Q1 was yes twice over, and the right procedure was to remove the dead etcd member and join a replacement: about forty minutes, zero loss. The responder went straight to the runbook section titled Restore From Snapshot, because the phrase matched the panic, and restored a four-hour-old snapshot onto the two survivors. That rewound the cluster past a Deployment rollout and past the creation of several PVCs, whose volumes were now orphaned in the storage backend. The failed disk cost nothing. The restore cost four hours of cluster state and a day of storage reconciliation, and Q1 would have prevented it in ninety seconds.

In production#

Write the four answers into the incident channel as one message before anyone takes a write action: peer, data, keys, backup, each yes or no with its evidence. It stops a second responder from re-running checks destructively, it makes the verdict reviewable by someone who is not typing, and when the answer is rebuild, saying so at minute twenty leaves you hours of API access to salvage from. Then make sure next time's Q3 is a yes: back up /etc/kubernetes/pki outside the cluster, encrypted, and count the private keys in the archive. A standard kubeadm control plane with stacked etcd has eleven.

The four questions, and every control plane, etcd, lockout and destructive action scenario they route to, are in Kubernetes Failure Recovery and Cluster Repair, whose first module is free.

More in Kubernetes Operations