What State Is Actually Irreplaceable

Your cluster is broken and you do not yet know how badly. Before you touch anything, there is one question that determines everything that follows: what in this cluster cannot be regenerated?

---

The Problem

Two things in a Kubernetes cluster cannot be regenerated from anything else: the private keys of the certificate authorities, and the data in etcd. Every other file under /etc/kubernetes is a function of those keys plus a configuration document you could retype by hand in ten minutes. Every server certificate, every client certificate, every kubeconfig, every static pod manifest, every bootstrap token, every node registration: derivable.

That sentence is the whole course. Every verdict in every later module is an application of it.

The reason you need it at 3 AM is that a broken cluster does not present its damage in order of severity. It presents everything at once. kubectl times out, the dashboards go red, the alerting pipeline that runs in the cluster stops alerting, and the on-call engineer is looking at fifteen symptoms with no way to rank them. The instinct in that moment is to fix the loudest thing. The loudest thing is almost always the API server, and the API server is one of the most disposable components in the system: a stateless process started from a YAML file that kubeadm will rewrite for you in one command.

The asymmetry is genuinely strange when you first see it in numbers. The material that decides whether your cluster survives is measured in kilobytes:

$ du -sh /etc/kubernetes/pki /var/lib/etcd /etc/kubernetes/manifests
28K     /etc/kubernetes/pki
1.4G    /var/lib/etcd
16K     /etc/kubernetes/manifests

Of that 28 KB of PKI, roughly 6 KB is irreplaceable: three CA private keys and a service account signing key. The 1.4 GB of etcd is irreplaceable in a different sense, because it is not identity, it is content. The 16 KB of manifests is worth nothing at all, and an hour spent hand-repairing a mangled kube-apiserver.yaml instead of running kubeadm init phase control-plane all is an hour spent on the number that does not matter.

There is a second failure of intuition, and it is more expensive. Derivable does not mean free. You can always generate a certificate authority. What you cannot generate is the certificate authority, the one whose public certificate is already embedded in the kubeconfig on every node and in every projected token mount. Generating a new one gives you a working cluster that no existing participant trusts. That distinction, between recreating a thing and recreating the same thing, is what separates the green verdicts in this course from the orange ones.

---

How It Works

Sort every artifact in the cluster into one of three classes. The class determines the verdict, and the verdict determines whether you are running a procedure tonight or writing a rebuild plan.

Class 1, derivable and identity-preserving. Regenerating it produces something the rest of the cluster accepts without any other change. Every leaf certificate falls here: the API server serving certificate, its client certificates for etcd and the kubelet, the etcd server and peer certificates, the front proxy client certificate, and the credentials embedded in admin.conf, controller-manager.conf and scheduler.conf. So do the static pod manifests, the kubelet configuration, bootstrap tokens, and the node objects themselves.

Class 2, reconstructible but identity-breaking. You can create a replacement, and the replacement works, but everything that trusted the original stops trusting it. The cluster CA, the etcd CA, the front proxy CA, and the service account signing keypair live here when the originals are gone. This is the entire population of the RECOVERABLE WITH LOSS verdict: the cluster comes back, and you pay in rejoins, reissued credentials, and restarted workloads.

Class 3, irreplaceable. No procedure returns it. The CA private keys as the keys rather than as new ones, the etcd data with no snapshot behind it, the encryption-at-rest key, and the contents of persistent volumes. These produce the red verdicts, and the honest work in those lessons is salvage, not repair.

The three CAs and what each one gates

A kubeadm cluster runs three independent certificate authorities, and confusing them wastes real time during an incident. Look at what is actually on disk:

$ ls /etc/kubernetes/pki /etc/kubernetes/pki/etcd
/etc/kubernetes/pki:
apiserver.crt                 apiserver-kubelet-client.crt  front-proxy-ca.crt
apiserver.key                 apiserver-kubelet-client.key  front-proxy-ca.key
apiserver-etcd-client.crt     ca.crt                        front-proxy-client.crt
apiserver-etcd-client.key     ca.key                        front-proxy-client.key
                              sa.key                        sa.pub

/etc/kubernetes/pki/etcd:
ca.crt   ca.key   healthcheck-client.crt   peer.crt   server.crt

ca.key is the cluster CA. It signs the API server serving certificate, the API server's client certificate to the kubelet, the component kubeconfig credentials, and every kubelet client certificate issued through the CSR API. Its public half, ca.crt, is the trust anchor carried by every kubeconfig and every projected token mount. etcd/ca.key is a separate authority signing the etcd server, peer, and health check certificates plus the API server's etcd client certificate. front-proxy-ca.key signs the aggregation layer client certificate, which matters if you run metrics-server or any extension API server.

sa.key is the odd one out and the one people miss. It is not a CA and it signs no certificates. It is the RSA private key that signs service account tokens, and sa.pub is what the API server uses to validate them. It is not derived from the cluster CA, so a PKI backup that captures only ca.key leaves every service account token in the cluster unrecoverable. Losing it invalidates every projected token at once, which is Module 4's RECOVERABLE WITH LOSS scenario and presents in application logs as a wave of 401s that looks nothing like a control plane problem.

Why the public certificate is not a backup

This is the trap that turns a green verdict orange in a backup review, and it survives most audits because a directory listing looks complete.

$ ls /mnt/backups/pki-2026-06-01/
apiserver.crt  ca.crt  front-proxy-ca.crt  sa.pub

That backup contains no private keys. It lets you verify signatures and build a trust store. It cannot sign anything, and signing is the entire point of holding a CA during a recovery. A backup with ca.crt and no ca.key is a Class 3 loss wearing a Class 1 costume.

The check that catches it is a modulus comparison, and it also catches the subtler version of the same failure, where the key and the certificate are both present but came from different generations of the cluster:

$ openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -modulus | openssl sha256
SHA2-256(stdin)= 4f1c8a2e93b7d605e1a4c7f28d3b9016a5c2e8f47b91d306c5a8e2f74b1d9c30

$ openssl rsa -in /etc/kubernetes/pki/ca.key -noout -modulus | openssl sha256
SHA2-256(stdin)= 4f1c8a2e93b7d605e1a4c7f28d3b9016a5c2e8f47b91d306c5a8e2f74b1d9c30

Matching digests mean the key signs certificates that chain to that certificate. Differing digests mean you are holding two unrelated halves and you will find out during the recovery, at the moment the API server refuses to start.

etcd, and the key that decides whether the data is readable

The CAs are irreplaceable because they are identity; etcd is irreplaceable because it is content. Every object in the cluster lives there and nowhere else. The API server is a stateless front end over it, controllers hold caches that expire, and each kubelet knows about its own pods and nothing more. There is no second copy anywhere unless you made one. Confirm what you have before you reason about it:

$ ETCDCTL_API=3 etcdctl \
    --endpoints=https://10.0.0.10:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    endpoint status --write-out=table
+------------------------+------------------+---------+---------+-----------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER |
+------------------------+------------------+---------+---------+-----------+
| https://10.0.0.10:2379 | 8e9f2a1c4d6b3057 |  3.5.12 |  1.4 GB |      true |
+------------------------+------------------+---------+---------+-----------+

There is a qualifier on all of this that the documentation does not connect for you. If encryption at rest is enabled, the etcd data is only as recoverable as the key that decrypts it. Secrets are stored as ciphertext, and the key lives in a provider configuration file on the control plane nodes, referenced from the API server manifest and pointedly not stored in etcd:

$ grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml

Back up a perfect etcd snapshot every hour, lose that file, and every Secret in the restored cluster is permanently unreadable while every other object comes back intact. That is the quietest catastrophe in this course and Module 3 covers it in full. The etcd Operations Masterclass owns the data model, sizing, and backup mechanics; here, etcd matters only as the thing that either survived or did not.

---

Doing It Safely

The audit below takes about ten minutes on a healthy cluster and is worth doing before you need it, because during an incident you will be answering these questions with a broken API server and a flashlight.

Start by proving you can actually derive what you claim is derivable. The strongest evidence is that kubeadm can rebuild the leaf certificates from the CAs on disk. Run it on a lab cluster, not production, and understand that it overwrites the leaf certificates in place while leaving the CAs untouched:

$ kubeadm init phase certs apiserver --config /tmp/kubeadm-config.yaml
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api.example.internal
cp-node-01 kubernetes kubernetes.default kubernetes.default.svc
kubernetes.default.svc.cluster.example.internal] and IPs [10.96.0.1 10.0.0.10]

That output is the derivability model working. The CA signed a fresh leaf, the leaf carries the names from your configuration, and no client anywhere needed to change. Note what the command needed: a ClusterConfiguration. Without it, kubeadm falls back to defaults and silently produces certificates that omit your controlPlaneEndpoint and any custom certSANs. The Production Kubernetes Operations course covers that failure in detail; the point here is that the configuration is the second input to the derivability model, and it is as worth backing up as the keys.

Then take the backup that makes all of this moot. It is 28 KB and it is the highest-value artifact in your infrastructure:

$ tar czf /mnt/backups/pki-cp-node-01-$(date +%F).tar.gz \
    -C /etc/kubernetes pki

$ kubectl -n kube-system get cm kubeadm-config \
    -o jsonpath='{.data.ClusterConfiguration}' \
    > /mnt/backups/cluster-config-$(date +%F).yaml

$ tar tzf /mnt/backups/pki-cp-node-01-$(date +%F).tar.gz | grep -c '\.key$'
8

That last command is the part people skip. Count the private keys in the archive before you trust it. Eight is what a standard kubeadm control plane produces, and a count that comes back lower than you expect means something excluded them, usually a backup tool with a default filter on files named *.key.

---

Tradeoffs and Decision Framework

The three verdicts used throughout this course are definitions, not moods. RECOVERABLE means you end with the cluster you started with: every object, every credential, every trust relationship intact, and downtime as the only cost. RECOVERABLE WITH LOSS means the cluster comes back and keeps serving, but you permanently surrender something specific and enumerable, usually a window of writes or the cluster's identity. UNRECOVERABLE means no procedure returns the thing; the remaining work is salvage and rebuild.

What you lost	Class	Regenerated from	What it costs	Verdict if that is all you lost
Static pod manifests	Derivable	kubeadm init phase plus ClusterConfiguration	Minutes of control plane downtime	RECOVERABLE
Leaf certificates and kubeconfigs	Derivable	The three CAs plus ClusterConfiguration	Minutes, no client changes	RECOVERABLE
etcd data, snapshot exists	Class 3 with a backup	The snapshot	Every write since the snapshot	RECOVERABLE WITH LOSS
CA private keys	Class 2 replacement only	Nothing; you generate new ones	Every node rejoins, every token invalid	RECOVERABLE WITH LOSS
Service account signing key	Class 2 replacement only	Nothing; you generate a new one	Every projected token invalid at once	RECOVERABLE WITH LOSS
etcd data, no snapshot	Irreplaceable	Nothing	Every object in the cluster	UNRECOVERABLE
Encryption at rest key	Irreplaceable	Nothing	Every Secret, permanently	UNRECOVERABLE

Four questions settle which row you are in. Do you still hold a private key that signs, rather than a certificate that only verifies? Is there a full etcd data directory or a snapshot you have actually validated, as opposed to a backup job that reports success? If encryption at rest is enabled, do you have the provider configuration file, and is it somewhere other than the node you just lost? And is there a surviving control plane peer, which answers the first three questions at once because a healthy peer holds a complete copy of both the PKI and the data?

Default: assume the identity material is the constraint, and check it first. Data loss is loud, well understood, and usually has a backup behind it. Identity loss is quiet, has no backup at most organizations, and is the difference between an afternoon of work and rejoining every node in the fleet. When you have thirty seconds and one question, ask whether ca.key and sa.key still exist somewhere you can reach.

---

Common Mistakes

Treating the API server as the valuable component. It is a stateless process defined by a YAML file that kubeadm regenerates in one command. Time spent hand-repairing its manifest before you know whether the CAs and etcd survived is time spent on the wrong question.

Backing up certificates instead of keys. A directory of .crt files looks like a complete PKI backup and can sign nothing. Verify by counting .key files in the archive, then verify the key and certificate match with a modulus digest comparison.

Forgetting sa.key because it is not a certificate. It sits in the same directory as the CAs, is not signed by any of them, and its loss invalidates every service account token in the cluster. Backup scripts that iterate over CA files specifically miss it.

Believing a GitOps repository covers you. Declarative manifests reconstruct your workloads, but they hold no CA keys, no signing key, no dynamically issued credentials, and none of the state controllers wrote back to the API. Git plus a fresh cluster is a rebuild, not a recovery.

Assuming the etcd snapshot is enough on an encrypted cluster. The snapshot preserves ciphertext. Without the encryption provider configuration file, every Secret restores as an unreadable blob while everything around it comes back perfectly, and the failure only surfaces when a pod tries to mount one.

Confusing the etcd CA with the cluster CA. They are separate trust domains. A recovery that regenerates one and reuses the other produces an API server that starts cleanly and cannot authenticate to etcd, which reads as a storage failure and is a PKI failure.

Keeping the only PKI backup on the control plane node. A backup that shares a failure domain with the thing it protects is a copy, not a backup. The 28 KB belongs in object storage or a secrets manager, encrypted, outside the cluster.

Deciding a certificate authority is expiring soon and therefore disposable. kubeadm CAs default to ten years while leaf certificates default to one, so the expiry you are chasing is almost never the CA. Confirm which certificate actually expired before touching anything that regenerates a trust root.

---