The Rules of Not Making It Worse

The cluster was recoverable when the incident started. It stopped being recoverable forty minutes later, because someone ran a command that overwrote the only copy of something.

---

The Problem

Almost nothing in Kubernetes is permanently destroyed by the original failure. Disks fail, nodes die, processes crash, and in nearly every case the irreplaceable material survives somewhere: on a peer, in a snapshot, in a data directory that is still sitting on the filesystem. What destroys it is a command typed afterward, by someone trying to help, under pressure, with incomplete information.

That is not a claim about carelessness. The commands that cause it are the ones a competent engineer reaches for first. kubeadm reset on a node that will not come up is a reasonable instinct and it deletes the PKI directory and the etcd data directory. Restoring a snapshot when writes are failing is a reasonable instinct and it discards every write since that snapshot. Removing a finalizer that is blocking a delete is a reasonable instinct and it orphans whatever the finalizer existed to clean up. Each of these is the documented remedy for some situation. The problem is that at 3 AM, the situation you are in and the situation the remedy was written for are not the same, and the command does not check.

The structural reason this keeps happening is that Kubernetes recovery has almost no undo. A cluster's destructive operations are overwhelmingly one-way doors: there is no revision history on a private key, no trash can for a deleted CRD, no rollback on an etcd restore. Meanwhile the pressure gradient in an incident points entirely toward action, because doing something feels like progress and reading feels like paralysis, and the person who is typing is the one who appears to be helping.

The rules below are not a code of conduct. They are a small set of mechanical habits that make the one-way doors visibly one-way, and every one of them costs less than a minute. They are also the reason the previous two lessons insisted on being read-only: triage that cannot destroy anything is what buys you the time to apply these.

---

How It Works

An incident that ends badly follows a shape consistent enough to be worth naming, because recognizing which stage you are in is most of the defense.

The fifth stage is the one worth staring at. A destructive mistake during an incident produces no new symptom, because the cluster was already broken and remains broken in the same visible way. There is no moment of feedback where the responder learns that the verdict just changed. That is entirely unlike normal operations, where a bad command breaks something that was working and you find out immediately. During an incident the feedback loop is severed, which is why the discipline has to be preventive rather than reactive.

Rule 1: back up the broken state, including the broken parts

The instinct is to back up things that work. During an incident the valuable artifact is the broken thing, because it is simultaneously the evidence and, frequently, the only remaining copy. A corrupted etcd data directory can often be partially read. An overwritten certificate still records what the cluster used to answer to. A half-mounted filesystem still holds keys.

Rule 2: read the flags, and read what the command deletes

Recovery tooling is written for clean-slate operations, not for surgery, and its defaults reflect that. kubeadm reset reverts a node to its pre-join state, which means removing the local etcd member from the cluster and deleting the Kubernetes configuration directory, PKI included. That is correct behavior for decommissioning and catastrophic for triage.

Rule 3: verify before you restart, not after

Restarting is what makes a change take effect, and on a control plane it is usually also what removes your ability to inspect what you just did. The specific trap is that the kubelet watches its manifest directory continuously, so editing a static pod manifest in place is not staging a change, it is deploying it, within a couple of seconds and with no confirmation step. Move the file out of the watched directory, edit it there, diff it against a known-good copy, and move it back when you are sure.

Rule 4: change one thing at a time, and never on every copy at once

The parallel version of a repair is the version that removes your fallback. On a three-node control plane, the two nodes you have not touched are your only remaining record of what a correct configuration looks like. Repair one, verify it fully, and only then move on. The same logic applies to a single node: two simultaneous changes give you no way to attribute the result to either.

Rule 5: keep a written record while you work

Not for the postmortem. For the next thirty minutes, and for the second responder. Human memory under adrenaline reliably compresses and reorders events, and the question "did anyone already run that?" comes up in every incident that runs longer than an hour.

Rule 6: prefer additive to in-place

Copy the kubeconfig and edit the copy. Restore the snapshot into a new data directory and point the manifest at it. Write the corrected manifest beside the original rather than over it. In-place edits are faster and cost you the ability to compare, which is the thing you will want most when the repair does not work the first time.

---

Doing It Safely

The mechanics take about two minutes at the start of an incident and one command before each destructive step.

Open a transcript first. Every command and every piece of output ends up in a file, which turns Rule 5 from a discipline into a side effect:

$ script -a /mnt/backups/incident-cp-node-01.log
Script started, output log file is '/mnt/backups/incident-cp-node-01.log'

$ export PS1='[\D{%H:%M:%S} cp-node-01] \$ '
[02:14:31 cp-node-01] $

The timestamped prompt matters more than it looks. A transcript with timestamps reconstructs the incident timeline exactly, including the gaps where everyone was thinking, and it settles the "when did we run that" question that otherwise consumes twenty minutes of the postmortem.

Then take the copy. This is destructive in one respect only, that it stops the kubelet and therefore the static pods on this node, so do it on a node you have already decided to work on and confirm that etcd quorum does not depend on it:

$ sudo systemctl stop kubelet
$ sudo tar czf /mnt/backups/broken-etc-kubernetes-cp-node-01-$(date +%FT%H%M).tgz \
    -C / etc/kubernetes
$ sudo cp -a /var/lib/etcd /mnt/backups/etcd-datadir-broken-cp-node-01
$ ls -la /mnt/backups/
-rw-r--r-- 1 root root  38912 Jun 12 02:16 broken-etc-kubernetes-cp-node-01-2026-06-12T0216.tgz
drwx------ 3 root root   4096 Jun 12 02:17 etcd-datadir-broken-cp-node-01

Back up to a path that is not on the failing disk and not inside the cluster. /mnt/backups/ here stands for whatever that is in your environment; /tmp on the node you are about to reset is not it.

Now the flag discipline. The next block is destructive and irreversible: it removes this node's etcd member from the cluster and deletes /etc/kubernetes including the PKI directory. Do not run it until the archive above exists and you have verified it contains private keys. It is shown here so you recognize the prompt, because the prompt is the last thing standing between you and the CA:

$ sudo kubeadm reset
[reset] Reading configuration from the cluster...
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join'
        will be reverted.
[reset] Are you sure you want to proceed? [y/N]:

Answer N unless you have consciously decided this node is being rebuilt from scratch. The -f flag suppresses that prompt entirely, which is why -f belongs in automation and never in an incident shell.

For anything that changes cluster objects, make the server show you the change before you commit to it. Server-side dry run runs the request through admission and returns the result without persisting it, and kubectl diff shows exactly what would change:

$ kubectl diff -f /tmp/repair-coredns.yaml
--- /tmp/LIVE-2649013894/apps.v1.Deployment.kube-system.coredns
+++ /tmp/MERGED-1892034481/apps.v1.Deployment.kube-system.coredns
@@ -18,7 +18,7 @@
   replicas: 2
-  minReadySeconds: 0
+  minReadySeconds: 10

$ kubectl apply -f /tmp/repair-coredns.yaml --dry-run=server
deployment.apps/coredns configured (server dry run)

And for the static pod path, respect the directory watch. Stage outside it, compare, then move the file in as a single atomic action:

$ sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/staging/
$ sudo vi /tmp/staging/kube-apiserver.yaml
$ diff -u /mnt/backups/manifests-known-good/kube-apiserver.yaml \
          /tmp/staging/kube-apiserver.yaml
$ sudo mv /tmp/staging/kube-apiserver.yaml /etc/kubernetes/manifests/

Moving the manifest out stops the pod, which on a single control plane node means the API server is down for the duration of the edit. That is a real cost and it is the correct trade, because the alternative is an in-place edit that the kubelet picks up mid-save.

---

Tradeoffs and Decision Framework

Command	What it reads as	What it actually destroys	The safer sequence
kubeadm reset	Clean up a node that will not start	The PKI directory, the etcd data directory, and this node's etcd membership	Archive /etc/kubernetes and /var/lib/etcd first, then reset only if rebuilding
etcdctl snapshot restore after rm -rf	Get etcd back to a known good state	Every write between the snapshot and now	Move the data directory aside, restore into a new path, keep the original
etcd --force-new-cluster	Recover a cluster that lost quorum	The other members permanently, and any entries not on this member	Copy the data directory first; the operation cannot be undone against it
kubectl delete pod --force --grace-period=0	Unstick a pod in Terminating	The guarantee that the container has stopped, which for a StatefulSet means two writers on one volume	Confirm the container is gone with crictl on the node, then force
kubectl patch removing a finalizer	Unstick an object that will not delete	The cleanup the owning controller was going to perform, orphaning external resources	Identify the controller and why it is blocked; remove the finalizer last
helm uninstall on an operator	Remove a broken component	Its CRDs, and by cascade every custom resource of those types	Scale the operator to zero, or delete only the specific object that is misbehaving

Four questions settle whether to run a destructive command now. Is there a copy of what this overwrites, in a location that does not share a failure domain with it? Is this the least destructive action that could produce the outcome I want, or merely the fastest one I know? If I am wrong about the diagnosis, what does this command do to the state I would have needed for the correct repair? And has anyone else touched this cluster in the last ten minutes, because two responders each making one safe change have jointly made an unsafe one.

Default: when in doubt, copy and wait. A recoverable cluster stays recoverable while you think about it. The only failure modes that genuinely worsen with time are the credential clocks from the previous lesson, and those run in hours, not minutes. Nothing else in a broken cluster is decaying fast enough to justify skipping a cp -a.

---

Common Mistakes

Backing up only the things that still work. The broken artifact is the evidence and often the last copy. A corrupted data directory and an overwritten certificate are both worth archiving before you touch them.

Treating a confirmation prompt as friction. The prompt on kubeadm reset exists because that command is a one-way door. Reaching for -f to skip it during an incident removes the only safeguard in the entire flow.

Editing static pod manifests in place. The kubelet watches that directory and applies whatever it finds, including a partially written file. Stage the edit outside the directory and move it in atomically.

Running the same repair on every control plane node in parallel. The untouched nodes are your only copy of a correct configuration and a working PKI. Serialize the repair and verify between steps.

Letting two people type at once. Every incident needs one pair of hands and as many reviewers as you like. Concurrent changes make attribution impossible and make each person's mental model quietly wrong.

Removing a finalizer to make a deletion complete. The finalizer is a controller saying it has cleanup to do. Forcing past it orphans cloud load balancers, volumes, and DNS records that nothing will ever come back for, which Module 8 covers in detail.

Rebuilding from an assumption instead of a verification. "The certs must be expired" and "the disk must be full" are hypotheses. Confirming one costs thirty seconds; acting on the wrong one costs the incident.

Closing the incident without recording the destructive actions. If nobody writes down that a snapshot was restored or a finalizer was removed, the resulting orphaned resources surface weeks later as an unrelated mystery, and the postmortem records the original disk failure as the whole story.

---