Assessing Blast Radius Before Acting

The API server is down and every dashboard is red. Before you declare a major incident, check whether your users can still reach the application, because very often they can.

---

The Problem

A control plane outage is not usually a traffic outage. Pods keep running, services keep routing, and DNS keeps resolving, because none of those depend on the API server being reachable once they are established. What stops is change: nothing schedules, nothing scales, nothing self-heals, and no new endpoint reaches the dataplane. Your cluster is frozen, not dead, and frozen is a fundamentally different incident.

This matters because your dashboards will not tell you. Almost every Kubernetes monitoring stack runs inside the cluster and reads from the API server, so a control plane failure blinds the observability layer at the same instant, and every panel goes red or blank simultaneously. The picture on the screen is indistinguishable between "the API server is down" and "every workload in the cluster is dead," and the first is a Tuesday while the second is a company-wide event. Responders act on the picture.

The consequence is that people take enormous risks to fix something that is not on fire. The risk you are entitled to take in a repair is a function of what is actually broken, and almost nobody measures that before deciding. A responder who believes checkout is down will restore a four-hour-old etcd snapshot in the first fifteen minutes. The same responder, knowing checkout is serving 200s at normal latency, will spend those fifteen minutes copying the etcd data directory to a backup path and reading the kubeadm-config ConfigMap out of the API before it goes away. Same incident, same information available, completely different outcome, and the only difference is whether anyone ran a curl.

There is a clock on the reprieve, and knowing its shape is what keeps this from becoming complacency. A frozen cluster degrades: pods that exit are not replaced, endpoints go stale, mounted ConfigMaps and Secrets stop refreshing, and eventually short-lived credentials expire. The degradation is gradual and mostly measured in hours rather than minutes, which is exactly the budget you need to do the recovery carefully.

---

How It Works

The reason workloads survive is architectural rather than lucky. The kubelet is a level-triggered reconciler with a local cache, and the container runtime is a separate process that outlives it. When the API server disappears, the kubelet keeps its last known pod set and keeps driving toward it. It does not interpret an unreachable API as an instruction to delete anything.

That last part is deliberate and worth knowing precisely, because it is the single design decision that stands between you and a cluster-wide outage every time the API server hiccups. The kubelet will not garbage collect a pod until every one of its configuration sources has reported at least once, so an API server it cannot reach never counts as a source saying "this pod is gone." A kubelet restarted mid-outage will log a stream of list and watch failures and still leave every running container alone.

$ sudo journalctl -u kubelet --since -5m | tail -3
E0612 02:14:07.441 reflector.go:150] Failed to watch *v1.Pod: failed to list
  *v1.Pod: Get "https://api.example.internal:6443/api/v1/pods?limit=500":
  dial tcp 10.0.0.10:6443: connect: connection refused

$ sudo crictl ps --state Running | wc -l
42

Forty-two containers running on a node whose kubelet cannot reach anything. That is the system working as designed.

What is still working, precisely

Existing pods keep running, and their containers keep restarting locally on liveness probe failures or non-zero exits, because restartPolicy is enforced by the kubelet against the runtime with no API involvement. Service routing keeps working because kube-proxy already wrote the rules into the kernel and nothing removes them. In-cluster DNS keeps answering because CoreDNS holds the service and endpoint state in an informer cache and serves from memory.

You can verify all three from a node without touching the API, and the DNS check in particular is worth knowing because kubectl exec is unavailable in exactly this situation. kube-proxy's rules are node-local, so the node itself can reach any ClusterIP:

$ dig +short @10.96.0.10 checkout.prod-checkout.svc.cluster.example.internal
10.96.14.207

$ sudo iptables -t nat -L KUBE-SERVICES -n | grep -c '^KUBE-SVC'
118

An answer from the DNS service IP and a populated KUBE-SERVICES chain together tell you the dataplane is intact. Then confirm the only thing that actually matters, from outside:

$ curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
    https://checkout.example.internal/healthz
200 0.087

What has stopped, and when each becomes visible

Scheduling stops, so any pod that exits and needs replacement stays gone. Scaling stops, including the HPA, so a traffic spike during the outage cannot be absorbed. Self-healing stops in the sense that matters: the kubelet restarts a crashed container, but replacing a pod requires a controller, and every controller is down.

Endpoint updates stop, and this is the subtle one. Readiness probes still run and their results are still computed locally, but the status update cannot reach the API, so no EndpointSlice changes. Traffic keeps flowing to pods that have failed their readiness probe, because the dataplane's view of readiness froze at the moment the control plane did. A rolling deployment that was mid-flight when the API server died leaves you with a mix of old and new endpoints permanently until the control plane returns.

Configuration updates stop. The kubelet refreshes mounted ConfigMap and Secret volumes by polling the API, so those mounts hold whatever they held at failure time. And credentials stop being reissued, which is the item with the hard deadline:

$ sudo find /var/lib/kubelet/pods -path '*kube-api-access*/token' | head -1
/var/lib/kubelet/pods/6b1f.../volumes/kubernetes.io~projected/kube-api-access-x9k2t/token

$ sudo cat /var/lib/kubelet/pods/6b1f.../volumes/kubernetes.io~projected/kube-api-access-x9k2t/token \
    | cut -d. -f2 | base64 -d 2>/dev/null | head -c 200
{"aud":["https://kubernetes.default.svc.cluster.example.internal"],
 "exp":1749695412,"iat":1749691812,...

Projected service account tokens are short lived by default, on the order of an hour, and the kubelet refreshes them against the TokenRequest API well before expiry. With the API server down, that refresh cannot happen. Any in-cluster client that authenticates with such a token starts failing once its current token expires, and workloads that validate each other's tokens fail with each other rather than with the control plane, which reads in application logs as a service-to-service auth bug.

The longer clock is certificate rotation. Kubelets rotate their client certificates through the CSR API well before expiry, and an outage spanning that window leaves nodes holding expired credentials that cannot reconnect even after the control plane returns. That is a multi-day concern rather than a tonight concern, but it is the reason a "frozen but serving" cluster is not a stable state you can leave over a weekend.

---

Doing It Safely

Assessment is a five-minute activity with a fixed order, and the order is chosen so the most decision-changing number arrives first.

Start outside the cluster. Hit the real user-facing endpoint, not a Kubernetes health check, and get a status code and a latency. If you have an external synthetic monitor or a CDN-level metric, read it, because it is the only observability you own that did not just go blind. Then check your error budget: a green SLI means you have hours, and hours is what a careful recovery costs.

Second, establish which nodes still have running workloads, which you do over SSH because the API is gone. A short loop across the fleet gives you a shape:

$ for n in worker-node-01 worker-node-02 worker-node-03; do
    printf '%s ' "$n"
    ssh "$n" 'sudo crictl ps --state Running -q | wc -l'
  done
worker-node-01 38
worker-node-02 41
worker-node-03 0

Third, identify what is degrading rather than what is broken. Anything mid-rollout is stuck mid-rollout. Anything that crash-loops is now permanently gone rather than rescheduled. Anything with a projected token has a deadline. Anything that was scaling is not.

Fourth, and only now, decide the pace. A green SLI buys you the read-only decision tree from the previous lesson, a copy of the etcd data directory, a copy of the PKI, and a careful single-change-at-a-time repair. A red SLI buys you none of that, and the correct response to a genuinely dark dataplane is different work entirely: get traffic to a healthy cluster or a healthy region first, then repair without a clock on you.

Fifth, plan to watch the recovery rather than celebrate it. The first successful kubectl get nodes is the start of the riskiest few minutes of the incident, because every node lease is stale and every controller queue replays at once:

$ kubectl get nodes
NAME             STATUS     ROLES           AGE    VERSION
cp-node-01       NotReady   control-plane   287d   v1.29.4
worker-node-01   NotReady   <none>          287d   v1.29.4
worker-node-02   NotReady   <none>          287d   v1.29.4

Every one of those nodes is running workloads normally. They report NotReady because their leases went unrenewed while the API was gone, and they clear within a lease period or two as kubelets reconnect. Watch them clear before you touch anything, because a responder who starts draining or deleting nodes in that window is acting on a status that was about to fix itself.

---

Tradeoffs and Decision Framework

Capability	During a control plane outage	Becomes user-visible when
Running pods and container restarts	Working, enforced locally by the kubelet	Never, unless a node is lost or rebooted
Service routing and ClusterIPs	Working, from frozen kube-proxy rules	A backing pod dies and traffic keeps being sent to it
In-cluster DNS	Working, served from the CoreDNS cache	A CoreDNS pod restarts and cannot rebuild its cache
Endpoint and readiness updates	Stopped	Immediately for anything mid-rollout or newly unhealthy
Scheduling, scaling, self-healing	Stopped	The first pod exit or the first traffic increase
ConfigMap and Secret refresh	Stopped	An application expects a rotated credential or config
Projected token refresh	Stopped, with a deadline	Roughly an hour in, as tokens reach expiry

Three questions convert that into a pace. What is the external SLI right now, which decides whether you have minutes or hours. What is currently mid-change, since a stuck rollout, a scaling event, or a rotating credential turns a frozen cluster into a degrading one on a known schedule. And what happens the moment the control plane returns, because every deferred reconciliation fires at once: node leases are stale, controllers replay their queues, and the node controller may start evicting from nodes it briefly considers unreachable. That last risk has a built-in brake worth knowing about, since kube-controller-manager reduces its eviction rate when a large fraction of a zone looks unhealthy, which is precisely the state a control plane recovery creates.

Default: assume the dataplane is alive until you have measured otherwise, and let a green SLI buy you a slow, reversible recovery. Speed is only worth its risk when users are actually affected. The most expensive incidents in this course are the ones where a responder spent recoverable state to buy urgency that the situation never required.

---

Common Mistakes

Trusting in-cluster dashboards during a control plane outage. Prometheus, Grafana, and your alert pipeline all read from the thing that broke. Their silence is a symptom, not a measurement, and treating it as one manufactures the panic that causes the second mistake.

Assuming a NotReady node means its pods are down. NotReady describes the kubelet's ability to report, not the containers' ability to serve. Nodes routinely go NotReady while every pod on them continues answering requests normally.

Restarting things to see if it helps. On a frozen cluster nothing is watching to put back what you take down. Every restart is a one-way bet until a controller exists to reconcile it.

Forgetting that traffic is still reaching unready pods. Endpoint updates are stopped, so readiness has no effect. A pod that started failing after the control plane died is still in the rotation, and that, not the API outage, may be what your users are noticing.

Overlooking CoreDNS as the fragile piece. DNS survives on cached state and dies on restart. A CoreDNS eviction, node reboot, or OOM kill during a control plane outage converts a non-event into an application-wide failure within seconds.

Ignoring the token clock. Short-lived projected tokens expire on a schedule that does not care about your incident, and the resulting 401s surface as service-to-service failures rather than control plane failures, which sends responders chasing the wrong layer.

Declaring the incident over when kubectl responds. Recovery is the moment every deferred reconciliation fires simultaneously. Watch for mass pod churn, stale leases clearing, and controllers replaying queues before you close the bridge.

Skipping the assessment because the failure is obviously severe. Severity of cause and severity of impact are different axes. An unrecoverable etcd loss with a fully serving dataplane still gives you hours of user-visible normality to salvage in, and the Kubernetes Debugging for SREs course is the right companion when you need to work out which of your workloads is genuinely affected.

---