0/5 nodes are available: Kubernetes Stopped Naming the Taint

A fine tuning job had been Pending for 41 minutes. The GPU node pool was rebuilt overnight from a new node template. Nobody touched the workload. This is what kubectl describe had to say about it:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  41m   default-scheduler  0/5 nodes are available:
  1 Insufficient nvidia.com/gpu, 4 node(s) had untolerated taint(s).
  preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

Read that again, because it is doing less than it appears to. It accounts for every node in the cluster, it names a resource, it uses the word taint, and it does not contain the answer. There is no taint key in it. There is no value. There is nothing to grep for.

If you have been running Kubernetes long enough, you remember this message naming the taint directly, in braces, right there in the event. That form is gone. Here is where it went, and what to do instead.

What the message is actually telling you#

Split it into its two clauses, because they describe different nodes.

1 Insufficient nvidia.com/gpu is one node that has no GPUs. In this cluster it is cpu-node-1, the general purpose worker where monitoring and other cluster services run. It was never a candidate for a GPU workload and it is not the problem. It is noise, and it is the clause most people chase first because it is the only one naming something specific.

4 node(s) had untolerated taint(s) is the other four: the control plane plus the three GPU nodes. Those are grouped together under a single reason with no key, no value, and no indication of which node carried which taint. The control plane's taint and the GPU pool's taint are different taints with different purposes, and the event flattens them into one count.

So the event tells you the category of failure and the number of nodes in it. That is genuinely useful. It is not the same as telling you what changed, and the gap between those two is where the 41 minutes went.

The taint is not in the event, and that is on purpose#

This is not a bug and it is not your cluster. It is what the scheduler's taint plugin returns. From pkg/scheduler/framework/plugins/tainttoleration/taint_toleration.go in current Kubernetes, the Filter method returns an UnschedulableAndUnresolvable status whose message is exactly this string:

node(s) had untolerated taint(s)

No formatting verbs, no key, no value. The same file still defines a constant that reads node(s) had taints that the pod didn't tolerate, but Filter does not return it, so it never reaches your event.

The reason is aggregation. The scheduler collects a per node status for every node it rejected, then rolls identical reason strings up into counts so that a cluster with 500 nodes does not emit 500 lines. If the message embedded the taint, every distinct taint would be its own reason string, and the rollup would stop working on exactly the large clusters that need it most. There is also a hard ceiling: the scheduler truncates the final message at 1024 characters before writing it to the event, so a message that grew with cluster size would get cut off mid sentence anyway.

The tradeoff is defensible. It also means the event can no longer be the end of your investigation.

Where the information actually went#

The taint is not discarded. It is logged, per node, at verbosity 4.

On a cluster where you control the scheduler, add --v=4 to the kube-scheduler static pod manifest at /etc/kubernetes/manifests/kube-scheduler.yaml. The kubelet notices the file change and restarts the pod on its own. Then reschedule the pod and read the scheduler log:

kubectl -n kube-system logs kube-scheduler-<control-plane-node> | grep untoleratedTaint

That produces one line per rejected node, and each one names the taint:

taint_toleration.go:132] "node had untolerated taints"
  pod="ml-train/finetune-worker-0" node="cp-1"
  untoleratedTaint={"key":"node-role.kubernetes.io/control-plane","effect":"NoSchedule"}

taint_toleration.go:132] "node had untolerated taints"
  pod="ml-train/finetune-worker-0" node="gpu-node-1"
  untoleratedTaint={"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}

taint_toleration.go:132] "node had untolerated taints"
  pod="ml-train/finetune-worker-0" node="gpu-node-2"
  untoleratedTaint={"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}

taint_toleration.go:132] "node had untolerated taints"
  pod="ml-train/finetune-worker-0" node="gpu-node-3"
  untoleratedTaint={"key":"nvidia.com/gpu","value":"present","effect":"NoSchedule"}

Key, value and effect, broken out per node, with the control plane's taint cleanly separated from the GPU pool's. Everything the event used to carry.

Put --v=4 back to its previous setting when you are done. Verbosity 4 on the scheduler is chatty, and on a busy cluster it will fill a disk faster than you expect.

Why that does not help you on EKS, GKE or AKS#

On a managed control plane you do not get to pass flags to kube-scheduler. There is no static pod manifest to edit, the scheduler pod is not in your kube-system, and its logs are not yours to read. Which is to say: on the clusters most people run in production, the method above is unavailable.

That makes the next section the one that actually matters. It works everywhere, because it only reads objects through the API.

Diff the taints against the tolerations yourself#

The scheduler is doing a set comparison. You can do the same comparison from outside with the pod spec and the node list, and the answer is exact rather than inferred:

#!/usr/bin/env bash
set -euo pipefail
NS="${1:?usage: untolerated <namespace> <pod>}"
POD="${2:?usage: untolerated <namespace> <pod>}"

kubectl get pod "$POD" -n "$NS" -o json > /tmp/_pod.json
kubectl get nodes -o json > /tmp/_nodes.json

jq -r --slurpfile pod /tmp/_pod.json '
  ($pod[0].spec.tolerations // []) as $tols
  | .items[] | .metadata.name as $node
  | [ (.spec.taints // [])[]
      | select(.effect == "NoSchedule" or .effect == "NoExecute")
      | . as $taint
      | select(
          ($tols | map(
            (((.key // "") == "") and (.operator == "Exists"))
            or ( (.key == $taint.key)
                 and ( (.operator == "Exists")
                       or (((.operator // "Equal") == "Equal")
                           and ((.value // "") == ($taint.value // ""))) )
                 and (((.effect // "") == "") or (.effect == $taint.effect)) )
          ) | any) | not )
    ] as $blocking
  | select($blocking | length > 0)
  | "\($node)\t\($blocking | map("\(.key)=\(.value // "")(\(.effect))") | join(", "))"
' /tmp/_nodes.json | column -t -s $'\t'

Run against the stuck pod, it prints exactly which nodes are blocked and by what:

cp-1        node-role.kubernetes.io/control-plane=(NoSchedule)
gpu-node-1  nvidia.com/gpu=present(NoSchedule)
gpu-node-2  nvidia.com/gpu=present(NoSchedule)
gpu-node-3  nvidia.com/gpu=present(NoSchedule)

There is the answer the event would not give you. The GPU nodes are tainted nvidia.com/gpu=present. Now look at what the pod tolerates:

tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

The key matches. The value does not. The node template used to taint with true and now taints with present, and operator: Equal compares both fields. One word changed in a node template and a job stopped scheduling.

Note the set -euo pipefail and the unquoted-variable guards at the top. Without them, a typo in the pod name makes kubectl fail, $pod[0] evaluates to null, .spec.tolerations // [] becomes an empty list, and the script cheerfully reports that every tainted node in the cluster is blocking. Wrong answers that look right are worse than errors. I made exactly that mistake writing this.

After switching the workload to operator: Exists, the same command narrows to one line:

cp-1  node-role.kubernetes.io/control-plane=(NoSchedule)

The control plane is still blocked, which is correct and desirable, and the pod schedules onto a GPU node.

The four matching rules that actually bite#

Toleration matching is small enough to hold in your head, and every production surprise I have seen comes from one of these four:

1. operator: Equal compares key and value. This is the default when you omit operator entirely. If the taint's value changes, the toleration silently stops matching. This is the failure above and it is the most common one by a wide margin.

2. operator: Exists ignores the value. It matches the key whatever the value happens to be. For "I want to run on GPU nodes" this is almost always what you meant, because you care that the node is a GPU node, not which string the platform team chose this quarter.

3. An empty key with operator: Exists tolerates everything. Every taint, every effect. It is occasionally right for a monitoring DaemonSet that must run on every node including the control plane. It is almost never right for a workload, and it will put your batch jobs on nodes you were deliberately protecting.

4. An empty effect matches all effects. A toleration that specifies key and value but omits effect tolerates NoSchedule, PreferNoSchedule and NoExecute alike. That is wider than most people intend, particularly for NoExecute, which evicts running pods rather than just blocking new ones.

Check the age of the event before you trust it#

One more trap, and it costs people more time than the missing taint key does. Events are deduplicated and counted, not re-emitted. A describe on a long Pending pod looks like this:

Warning  FailedScheduling  45h (x23 over 46h)  default-scheduler  0/5 nodes are available: ...

That event fired 23 times across 46 hours, and the most recent firing was 45 hours ago. The scheduler backs off exponentially on a pod it cannot place, so a pod that has been Pending for two days may have been retried only a couple of dozen times, with hours between the last few attempts. The message you are reading describes the cluster as it was, not as it is.

This matters in both directions. If you have just changed a taint or a toleration, the event will not update to reflect it for a while, and you will read a stale failure as a current one. If something else changed the cluster hours ago, the event may predate it entirely.

Delete and recreate the pod, or check .status.conditions and the event's count and timestamps, before concluding that what you are reading is live. Everything in this post was verified against a freshly created pod for exactly that reason.

Quick reference#

When a pod is Pending and the event mentions taints:

Step	Command
Confirm the category	kubectl describe pod <pod> -n <ns>
Get the real taints	the jq script above, or kubectl get nodes -o json | jq '.items[] | {name:.metadata.name, taints:.spec.taints}'
Get the tolerations	kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.tolerations}' | jq
Compare key, value and effect	by eye, or the script
Self managed only	--v=4 on kube-scheduler, then grep untoleratedTaint

And ignore the Insufficient <resource> clause until you have ruled out the taint clause, unless the count covers every node you expected to be a candidate.

Why this matters beyond one Pending pod#

The useful lesson is not about taints. It is that a scheduling event narrows the category of problem and rarely hands you the field that changed. The scheduler told the truth, in a form optimised for a cluster with 500 nodes rather than for the person reading it at 3am. Both of those things can be true at once.

The production rule that falls out of it: keep taint keys and values in one shared node template, and treat a change to either as a breaking change for every workload that tolerates them. Prefer operator: Exists in workload tolerations when you care that a node is a GPU node rather than which value the taint happens to carry. If you need the stronger match, add an admission policy that rejects workloads whose tolerations match no taint currently in the cluster, so the mismatch is caught at deploy time instead of after a node pool rebuild.

A node pool rebuild is a routine, scheduled, low drama operation. It is also a rename of every taint value on every node in that pool, applied at once, to a cluster full of workloads that encoded the old value in their specs. Those two sentences describe the same event.

---

Scheduling for GPU node pools, taints and tolerations, dedicated pools, priority classes and the operational behaviour of GPU nodes under contention are covered in depth in Production GPU Infrastructure on Kubernetes. The systematic approach to Pending pods and the rest of the production failure catalogue lives in Kubernetes Debugging for SREs, and node pool design and upgrade operations in Production Kubernetes Operations. Related reading: What Happens When You Create a Pod for where the filter phase sits in the full scheduling path, The Device Plugin Era Is Ending for device level taints and what DRA changes about GPU scheduling, and Spot vs On-Demand GPU Economics for why spot pools get taints in the first place.