Version Skew Policy Deep Dive

You upgraded the control plane to 1.30 last week. Everything looked fine: the API server was healthy, the scheduler was running, etcd was happy. You planned to upgrade the kubelets "next sprint."

Then the tickets started. Pods were not scheduling on certain nodes. A StatefulSet was stuck in a rolling update. The kubelet on three nodes was reporting MatchNodeSelectorTerms errors that did not exist before the control plane upgrade.

The kubelets were still on 1.28. The control plane was on 1.30. That is a two-version gap, and it broke things. Not because 1.28 kubelets cannot talk to a 1.30 API server (they can, technically), but because the API server started returning fields in a format the old kubelets did not understand.

This is the version skew policy. It is not a suggestion. It is the line between "upgrade worked" and "upgrade broke production."

---

Part 1: The Skew Rules

Kubernetes is a distributed system with multiple components that talk to each other over APIs. The version skew policy defines which version combinations are supported and tested. Running outside these bounds means you are in untested territory: things might work, or they might break in subtle, hard-to-debug ways.

The Core Components and Their Relationships

The components that matter for version skew:

Component	Role	Runs On
kube-apiserver	Central API, all components talk to it	Control plane nodes
kube-controller-manager	Runs controllers (deployments, replicasets, etc.)	Control plane nodes
kube-scheduler	Assigns pods to nodes	Control plane nodes
kubelet	Manages pods on each node	Every node
kube-proxy	Manages network rules on each node	Every node
kubectl	CLI client	Developer machines, CI/CD
etcd	Key-value store for all cluster state	Control plane nodes (or separate)

The Rules

Here are the supported version skew ranges, where n is the kube-apiserver version:

Let us break down each rule:

kube-apiserver to kube-controller-manager / kube-scheduler: n-1 to n

The controller manager and scheduler must be within one minor version of the API server. If your API server is 1.30, these components can be 1.29 or 1.30.

In practice: always upgrade the API server first, then the controller manager and scheduler. They can lag by one version during the upgrade window.

kube-apiserver to kubelet: n-3 to n

Kubelets can be up to three minor versions older than the API server. If your API server is 1.30, kubelets can be 1.27, 1.28, 1.29, or 1.30.

In practice: this is the most generous skew allowance because upgrading hundreds of worker nodes takes time. The 3-version window gives you room to roll out node upgrades gradually.

kube-apiserver to kube-proxy: n-3 to n

Same as kubelet. kube-proxy runs on every node alongside the kubelet and follows the same skew rules.

kubectl to kube-apiserver: n-1 to n+1

kubectl can be one version older or one version newer than the API server. If your API server is 1.30, kubectl 1.29, 1.30, or 1.31 all work.

In practice: developers and CI/CD pipelines often have kubectl versions that drift. This is usually fine, but if someone is running kubectl 1.27 against a 1.30 cluster, they may see unexpected behavior.

# Check kubectl version
kubectl version --client -o json | jq '.clientVersion.gitVersion'

# Check API server version
kubectl version -o json | jq '.serverVersion.gitVersion'

# Compare them — they should be within 1 minor version

etcd to kube-apiserver

etcd version compatibility is documented separately. The Kubernetes project tests specific etcd versions with each release. Generally:

Kubernetes Version	Supported etcd Versions
1.28 - 1.30	3.5.x
1.31+	3.5.x (check release notes)

---

Part 2: Why Skew Breaks Things

The version skew policy is not arbitrary. It exists because of real incompatibilities that surface when components disagree on API versions, feature gates, and default behaviors.

API Deprecations and Removals

When the API server is upgraded to a version that removes an API, older kubelets and controllers that still use that API will fail.

Example: The flowcontrol.apiserver.k8s.io/v1beta2 API was removed in 1.29. If your controller manager is 1.28 and still references this API, it will get 404 errors from a 1.29 API server.

# This worked on 1.28 API server
kubectl get flowschemas.flowcontrol.apiserver.k8s.io/v1beta2

# On 1.29 API server: error
# error: the server does not support API version "flowcontrol.apiserver.k8s.io/v1beta2"

Changed Defaults and Feature Gates

Some feature gates change between versions. A feature that was disabled by default in 1.29 might become enabled by default in 1.30. If the kubelet is on 1.29 (feature off) and the API server is on 1.30 (feature on), the two components may disagree on behavior.

Example: The PodDisruptionConditions feature gate graduated to stable in 1.31 and the gate was removed. The API server on 1.31 always adds disruption conditions to pods. A 1.28 kubelet does not understand these conditions and ignores them, which is usually fine. But a 1.28 controller that watches for specific pod conditions might not see the disruption events it expects.

Field Validation Changes

The API server validates fields more strictly over time. A field that was accepted with a warning in 1.29 might be rejected outright in 1.30. If an older kubelet or controller sends requests with deprecated field formats, the newer API server may reject them.

---

Part 3: The Upgrade Path: One Version at a Time

The single most important rule of Kubernetes upgrades:

You must upgrade one minor version at a time.

You cannot go from 1.27 to 1.30 directly. You must go 1.27 to 1.28, then 1.28 to 1.29, then 1.29 to 1.30. Each hop requires its own validation and testing.

Why You Cannot Skip Versions

1. Schema migrations: etcd data may undergo schema changes between versions. Each migration is designed to work from version n to version n+1. Skipping a migration step can corrupt data.

2. Feature gate removals: A feature gate removed in 1.29 means the feature is always on in 1.29+. If you jump from 1.27 (where you could disable it) to 1.30, you miss the transition period and may hit issues.

3. API conversions: The API server converts between stored versions and requested versions. Conversion logic is only tested for adjacent versions.

4. Testing matrix: The Kubernetes project only tests upgrades from n-1 to n. No one tests 1.27 to 1.30 directly. You are on your own.

# The correct upgrade path from 1.27 to 1.30:

# Hop 1: 1.27 → 1.28
kubeadm upgrade plan    # shows available 1.28.x versions
kubeadm upgrade apply v1.28.12
# ... upgrade kubelets to 1.28.x ...
# ... validate everything works ...

# Hop 2: 1.28 → 1.29
kubeadm upgrade apply v1.29.7
# ... upgrade kubelets to 1.29.x ...
# ... validate everything works ...

# Hop 3: 1.29 → 1.30
kubeadm upgrade apply v1.30.3
# ... upgrade kubelets to 1.30.x ...
# ... validate everything works ...

The Upgrade Sequence Within a Single Version Hop

Within each minor version upgrade, the component upgrade order matters:

1. etcd: upgrade first if needed (check compatibility matrix)
2. kube-apiserver: upgrade next (all other components can tolerate a newer API server)
3. kube-controller-manager: upgrade after API server
4. kube-scheduler: upgrade after API server (can be parallel with controller-manager)
5. kubelet + kube-proxy: upgrade last, node by node
6. kubectl: upgrade on developer machines and CI/CD

---

Part 4: Checking Your Current Skew

Before planning an upgrade, audit the version skew across your cluster.

# Full component version audit
echo "=== API Server ==="
kubectl version -o json | jq '.serverVersion.gitVersion'

echo "=== Controller Manager ==="
kubectl get pods -n kube-system -l component=kube-controller-manager \
  -o jsonpath='{.items[0].spec.containers[0].image}'

echo "=== Scheduler ==="
kubectl get pods -n kube-system -l component=kube-scheduler \
  -o jsonpath='{.items[0].spec.containers[0].image}'

echo "=== etcd ==="
kubectl get pods -n kube-system -l component=etcd \
  -o jsonpath='{.items[0].spec.containers[0].image}'

echo "=== Kubelet versions (all nodes) ==="
kubectl get nodes -o json | jq -r \
  '.items[] | "\(.metadata.name)\t\(.status.nodeInfo.kubeletVersion)"'

echo "=== kube-proxy ==="
kubectl get daemonset -n kube-system kube-proxy \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

echo "=== kubectl ==="
kubectl version --client -o json | jq '.clientVersion.gitVersion'

Look for these red flags in the output:

• Kubelets at different versions: some nodes may have missed a previous upgrade
• Controller manager or scheduler lagging, they should match the API server
• etcd version mismatch across replicas, all etcd members must be the same version
• kubectl more than one version away from the API server, developers may see unexpected behavior

---

Part 5: HA Control Plane Skew During Upgrades

If you run a highly available (HA) control plane with multiple API server instances, the API servers themselves can be at different versions during the upgrade. Kubernetes supports this:

• kube-apiserver to kube-apiserver: n to n+1, during a rolling upgrade, API servers can differ by one minor version
• Traffic may be routed to either version, clients may see inconsistent behavior during the upgrade window
• Keep this window as short as possible

# Check all API server pod versions in HA setup
kubectl get pods -n kube-system -l component=kube-apiserver \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

---

Key Concepts Summary

• kube-apiserver is the reference version, all other component skew is measured relative to the API server
• Controller manager and scheduler: within 1 version of the API server (n-1 to n)
• Kubelet and kube-proxy: within 3 versions of the API server (n-3 to n)
• kubectl: within 1 version in either direction (n-1 to n+1)
• You must upgrade one minor version at a time, no skipping from 1.27 to 1.30
• Upgrade order matters: etcd first, then API server, then controller-manager/scheduler, then kubelets
• Version skew breaks things silently, wrong behavior without error messages is harder to debug than crashes
• HA control planes have a transient skew window, keep it short during rolling upgrades

Common Mistakes

• Upgrading the control plane and leaving kubelets "for later" with a 3+ version gap, technically possible for n-3 but begging for subtle bugs
• Upgrading controller-manager or scheduler before the API server, violates the supported sequence
• Not checking kubelet versions across all nodes before upgrading, one node at 1.27 in a 1.30 cluster is a ticking time bomb
• Assuming kubectl version does not matter, a 2+ version gap can cause confusing CLI behavior
• Upgrading etcd and the API server simultaneously, if something breaks, you will not know which component caused it
• Not backing up etcd before starting any upgrade, if the schema migration fails, you need to restore

---