IP Address Management

The cluster has been running for a year. A new node joins and no pods can be scheduled on it. Nothing is broken. You ran out of addresses, and the decision that caused it was made at cluster creation.

Of everything in this course, this is the lesson most likely to save you from a cluster rebuild, because the mistake is made on day one and discovered in year two, and by then the fix is expensive.

What is actually happening

The common allocation model works in two layers, and the arithmetic between them is where clusters die.

There is a cluster CIDR, one range covering every pod address in the cluster. Something carves that range into per-node blocks, and each node allocates addresses to its own pods from its own block, locally, without asking anyone.

That local allocation is deliberate. A node starting a pod must assign an address immediately, and a design requiring a call to a central allocator would put the control plane in the path of every pod start. Giving each node a block up front removes that entirely.

The price is that the block is reserved whether or not it is used. A node allocated 256 addresses holds all 256, even when running four pods.

So the ceiling is arithmetic:

maximum nodes  =  size of the cluster CIDR  /  size of each per-node block

Work an example. Cluster CIDR 10.0.0.0/16, which is 65,536 addresses. Per-node block of /24, which is 256 addresses. That gives 256 blocks, so 256 nodes maximum, no matter how many pods you actually run. Grow to node 257 and it gets no block, and pods cannot start on it.

Notice what is not in that equation: the number of pods you run. A cluster running six pods per node hits the same ceiling as one running a hundred, because the block is reserved per node.

The Kubernetes model versus the CNI's choice

The model requires only that every pod gets an address. Allocation is entirely delegated, and there are two broad approaches with very different exhaustion behaviour.

Blocks from a cluster CIDR. The arrangement above. Addresses come from a private range that exists only inside the cluster. Exhaustion is a function of the two numbers, and it is entirely predictable if anyone does the arithmetic.

Addresses from the underlying network. Some plugins, particularly cloud native ones, give pods real addresses from the same address space as the nodes. Pods become routable on the wider network with no encapsulation, which is a genuine advantage.

The exhaustion story changes completely. You are no longer limited by a range you invented. You are limited by the real address space of the subnet the cluster sits in, which you probably share with other things, and which may have been sized by a network team years ago with no knowledge of Kubernetes. Teams choosing this model routinely discover that a subnet comfortable for a hundred virtual machines is exhausted by three thousand pods.

There is also a per-node limit in this model that catches people out: how many addresses a node can hold is often capped by the cloud provider's own limits on the node's network interfaces, and that cap varies by instance type. Change to a smaller instance type and the pod density ceiling drops with it.

How it works in practice

Do the arithmetic for your cluster now. It takes two minutes and the answer is either reassuring or the most important thing you learn this week.

# The per-node blocks the control plane assigned
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

node-01   10.0.0.0/24
node-02   10.0.1.0/24
node-03   10.0.2.0/24

Then find the cluster CIDR, which lives in the control plane configuration rather than in any object you can query directly. On a self managed cluster it is a flag on the controller manager; on a managed one it is a property of the cluster in the provider API.

With both numbers you have your ceiling. Compare it against your growth: current node count, expected in a year, and the node count during a rolling upgrade, which is higher than steady state because old and new nodes coexist.

Also check whether your per-node block is generous relative to your actual pod density. A /24 giving 256 addresses to nodes that never run more than 30 pods is wasting seven eighths of your address space, and that waste is what converts into a premature node ceiling.

Common failure modes

Node ceiling reached. New nodes join, go Ready, and cannot start pods. Every existing node is fine. The scheduler keeps trying and pods stay Pending. It looks like a scheduling problem and it is an address problem.

Underlying subnet exhausted. Under the second allocation model, the cluster consumes the shared subnet and something entirely outside Kubernetes fails to get an address. This is the version that makes the network team unhappy, and it is not visible from inside the cluster at all.

Per-node density capped by instance type. Under cloud native addressing, a node cannot hold more addresses than its interfaces allow. Pods stay Pending on a node with plenty of CPU and memory free, which is a genuinely confusing signal.

Addresses not released promptly. Pods churn quickly, addresses are not reclaimed at the same rate, and a node exhausts its block despite running far fewer pods than the block holds. Usually a plugin issue and it presents identically to genuine exhaustion.

Overlapping ranges with something else. The cluster CIDR was chosen from a private range already used elsewhere in the organisation. Everything works until a pod tries to reach that other thing, at which point packets go somewhere unintended and the symptom looks nothing like an addressing problem.

What people get wrong

Sizing the cluster CIDR by pod count. The ceiling is set by node count multiplied by block size, and the block is reserved regardless of use. Sizing by expected pods produces a range that looks generous and is not.

Assuming the range can be grown later. In practice it cannot. Every running pod holds an address from the current arrangement, and there is no in place renumbering.

Forgetting that Services need addresses too. The Service CIDR is a separate range with its own ceiling, and it is separately exhaustible. The Services module covers what those addresses actually are, which is stranger than it sounds.

Treating it as an operations detail. It is an architecture decision with a permanent consequence, made once, usually by a default.

Ignoring the difference between the two allocation models. Whether pod addresses come from a private cluster range or from the real network changes what exhaustion means, who else it affects, and who has to be in the room to fix it.

Two numbers, one division, and a decision you cannot revisit. Do the arithmetic on every cluster you are responsible for, include the peak during upgrades rather than the steady state, and write the answer somewhere your capacity planning will actually see it.