Kubernetes Specifies Four Things About Networking. Your CNI Does the Rest

Two clusters, same Kubernetes version, same manifests. Pod to pod throughput is measurably lower in one of them and large payloads occasionally fail there. You search the Kubernetes networking documentation and find nothing about MTU, nothing about throughput, nothing that explains it.

That is not a gap in the documentation. The behaviour was never specified. Kubernetes states four requirements about networking and stops, and almost everything that will page you lives on the other side of that line, in a component you installed separately and whose documentation is somewhere else entirely.

The four requirements, and nothing else#

The model is four statements about what must be true, with no instruction about how.

Every pod gets its own IP address. Not a shared address with port mapping, not an address borrowed from the node. Its own, from a range the cluster controls.

Pods can communicate with all other pods without NAT. A pod on one node reaches a pod on another node directly, and the destination sees the real source address.

Nodes can communicate with all pods without NAT, and pods can reach nodes the same way. This is what lets the kubelet reach a pod to run a health probe.

The IP a pod sees itself as is the IP others see it as. Run hostname -i inside the pod and the answer is the address the rest of the cluster uses to reach it.

Read that again and notice what is absent: nothing about how addresses are allocated, nothing about how packets cross a node boundary, nothing about encryption, policy, performance, or what happens when you run out of addresses.

The fourth one is the one people skim, and it does the most work. It is what lets an application advertise its own address to a peer and have that address work, which is the foundation of peer to peer protocols and service registration. It also rules out the entire class of solutions that would otherwise be easiest: port mapping, address translation between pods, anything where a pod's identity depends on where you are standing when you look at it.

The requirements are phrased as requirements rather than as a design on purpose. Kubernetes runs on cloud VPCs, on bare metal with real routers, on a laptop, and inside another Kubernetes cluster. No single networking implementation is correct across all of those, so the project specified the contract and left the implementation open. That is why Kubernetes networking is portable. It is also why it is confusing.

What that leaves to your plugin#

Everything not in those four statements is delegated to the Container Network Interface plugin, which you install separately and which most teams inherit from whatever their cluster provisioner chose.

Kubernetes requires	Your CNI decides
Every pod has an IP	Which address, from which range, allocated how
Pods reach pods without NAT	Whether that happens by encapsulation or native routing
Nodes reach pods without NAT	What the host side of the pod interface attaches to
A pod's address is consistent	Everything about performance and MTU
	Whether NetworkPolicy is enforced at all
	What you can observe, and how
	Encryption between nodes, if any

Look at the right hand column. Address exhaustion, MTU failures, policy that silently does nothing, traffic you cannot see. Those are the problems you will actually have, and none of them are Kubernetes problems.

It is worth being precise about what the CNI specification standardises, because "CNI compliant" gets read as "behaves like the other ones". As of CNI spec 1.1.0 it standardises an invocation contract (the runtime executes a binary, passes JSON on stdin and context in environment variables, reads a result from stdout), a set of operations (ADD, DEL, CHECK, GC and VERSION, with STATUS defined separately and explicitly informational), a result format, and plugin chaining.

That is the entire scope. It defines an interface, not a network. Two plugins can both be fully compliant and share almost no behaviour.

Where the choice actually gets made#

When a pod is created, the kubelet creates the sandbox, which includes a fresh network namespace containing nothing but a loopback interface. Then it invokes the plugin with ADD, passing the namespace path and the container identity. The broader sequence is in what happens when you create a pod.

From that invocation until the plugin returns, every decision is the plugin's:

• Address allocation. From a per-node range carved out of a cluster CIDR, from a centrally managed pool, or from the underlying cloud network's own address space. This determines whether you can exhaust addresses, whether pod addresses are routable outside the cluster, and how fast a node can start pods.
• Interface creation. Usually a virtual ethernet pair. Some plugins attach the pod directly to hardware, which changes the performance profile completely.
• Route installation. What the pod's default gateway is. Some plugins use a real address on a bridge. Others use a link local address assigned to nothing and answer for it with a kernel flag. Both are correct and they debug completely differently.
• Host side attachment. A software bridge shared by all pods on the node, an individual route per pod, or a program attached to the interface. This is the decision that most affects what you see when you go looking on the node.

Then the plugin returns, the kubelet reads the result, and the pod has a network.

How to tell which side of the line you are on#

Start with what your cluster actually runs, because a surprising number of engineers do not know:

# What is providing the network? Look for the CNI's own workloads.
kubectl get daemonset -n kube-system

NAME              DESIRED   CURRENT   READY   AGE
kube-proxy        6         6         6       284d
<your-cni-agent>  6         6         6       284d

Then what it decided about addresses:

# The per-node pod range
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

node-01   10.244.0.0/24
node-02   10.244.1.0/24
node-03   10.244.2.0/24

And how packets get between those ranges, which you can only see from a node:

# On a node. The answer here is entirely your CNI's choice.
ip route | grep 10.244

10.244.1.0/24 via 10.0.1.12 dev eth0        # native routing, next hop is another node
10.244.2.0/24 dev <tunnel-interface>        # or an overlay, via a tunnel device

The single most informative file for this is the one almost nobody opens:

# On a node. The runtime reads this directory to decide what to invoke.
ls /etc/cni/net.d/
10-<plugin>.conflist

cat /etc/cni/net.d/10-<plugin>.conflist

That names the plugin, any chained plugins, and the IPAM configuration.

Finally, compare what the pod believes against what the host believes:

# What the plugin gave the pod
kubectl exec -it payments-api-0 -- ip addr show eth0
kubectl exec -it payments-api-0 -- ip route

# On the node hosting it, the other side of the same connection
ip link
ip route | grep 10.0

Those commands tell you more about how your cluster networks than any amount of reading about Kubernetes, because most of what they show you are decisions Kubernetes never made.

The silent one: policy that applies and does nothing#

Most of these failures are loud. One is not.

NetworkPolicy is a Kubernetes API, so a NetworkPolicy manifest is valid Kubernetes YAML and the API server accepts it on any conforming cluster. Enforcement is not part of the model. A plugin that ignores policy entirely is fully conforming.

So a team writes policies, tests them on a cluster whose plugin enforces them, and ships to a cluster whose plugin does not. The manifests apply cleanly on both. On one of them they do nothing at all, silently, and kubectl get networkpolicy looks identical either way.

There is no API field to check for this. The only reliable test is to assert a policy is working from inside the cluster: apply a default deny, then confirm from a pod that traffic you expect to be blocked is actually blocked. The mechanics of writing policies that hold up are in default deny and egress control.

The fixes, cheapest first#

1. Classify before you search. Decide which side of the line the behaviour is on. This costs nothing and is the difference between an hour and an afternoon.

2. Read /etc/cni/net.d/ on a node. One file tells you the plugin, the chain and the IPAM configuration. Do this once and write the answer down somewhere your team can find it.

3. Record your plugin and its version alongside your Kubernetes version. Runbooks that name only the Kubernetes version send the on call engineer to the wrong documentation under pressure.

4. Test NetworkPolicy enforcement, do not assume it. One default deny plus one blocked connection attempt, in CI if you can. A policy you have never seen block anything is not a control.

5. Size the pod CIDR for the cluster you will have. Address exhaustion is a provisioning time decision that becomes expensive to change later, and the symptom arrives as pods stuck in ContainerCreating rather than as anything that says "addresses".

6. Stop expecting conformance to mean similarity. Conformance tests the four requirements. Two conforming clusters can differ in every way that will actually affect you.

In production#

When something about networking surprises you, decide which side of the line it is on before you do anything else. Four requirements are Kubernetes. Everything else, including nearly everything that will page you, belongs to a component the Kubernetes documentation does not describe, and the hour you spend in the wrong manual is the most common avoidable cost in cluster networking.

Once you know the behaviour is yours rather than Kubernetes', the next hops are the request path in how a request reaches your pod, the Service layer in kube-proxy iptables versus IPVS, and resolution behaviour in the ndots problem.

The network model, what each plugin class decides, and how to debug across the line are module 1 of Kubernetes Networking, whose first lessons are free.