What the CNI Decides
Two clusters, same Kubernetes version, same manifests. Pod to pod traffic behaves differently in each. Nothing in the Kubernetes documentation explains why, because the difference is not in Kubernetes.
The previous lesson drew the line. This one is about what lives on the far side of it, which turns out to be almost everything you will ever operate.
The useful surprise here is how little the CNI specification actually standardises. People assume "CNI compliant" means plugins behave alike in the ways that matter. It means something much narrower, and knowing exactly how narrow is what stops you expecting portability you do not have.
What is actually happening
CNI is a specification for how a container runtime asks something else to set up networking for a container. That is the entire scope. It defines an interface, not a network.
Concretely, it standardises four things:
An invocation contract. The runtime executes a binary, passes configuration on stdin as JSON and context in environment variables, and reads a result from stdout. That is the whole calling convention. A CNI plugin is a program the runtime runs, not a service it talks to.
A set of operations. ADD when a container needs networking, DEL when it goes away, CHECK to verify, VERSION to negotiate. The interesting one is ADD.
A result format. The plugin reports back what it did: which interfaces it created, which addresses it assigned, which routes it installed.
Chaining. Plugins can be composed, so one plugin handles addressing and another adds policy or bandwidth limits on top.
What it does not standardise is the part you care about. Nothing about how addresses are chosen. Nothing about how a packet reaches another node. Nothing about performance, encryption, policy enforcement, or observability. Two plugins can both be entirely compliant and share almost no behaviour.
CNI compliance means a plugin implements a calling convention correctly. It says nothing about capability. A minimal plugin that satisfies the specification and does nothing beyond the four Kubernetes requirements is exactly as compliant as one offering policy, encryption, multi-cluster reachability and full flow observability. When someone says "we are CNI compliant," they have told you the plugin can be invoked, and nothing else.
The Kubernetes model versus the CNI's choice
Here is what happens when a pod is created, with the required steps separated from the chosen ones.
The kubelet creates the pod sandbox, which includes a fresh network namespace with nothing in it but a loopback interface. Then it invokes the CNI plugin with ADD, passing the namespace path and the container identity.
From that moment until the plugin returns, everything is the plugin's decision. It must produce a namespace satisfying the four requirements. How it does that is unconstrained.
IPAM, meaning address allocation. The plugin picks an address for this pod. Whether that comes from a per-node range carved out of a cluster CIDR, from an address pool managed centrally, or from the underlying cloud network's own address space is a plugin decision with large consequences. It determines whether you can exhaust addresses, whether pod addresses are routable outside the cluster, and how quickly a node can start pods.
Interface creation. The plugin creates an interface inside the pod namespace and connects it to the host. The overwhelmingly common approach is a virtual ethernet pair, and it is not the only possibility. Some plugins attach the pod directly to a hardware interface, which changes the performance profile completely.
Route installation. The plugin writes the pod's routing table, deciding what the pod's default gateway is and how it reaches the rest of the cluster. Some plugins use a real gateway address on a bridge. Others use a link local address that is not assigned to anything and answer for it with a kernel flag. Both are correct, and they debug completely differently.
Host side attachment. What the other end of the pod interface connects to. A software bridge shared by all pods on the node, or an individual route per pod, or a program attached to the interface. This is the decision that most affects what you see when you go looking on the node.
Then the plugin returns, the kubelet reads the result, and the pod has a network.
How it works in practice
You can read your plugin's decisions directly, and you should, because most teams have never looked.
The configuration the runtime uses lives on every node:
# On a node. The runtime reads this directory to decide what to invoke.
ls /etc/cni/net.d/
10-<plugin>.conflist
cat /etc/cni/net.d/10-<plugin>.conflistThat file names the plugin, any chained plugins, and the IPAM configuration. It is the single most informative file on the node for this purpose and almost nobody opens it.
Then compare what the pod believes against what the host believes:
# What the plugin gave the pod
kubectl exec -it payments-api-0 -- ip addr show eth0
kubectl exec -it payments-api-0 -- ip route
# On the node hosting it, the other side of the same connection
ip link
ip route | grep 10.0The two views together tell you which of the choices above your plugin made. A default route pointing at a real address on a bridge means one design. A default route pointing at an address that appears nowhere else means another. Neither is wrong and they are debugged with different commands.
When you join a team, run those commands once and write the answers down. Which plugin, how addresses are allocated, whether traffic between nodes is encapsulated, what the host side of a pod interface attaches to, and whether NetworkPolicy is actually enforced. Five facts. They determine what every future networking investigation looks like, and the alternative is rediscovering them under pressure.
The kernel primitives underneath, meaning namespaces and virtual interfaces as mechanisms in their own right, are the subject of Container Internals and Runtime Engineering. Here we care about which of them your plugin chose and what that implies.
Common failure modes
The plugin fails and the pod is stuck. When ADD fails, the pod cannot get a network, so it sits unready with an error mentioning the network setup rather than the application. The instinct is to look at the workload. The cause is on the node, in the plugin's own logs, which are usually in its DaemonSet pod rather than anywhere you would think to look.
Address allocation fails silently at the node level. The plugin cannot allocate, pods stop scheduling successfully on that node, and the rest of the cluster is fine. This looks like a scheduling problem and is not.
Configuration drift between nodes. The plugin configuration is a file on each node. If nodes were provisioned at different times or from different images, they can differ. This produces the worst class of problem in this course: behaviour that depends on which node the pod landed on.
Chained plugins failing independently. When one plugin does addressing and another adds policy or bandwidth control, either can fail. The symptom is often the failure of the second one while the pod appears to have a perfectly good network.
A cluster was expanded with a new node pool built from a newer machine image. Pods scheduled on the new nodes worked, mostly. Occasionally a connection between a new node and an old one failed for large payloads only. The plugin configuration on the new image had a different MTU default from the old one, so pods on the two pools disagreed about the largest packet they could send. Every Kubernetes requirement was satisfied on both pools. The plugin was compliant on both. The two pools simply had different plugin configuration, and nothing in Kubernetes describes, validates, or reconciles that file. The fix was one number. Finding it took three days because everyone was looking at the workload, which was identical, rather than at the node, which was not.
What people get wrong
Assuming CNI compliance implies comparable behaviour. It implies a calling convention. Nothing else.
Believing the plugin is a running service. For the addressing path, most plugins are a binary the runtime executes. There is usually also a long running agent doing routing and policy, and the two fail differently. Knowing which part you are debugging matters.
Expecting the plugin to be swappable. Changing it means renumbering every pod and, in practice, rebuilding the cluster. It is a founding decision that looks like a configuration choice.
Forgetting the plugin has its own logs. When networking is broken at the pod creation stage, the answer is usually in the plugin's agent logs, not in the kubelet and not in the workload.
Treating the plugin's extra features as portable. Policy, encryption and observability beyond the four requirements are genuinely useful and genuinely not guaranteed. Depending on them is a reasonable decision to make deliberately and an expensive one to make by accident.
A pod is stuck not ready. Its events mention that setting up the network for the sandbox failed. The application image has not changed and the same manifest works on other nodes. Where do you look first?
The plugin is where your cluster's networking personality comes from. The specification guarantees it can be called. Everything about how it behaves, and therefore everything you will debug, is a choice it made that Kubernetes never saw.
Walk through what a CNI plugin does when a pod is created. Which of its behaviours are required by the specification and which are the plugin's own choice?