Pod Networking Across Nodes

The pod on node A needs to reach a pod on node B. The underlying network has never heard of pod addresses. Two broad approaches solve this, and the choice constrains everything downstream.

This is the decision that shapes your cluster's networking more than any other, it was almost certainly made by whoever provisioned the cluster rather than by you, and most teams cannot say which one they have.

What is actually happening

Start with the problem, because it is easy to miss how strange it is.

Pod addresses come from a range the cluster invented. The physical or virtual network your nodes sit on knows nothing about that range. Its routers have no entry for it. If a node simply put a packet addressed to 10.0.3.42 onto the wire, the next device would have nowhere to send it and would drop it.

Yet the model requires that a pod on one node reaches a pod on another without translation, and that the destination sees the real source address. So something has to bridge a gap between an address space the cluster made up and a network that has never heard of it.

There are exactly two shapes of answer.

Hide the pod addresses from the network. Wrap the original packet inside another packet addressed from node to node. The underlying network only ever sees node to node traffic, which it understands perfectly. The receiving node unwraps and delivers. The pod addresses were never visible to anything in between.

Teach the network about pod addresses. Make the underlying network aware of which node owns which pod range, so it can route pod addressed packets natively with no wrapping at all.

Everything else is a variation on those two.

The Kubernetes model versus the CNI's choice

The model requires the outcome and says nothing about the method. Here is what each choice actually costs.

Overlay encapsulation puts the original packet inside an outer header, commonly using a UDP based tunnelling scheme, addressed from the source node to the destination node. The underlying network sees ordinary node to node traffic.

What it costs you:

• MTU. The outer header consumes bytes. If the underlying network carries 1500 byte packets and the header takes 50, your pods can only send 1450. Get this wrong and you get the failure mode where small requests work and large ones hang, which is the subject of Module 9.
• CPU. Wrapping and unwrapping every packet is work. Modern kernels and network cards offload much of it, and it is not free.
• Opacity. Anything inspecting the network between nodes sees tunnel traffic. Existing firewalls, flow logs and monitoring lose visibility into what is actually being carried.

What it buys you: it works on any network, immediately, with no cooperation from anyone.

Native routing installs routes so the network knows that 10.0.3.0/24 lives on node B. Packets go out with pod addresses on them, unwrapped, and the network handles them like any other traffic.

What it costs you: the network has to learn those routes, by one of two mechanisms. In a cloud, the plugin programs the provider's route table, which ties you to that cloud's API and its limits on route table size. On your own network, nodes speak a routing protocol to your routers, which means your network team is now part of your cluster's control plane.

What it buys you: full MTU, no encapsulation overhead, and pod addresses that are visible and routable, which makes everything from firewall rules to packet capture behave normally.

How it works in practice

You can tell which one you have in a single command, and it is worth doing now rather than during an incident:

# On a node. How does it reach another node's pod range?
ip route | grep 10.0

# Native routing: the next hop is another node's real address
10.0.1.0/24 via 192.168.4.12 dev eth0
10.0.2.0/24 via 192.168.4.13 dev eth0

# Overlay: the route points at a tunnel device instead
10.0.1.0/24 dev <tunnel-interface> scope link
10.0.2.0/24 dev <tunnel-interface> scope link

Then check the MTU, because that is where the encapsulation cost becomes a number:

# The node's real interface, and the pod's
ip link show eth0 | grep mtu
kubectl exec -it payments-api-0 -- ip link show eth0 | grep mtu

If the pod MTU is lower than the node MTU, you are encapsulating and the plugin has accounted for it. If they are equal and you are encapsulating, that is a misconfiguration waiting to produce intermittent large payload failures.

To see it happening, capture on the node while sending pod to pod traffic across nodes. Under an overlay you will see tunnel traffic between node addresses. Under native routing you will see the pod addresses directly on the wire, which is the clearest possible demonstration that the network genuinely knows about them.

Common failure modes

MTU mismatch. The pod MTU is set too high for the encapsulation in use. Small packets are fine. Large ones need to fragment, something in the path will not, and they are dropped. The signature is that small requests succeed and large ones hang, which looks nothing like a network problem and everything like an application bug.

Cloud route table limits. Native routing in a cloud consumes an entry per node. Route tables have limits. The cluster grows past the limit and new nodes come up with pods that cannot be reached from anywhere else, while every existing node keeps working perfectly.

Routing protocol sessions failing. Where nodes advertise routes to your network, those sessions can drop. When one does, that node's pods become unreachable from other nodes while the node itself is healthy, ready, and happily accepting scheduled work.

Underlying network filtering the tunnel. Encapsulation uses specific ports. A security group or firewall rule that does not permit them produces a cluster where pods talk fine within a node and not at all between nodes. Extremely common on first setup and unmistakable once you know the signature.

Asymmetry between node pools. Nodes provisioned at different times with different plugin configuration, disagreeing about MTU or mode. Produces failures that depend on which pair of nodes the two pods landed on.

What people get wrong

Believing the choice is only about performance. The performance difference is real and usually smaller than expected. The differences that matter more are whether traffic is visible to your existing network tooling, and whether your cluster now has a dependency on a route table limit or a routing protocol session.

Assuming overlay is the beginner option. It requires nothing of the network, which makes it the correct answer in many serious production environments, particularly where the cluster does not own the network it sits on.

Forgetting that native routing makes pod addresses real. They become routable, which means they can be reached, filtered and logged by things outside the cluster. That is often the whole reason to choose it, and it is also a security consideration rather than only a networking one.

Treating the choice as changeable. Switching modes means renumbering or restarting every pod and, in practice, is a cluster rebuild. It is a founding decision.

Two shapes of answer, one question underneath: may you change the network your nodes sit on. Everything else, including the MTU number that will eventually cause you an afternoon of confusion, follows from which answer your cluster was built with.