How a Request Actually Reaches Your Pod

"Walk me through what happens when a user hits your application."

Most candidates do well for about thirty seconds. Browser, DNS, TLS, load balancer. Then comes the sentence that ends the good part: "and then Kubernetes routes it to a pod."

The interviewer waits, because that sentence is the entire question. Everything before it is standard web architecture. Everything after it is the part people have never been shown, and "Kubernetes routes it" is covering a chain of specific mechanisms, each of which you can look at and each of which fails distinctly.

This post walks every hop and names the mechanism behind it. None of it is magic. It is a kernel feature, then another kernel feature, then a controller that wrote some rules earlier, then more kernel features. Once you can name each one, connectivity debugging stops being guesswork and becomes a matter of walking the chain until you find where the packet stops.

Before the cluster#

Two hops happen before anything you operate gets involved, and they matter mostly because of what they leave behind.

DNS resolves your hostname to an address. This is ordinary recursive DNS terminating at a public record you published, usually pointing at a cloud load balancer. Kubernetes DNS is an entirely separate system that resolves names inside the cluster and plays no part here.

TLS is negotiated with whatever terminates it, and where that happens is an architectural decision with consequences at every later hop. The client sends the hostname twice: in the DNS query, and again in the SNI field of the TLS ClientHello, in the clear before encryption starts. That second copy is what lets something route on hostname without decrypting.

Three common choices, and they are not equivalent. At the load balancer, everything downstream is plaintext unless you re-encrypt, and your pods never handle certificates. At an ingress controller, the load balancer runs at L4 and cannot see hostnames, so it forwards bytes to a controller that is itself a pod behind a Service, which means every hop in this post happens twice. At the pod, encryption is end to end and nothing in between can route on anything above L4.

The packet now arrives at a cloud load balancer with a decision to make.

The load balancer to the node#

The load balancer does not know what a pod is. It has a target pool and a health check, and it picks a target. What is in that pool is the first branch in the path.

The traditional model targets nodes. Each node opens a NodePort, a high port accepting traffic for that Service. The pool contains node addresses, and any node accepts traffic for any Service whether or not it runs a pod for it. A node without a local pod forwards to one that has it, which costs an extra hop and rewrites the source address on the way.

The direct model targets pod IPs. Where the CNI assigns pods addresses that are routable in the underlying network, the load balancer's pool can contain pod addresses directly. The NodePort hop disappears entirely, and so does the source address rewriting that came with it.

[VERIFY: which cloud provider and CNI combinations currently support load balancer targeting of pod IPs directly, since this varies by provider and by CNI and is worth confirming against current documentation rather than asserted]

The setting that governs the first model is externalTrafficPolicy, and it is one of the highest leverage single fields in Kubernetes networking.

With Cluster, every node is a valid target. A node that receives traffic for a pod it does not host forwards it onward and applies source NAT so the reply returns through it, so the pod sees the node's address rather than the real client.

With Local, only nodes running a pod for that Service accept traffic. The extra hop disappears and the client address survives. The cost is that distribution now depends on how your pods are spread, because the load balancer balances across nodes and each node serves its local pods.

[VERIFY: how the major cloud load balancers consume the health check node port that externalTrafficPolicy Local allocates, since the Kubernetes side is fixed but provider probe behaviour and timing differ, and describe as the common case rather than universal]

Arriving at the node#

The packet lands on the node's primary network interface. From the kernel's point of view nothing unusual has happened: a frame arrived, the driver handed it up, and the packet entered the network stack.

It now traverses netfilter, the kernel's packet filtering framework, which exposes a fixed set of hook points as a packet moves through the stack. The relevant one here is the hook that runs before routing decisions are made, because that is where the destination address gets rewritten.

Here is the thing worth stating plainly, because it is the most common misconception about Kubernetes networking:

That explains a class of confusing incidents. A crashlooping kube-proxy causes no immediate outage. It causes traffic to keep going to endpoints that no longer exist, surfacing minutes later as connections to dead pods.

What kube-proxy actually did, before any packet arrived#

kube-proxy's real job happens continuously in the background, unrelated to any request.

It watches the API server for two kinds of object: Services, which declare a stable virtual address, and EndpointSlices, which list the pod addresses currently backing that Service. Whenever either changes, it translates the current state into a set of kernel rules and installs them.

In iptables mode, the result is a chain structure. Conceptually it works in three levels:

A top level chain matches on the destination address and port of every Service and jumps to a per-Service chain. The per-Service chain selects among the available endpoints. Each per-endpoint chain performs the actual address rewrite.

top level        dst 10.96.0.10:80        -> service chain
service chain    probability 0.33333      -> endpoint chain A
                 probability 0.50000      -> endpoint chain B
                 (fall through)           -> endpoint chain C
endpoint chain A DNAT to 10.244.1.5:8080

[VERIFY: that the specific iptables chain naming used by the kube-proxy version in your clusters matches what you describe here, since the conceptual structure is stable but chain naming has changed across releases]

Two details in that structure matter more than they look.

The selection is probabilistic, not round robin. The kernel evaluates rules in order, each firing with a probability chosen so the cumulative effect is an even split: one third, then half the remaining two thirds, then the rest. That is why the numbers look strange. The consequence is no per-connection state and no fairness guarantee. It is even in expectation and can be visibly uneven over any short window, because nothing is counting requests per backend.

The rewrite is DNAT. Destination Network Address Translation. The packet arrived addressed to the Service's virtual address, and the kernel rewrites the destination to a specific pod address and port. From this point on, the packet is addressed to a real pod on a real node and the Service has ceased to be involved.

At the same moment, conntrack records the translation. The connection tracking subsystem stores a tuple describing this flow along with the fact that the destination was rewritten, so that when the reply comes back it can be reversed correctly. This entry is the reason the return path works at all, and it is the subject of its own section below.

The honest problem is scale. Rule count grows with Services multiplied by endpoints, and two costs grow with it: the kernel evaluating rules to find a match, and kube-proxy rewriting rule sets on change. At a few dozen Services neither is measurable. At several thousand both are real, and the second bites harder, because a slow rule update means traffic going to stale endpoints. That pressure is why IPVS mode exists, and why eBPF dataplanes exist after it.

Reaching the pod#

The packet now carries a pod address as its destination. The kernel makes an ordinary routing decision about it, and the answer depends on where that pod actually is.

If the pod is on another node, the packet has to get there, and how is your CNI's defining choice.

Overlay networking wraps the packet in an outer header, commonly VXLAN or Geneve, addressed node to node, so the underlying network never needs to know pod addresses exist. It works on any network, including ones you do not control, and costs CPU for encapsulation and MTU for the outer header.

Native routing skips the wrapper because the network already knows how to reach pod subnets, through cloud VPC route tables or BGP. Faster, no MTU cost, and it requires the network to cooperate.

On the node that hosts the pod, the route points at one end of a veth pair, and this is the piece worth understanding properly because it is where the container abstraction becomes something concrete.

A veth pair is a virtual ethernet cable with two ends. One end lives inside the pod's network namespace, where it appears as eth0. The other end lives in the host's namespace with a generated name. Anything written into one end emerges from the other. That is the whole mechanism.

# From inside the pod: its own eth0, its own address
kubectl exec -it my-pod -- ip addr show eth0

# On the node: the other end of that same cable
ip link | grep veth

How the host end is attached varies genuinely by CNI, and asserting a single answer here would be wrong. Some attach every veth to a Linux bridge, so pods on a node share an L2 segment. Others install a route to each veth individually, so same-node pod traffic is routed rather than switched. Others attach the host end to an eBPF program. All three are common, and the debugging commands differ accordingly.

The pod's network namespace is the kernel primitive making this possible: an independent copy of the entire network stack, with its own interfaces, routing table, iptables rules, conntrack table, and socket tables. The pod is not simulated. It has a real, separate network stack connected to the host's by a virtual cable. The Container Internals course builds this up from the kernel primitives if the namespace model still feels abstract.

Into the process#

The last hop is the one people forget is a hop at all.

The packet arrives on the pod's eth0, traverses the pod namespace's own network stack, and the kernel delivers it to whichever socket is bound to the destination port. The process calls accept() and reads bytes.

What the process sees is an ordinary TCP connection on an ordinary interface. It cannot know that its destination was rewritten on the node, that it may have been encapsulated across a physical network, or that the client thought it was talking to something else. That opacity is the point, and it is also why application logs are rarely enough to debug a connectivity problem.

If a sidecar proxy is present, it intercepts here, and it is worth naming precisely because it is often described as something new. Sidecar injection installs iptables rules inside the pod's own network namespace that redirect traffic to a proxy on a local port. Same mechanism as kube-proxy, one namespace deeper. Once you know that, inspecting nat rules inside the pod namespace becomes an obvious debugging step rather than an exotic one.

The return path#

Most explanations stop at the process. The reply is where conntrack earns its existence, and where a specific class of production failure lives.

The pod responds, with the pod address as source. But the client never sent anything to the pod address. It sent to a Service address, a NodePort, or a load balancer, and a reply from an address it never contacted is not a valid response to its connection. Its kernel discards it.

The conntrack entry created on the way in is what prevents that. On the way out, the kernel matches the reply against the stored tuple and reverses the translation, rewriting the source back to the address the client originally used. The client receives a reply from exactly the address it sent to, which is the only thing it will accept.

Two consequences follow directly.

Asymmetric routing breaks connections. The conntrack entry lives on the node that saw the inbound packet. If the reply leaves through a different node, that node has no entry for the flow, cannot reverse the translation, and the reply is either dropped or arrives with an address the client rejects. This is why traffic that works from one node and fails from another is so often a routing asymmetry rather than anything Kubernetes owns.

Conntrack table exhaustion is a real outage. The table is finite. Under high connection rates it fills, new connections fail to get an entry and are dropped, and it presents as random failures under load with clean application logs. A kernel limit, tunable, and invisible unless you watch for it.

What eBPF changes#

Everything above describes the iptables dataplane, which is what most clusters still run. An eBPF dataplane replaces a large part of it, and the contrast is the clearest way to see what the earlier sections were actually doing.

With an eBPF dataplane, kube-proxy and its iptables rules can be removed entirely. Service resolution and endpoint selection are performed by eBPF programs attached at the socket layer or close to the network driver, using hash map lookups rather than sequential rule evaluation.

The practical differences follow from that one change:

Lookup becomes constant time. A hash map lookup does not care whether you have fifty Services or five thousand. Rule traversal does.

Updates get cheaper. Changing an endpoint means updating a map entry rather than recomputing and reinstalling a rule set, which removes the slow propagation problem that bites iptables mode at scale.

Some hops disappear rather than getting faster. This is the most useful framing. When translation happens at the socket layer, when the application calls connect(), the packet is addressed to the pod from the first byte. There is no per-packet DNAT because no packet ever had the wrong destination. Several boxes in the diagram above are not optimised, they are absent.

There is a great deal more to this, including how policy enforcement and observability change when the dataplane is programmable. That is the subject of eBPF and Cilium for Platform Engineers rather than this post. The point here is only the contrast: the earlier sections describe work that a different dataplane does not need to do.

Why knowing this matters#

The payoff is not the interview answer, although it is a good interview answer. The payoff is that every hop in this chain is inspectable, so debugging becomes walking the chain until you find where the packet stops.

A few symptoms and the hop they point at:

• The Service name resolves but connections never establish. DNS worked, so the failure is downstream. Check the EndpointSlice first: a Service with no endpoints produces exactly this, and it usually means a label selector mismatch or failing readiness probes rather than a network problem.
• Connections are refused immediately rather than timing out. Something answered. Usually a process bound to 127.0.0.1 inside the container, which is unreachable across the veth because the pod's loopback is not the same loopback.
• The load balancer marks nodes unhealthy while the pods pass their probes. Under externalTrafficPolicy: Local, Kubernetes allocates a health check node port that answers only on nodes with a local endpoint, so a node running no pod for that Service is correctly unhealthy. Pod health is necessary and not sufficient.
• It works from one node and not another. The rules differ between nodes, or routing is asymmetric. Compare the rule sets, and check whether replies are leaving through the node that saw the request.
• Intermittent failures that scale with load. Conntrack table pressure or ephemeral port exhaustion. Both present as random connection failures with clean application logs.
• Traffic reaches some replicas and not others. One endpoint is unhealthy but still in the EndpointSlice, and the probabilistic selection keeps sending it a share. The readiness probe is not doing its job.
• Large requests hang while small ones work. MTU, almost always. Overlay encapsulation added header bytes and something in the path is not honouring path MTU discovery.

Notice how few of those are Kubernetes problems in any deep sense. They are Linux networking problems that Kubernetes arranged for you to have, and the tools are the ordinary ones: ip, iptables, conntrack, ss, tcpdump.

People call Kubernetes networking magic because nobody showed them the chain. It is a controller that wrote some rules, a kernel that follows them, a virtual cable, and a socket. Every one is a thing you can look at, and once you have, the question at the top of this post stops being intimidating and starts being a list.