Kubernetes Service Types Deep Dive

You create a Service of type LoadBalancer in your Kubernetes cluster. You run kubectl get svc and wait. The EXTERNAL-IP column says <pending>. Five minutes pass. Still pending. Ten minutes. Still pending.

You check the pod — it is Running. The Endpoints are populated. The Service has a ClusterIP. Everything looks correct from inside the cluster. But no external IP appears.

The problem? Your cluster does not have a cloud controller manager configured. There is nothing in the cluster that knows how to talk to your cloud provider and provision a load balancer. The Service is waiting for something that will never come.

To debug this kind of issue, you need to understand what each Service type actually creates under the hood — not just what kubectl get svc shows you.

---

Part 1: ClusterIP — The Foundation of All Services

Every Kubernetes Service starts as a ClusterIP. Even NodePort and LoadBalancer services have a ClusterIP underneath. Understanding ClusterIP means understanding how all Services work.

What ClusterIP Creates

When you create a ClusterIP Service, Kubernetes assigns a virtual IP address from the Service CIDR range (typically 10.96.0.0/12). This IP does not belong to any network interface. No pod, no node, no device has this IP. It exists only in iptables/IPVS rules.

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: ClusterIP          # default
  selector:
    app: api
  ports:
  - port: 80               # Service port (what clients connect to)
    targetPort: 8080        # Pod port (where the app listens)
    protocol: TCP

$ kubectl get svc api-service
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
api-service   ClusterIP   10.96.45.123   <none>        80/TCP    5m

$ kubectl get endpoints api-service
NAME          ENDPOINTS                                      AGE
api-service   10.244.1.5:8080,10.244.2.8:8080,10.244.3.2:8080   5m

The Service maps 10.96.45.123:80 to the three pod IPs on port 8080. But how does the traffic actually get from the ClusterIP to the pod? That depends on which kube-proxy mode your cluster uses.

kube-proxy Mode: iptables (Default)

In iptables mode, kube-proxy writes NAT rules that intercept traffic destined for the ClusterIP and DNAT (destination NAT) it to a randomly chosen pod IP.

# Simplified view of what kube-proxy creates for a 3-pod Service:

# Step 1: Match traffic to the Service ClusterIP
iptables -t nat -A KUBE-SERVICES \
  -d 10.96.45.123/32 -p tcp --dport 80 \
  -j KUBE-SVC-ABCDEF

# Step 2: Probabilistic load balancing
# Rule 1: 33.3% chance → Pod A
iptables -t nat -A KUBE-SVC-ABCDEF \
  -m statistic --mode random --probability 0.33333 \
  -j KUBE-SEP-POD-A

# Rule 2: 50% of remaining (= 33.3% total) → Pod B
iptables -t nat -A KUBE-SVC-ABCDEF \
  -m statistic --mode random --probability 0.50000 \
  -j KUBE-SEP-POD-B

# Rule 3: Everything else (33.3%) → Pod C
iptables -t nat -A KUBE-SVC-ABCDEF \
  -j KUBE-SEP-POD-C

# Step 3: DNAT to the actual pod IP
iptables -t nat -A KUBE-SEP-POD-A \
  -p tcp -j DNAT --to-destination 10.244.1.5:8080

kube-proxy Mode: IPVS

IPVS (IP Virtual Server) is a Linux kernel module designed specifically for load balancing. When kube-proxy runs in IPVS mode, it creates IPVS virtual servers instead of iptables rules.

# Check if your cluster uses IPVS
kubectl logs -n kube-system -l k8s-app=kube-proxy | grep "Using ipvs"

# View IPVS rules
ipvsadm -Ln
# IP Virtual Server version 1.2.1 (size=4096)
# Prot LocalAddress:Port Scheduler Flags
#   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
# TCP  10.96.45.123:80 rr
#   -> 10.244.1.5:8080              Masq    1      12         45
#   -> 10.244.2.8:8080              Masq    1      11         42
#   -> 10.244.3.2:8080              Masq    1      13         48

IPVS provides real load balancing algorithms:

Algorithm	Flag	Description
Round Robin	rr	Sequential rotation (default)
Least Connections	lc	Fewest active connections
Destination Hashing	dh	Hash destination IP — consistent routing
Source Hashing	sh	Hash source IP — session affinity
Shortest Expected Delay	sed	Factors in connection count AND weight
Never Queue	nq	Sends to idle server if available, else SED

# Switch kube-proxy to IPVS mode
kubectl edit configmap kube-proxy -n kube-system
# Change: mode: "ipvs"
# Change: ipvs.scheduler: "lc"   # least connections

# Restart kube-proxy pods to apply
kubectl rollout restart daemonset kube-proxy -n kube-system

kube-proxy Replacement: eBPF (Cilium)

Cilium can replace kube-proxy entirely using eBPF programs attached to the kernel networking stack. This is the most performant option.

# Cilium kube-proxy replacement:
# - No iptables rules for Services
# - No IPVS virtual servers
# - eBPF programs attached to socket and TC hooks
# - Socket-level redirection: traffic is redirected BEFORE
#   it even enters the kernel networking stack

# Check if Cilium is replacing kube-proxy
cilium status | grep KubeProxyReplacement
# KubeProxyReplacement:   True

---

Part 2: NodePort — Exposing Services Outside the Cluster

NodePort builds on top of ClusterIP. It takes a ClusterIP Service and adds one thing: it opens a specific port on every node in the cluster.

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: NodePort
  selector:
    app: api
  ports:
  - port: 80              # ClusterIP port (internal)
    targetPort: 8080       # Pod port
    nodePort: 30080        # Port opened on every node (30000-32767)
    protocol: TCP

$ kubectl get svc api-service
NAME          TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
api-service   NodePort   10.96.45.123   <none>        80:30080/TCP   5m

# Now reachable at ANY node IP on port 30080:
curl http://node-1-ip:30080    # works
curl http://node-2-ip:30080    # works (even if no api pods run on node-2)
curl http://node-3-ip:30080    # works

The traffic flow: Client → NodeIP:30080 → iptables DNAT → PodIP:8080

externalTrafficPolicy: Cluster vs Local

This is one of the most important and least understood Service settings.

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: NodePort
  externalTrafficPolicy: Local   # or Cluster (default)

Cluster (default): Traffic arriving at any node is distributed to ALL pods in the Service, regardless of which node they are on. This means a request might arrive at Node A, get forwarded to a pod on Node C. The downside: an extra network hop, and the source IP is lost (because the node SNATs the packet).

Local: Traffic arriving at a node is only sent to pods running ON THAT NODE. If no pods run on that node, the traffic is dropped (the node returns ICMP unreachable). The upside: no extra hop, and the source IP is preserved. The downside: uneven distribution if pods are not evenly spread across nodes.

# externalTrafficPolicy: Cluster (default)
# Client (1.2.3.4) → Node A:30080
#   → iptables SNATs source to Node A IP
#   → Forwards to Pod on Node C
#   → Pod sees source IP = Node A (NOT the client)

# externalTrafficPolicy: Local
# Client (1.2.3.4) → Node A:30080
#   → iptables forwards to Pod on Node A (local only)
#   → Pod sees source IP = 1.2.3.4 (the real client IP!)

---

Part 3: LoadBalancer — Cloud Integration

LoadBalancer type builds on NodePort. It creates a NodePort Service AND asks the cloud provider to create an external load balancer pointing at the NodePort on all nodes.

apiVersion: v1
kind: Service
metadata:
  name: api-service
  annotations:
    # AWS-specific: use NLB instead of Classic LB
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    # AWS: make it internal (VPC only)
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

$ kubectl get svc api-service
NAME          TYPE           CLUSTER-IP     EXTERNAL-IP                          PORT(S)        AGE
api-service   LoadBalancer   10.96.45.123   a1b2c3-1234567890.elb.amazonaws.com  80:31245/TCP   2m

How It Works Under the Hood

The cloud controller manager (running in the cluster or as a cloud-hosted component) watches for Services of type LoadBalancer. When it sees one:

1. Creates a cloud load balancer (AWS NLB/CLB, GCP Network LB, Azure LB)
2. Configures health checks pointing at the NodePort on all nodes
3. Registers all nodes as targets
4. Updates the Service with the external IP/hostname in the status.loadBalancer.ingress field

Cloud Provider Specifics

# AWS: Creates an NLB by default (used to be CLB)
# Annotations control NLB behavior:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."

# GCP: Creates a Network Load Balancer (L4)
# For L7, use GKE Ingress with BackendConfig instead

# Azure: Creates an Azure Load Balancer
# Annotations control SKU and internal/external
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

---

Part 4: ExternalName and Headless Services

ExternalName — DNS Alias

ExternalName is the simplest Service type. It creates a DNS CNAME record. No proxying, no load balancing, no iptables rules.

apiVersion: v1
kind: Service
metadata:
  name: database
spec:
  type: ExternalName
  externalName: mydb.abc123.us-east-1.rds.amazonaws.com

# Inside any pod:
dig database.default.svc.cluster.local
# Returns: CNAME mydb.abc123.us-east-1.rds.amazonaws.com

# Your app connects to "database:5432" and it resolves to the RDS endpoint.
# If you migrate databases, change the ExternalName — no app changes needed.

Headless Service (clusterIP: None)

A headless Service has no ClusterIP. Instead of returning a virtual IP, DNS returns the IP addresses of all pods directly.

apiVersion: v1
kind: Service
metadata:
  name: database
spec:
  clusterIP: None          # Headless — no virtual IP
  selector:
    app: database
  ports:
  - port: 5432

# Normal Service DNS:
dig api-service.default.svc.cluster.local
# Returns: 10.96.45.123 (ClusterIP — virtual IP)

# Headless Service DNS:
dig database.default.svc.cluster.local
# Returns: 10.244.1.5, 10.244.2.8, 10.244.3.2 (actual pod IPs!)

When to use headless Services:

• StatefulSets — each pod needs a stable DNS name (database-0.database.default.svc.cluster.local)
• Client-side load balancing — the client (or client library) picks which pod to connect to
• Service discovery — the client needs to know all pod IPs (e.g., for gossip protocols, Elasticsearch cluster formation)

---

Part 5: Debugging Service Connectivity

When a Service is not routing traffic, work through this checklist:

# Step 1: Does the Service exist?
kubectl get svc api-service -n production
# Check: TYPE, CLUSTER-IP, PORT(S), EXTERNAL-IP

# Step 2: Do Endpoints exist?
kubectl get endpoints api-service -n production
# If empty: no pods match the selector, or pods are not Ready
# CRITICAL: selector labels must EXACTLY match pod labels

# Step 3: Does DNS resolve?
kubectl run debug --image=nicolaka/netshoot -it --rm -- \
  dig api-service.production.svc.cluster.local
# Should return the ClusterIP

# Step 4: Can you reach the ClusterIP from inside the cluster?
kubectl run debug --image=nicolaka/netshoot -it --rm -- \
  curl -v http://api-service.production.svc.cluster.local:80
# If timeout: check NetworkPolicy, kube-proxy logs, iptables rules

# Step 5: Can you reach the pod directly (bypassing the Service)?
kubectl run debug --image=nicolaka/netshoot -it --rm -- \
  curl -v http://10.244.1.5:8080
# If this works but Step 4 fails: the issue is in kube-proxy/iptables

# Step 6: Check kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy | tail -50

# Step 7: Check iptables rules on the node (if iptables mode)
# SSH to the node, then:
iptables -t nat -L KUBE-SERVICES | grep api-service

---

Key Concepts Summary

• ClusterIP is the foundation — a virtual IP that exists only in iptables/IPVS/eBPF rules, reachable only from inside the cluster
• kube-proxy iptables mode uses random selection, O(n) rule evaluation — fine for small clusters, degrades at scale
• kube-proxy IPVS mode provides real load balancing algorithms and O(1) lookups — switch to it for clusters with more than 1,000 Services
• eBPF (Cilium) replaces kube-proxy entirely with socket-level redirection — most performant option
• NodePort opens a port on every node (30000-32767) — all nodes route traffic, not just nodes with pods
• externalTrafficPolicy: Local preserves client source IPs but requires even pod distribution across nodes
• LoadBalancer creates a cloud load balancer automatically — each Service creates a separate LB, which gets expensive
• ExternalName is a DNS CNAME alias — no proxying, useful for referencing external services with cluster-internal DNS
• Headless Services (clusterIP: None) return pod IPs directly in DNS — required for StatefulSets and client-side load balancing
• Empty Endpoints is the number one cause of "Service not routing" — always check selector labels match pod labels

Common Mistakes

• Assuming you can ping a ClusterIP — it is a virtual IP in iptables rules, not a real network interface, and ping (ICMP) is not translated by DNAT rules
• Using externalTrafficPolicy: Local without pod anti-affinity — leads to severely uneven load distribution
• Creating many LoadBalancer Services instead of one Ingress Controller — wastes cloud load balancers and money
• Forgetting that NodePort range is 30000-32767 — specifying a port outside this range will be rejected
• Mismatched selector labels between Service and pods — the most common cause of empty Endpoints
• Not checking EndpointSlices in large clusters — kubectl get endpoints may appear empty when EndpointSlices actually exist
• Leaving externalTrafficPolicy: Cluster when source IP preservation is needed — all requests appear to come from internal node IPs

---