Reverse Proxies & Kubernetes Ingress

You deploy an Ingress resource to your Kubernetes cluster. You run kubectl get ingress and see it has been created. You update DNS to point to your load balancer.

Ten minutes later, external traffic starts reaching your pod. What happened in those ten minutes? What layers of infrastructure did the request pass through? And which component actually decided to send the request to your specific pod out of the dozens running in the cluster?

The answer is a reverse proxy — and in Kubernetes, that reverse proxy is your Ingress Controller. Understanding how it works is the difference between confidently debugging routing issues and staring at NGINX config files hoping something makes sense.

---

Part 1: What a Reverse Proxy Does

A reverse proxy sits between clients and your backend servers. Unlike a forward proxy (which acts on behalf of clients), a reverse proxy acts on behalf of your servers. Every request from the outside world hits the reverse proxy first. The reverse proxy decides which backend handles the request.

The core responsibilities of a reverse proxy:

1. Connection termination — The client establishes a TCP + TLS connection with the reverse proxy, not with your backend. Your backend can run plain HTTP internally.

2. Request routing — Based on the Host header, URL path, or other HTTP attributes, the proxy picks the right backend.

3. TLS termination — The proxy decrypts HTTPS, inspects the request, and can forward plain HTTP to backends. This centralizes certificate management.

4. Load balancing — The proxy distributes requests across multiple backend instances.

5. Connection pooling — Instead of each client holding a connection to each backend, the proxy maintains a pool of persistent connections to backends. 10,000 client connections might map to 100 backend connections.

# Without a reverse proxy:
# Client 1 → Backend 1
# Client 2 → Backend 2
# Client 3 → Backend 1  (who decides this? nothing — you need DNS round robin)
# Client 4 → Backend 3
# Problem: no TLS management, no routing, no health checks,
# clients directly connect to backends

# With a reverse proxy:
# Client 1 ─┐
# Client 2 ─┤                    ┌→ Backend 1
# Client 3 ─┼→ Reverse Proxy ────┼→ Backend 2
# Client 4 ─┤    (NGINX/Envoy)   └→ Backend 3
# Client 5 ─┘
# Proxy handles: TLS, routing, health checks, load balancing,
# compression, rate limiting, caching

---

Part 2: The Big Three — NGINX, Envoy, Traefik

In the Kubernetes ecosystem, three reverse proxies dominate. Each has a different philosophy.

NGINX Ingress Controller

NGINX is the most deployed Ingress Controller in Kubernetes. It runs NGINX inside pods and reloads its configuration whenever Ingress resources change.

How it works:

1. The NGINX Ingress Controller watches the Kubernetes API for Ingress resources
2. When an Ingress is created/updated/deleted, the controller generates an nginx.conf
3. The controller writes the new config and sends NGINX a reload signal (nginx -s reload)
4. NGINX gracefully picks up the new config (existing connections finish on old config, new connections use new config)

# The generated nginx.conf inside the Ingress Controller pod looks like:
# (simplified — the real one is thousands of lines)

upstream api-backend {
    server 10.244.1.5:8080;    # api-pod-1
    server 10.244.2.8:8080;    # api-pod-2
    server 10.244.3.2:8080;    # api-pod-3
}

server {
    listen 443 ssl;
    server_name api.example.com;

    ssl_certificate     /etc/tls/tls.crt;
    ssl_certificate_key /etc/tls/tls.key;

    location /api/ {
        proxy_pass http://api-backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Configuration: global via ConfigMap, per-Ingress via annotations.

# Global NGINX settings (apply to all Ingress resources)
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  proxy-body-size: "50m"
  proxy-read-timeout: "120"
  use-gzip: "true"
  worker-processes: "auto"

---
# Per-Ingress settings via annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

Envoy Proxy

Envoy is a high-performance proxy created by Lyft. While NGINX uses static config files that require reloads, Envoy uses a dynamic configuration API called xDS (x Discovery Service). This means Envoy can update its routing without reloading — zero downtime, zero dropped connections.

# NGINX workflow when backends change:
# 1. Controller detects Endpoints change
# 2. Generates new nginx.conf
# 3. Writes file to disk
# 4. Sends NGINX reload signal
# 5. NGINX forks new worker process with new config
# 6. Old workers drain existing connections
# Time: 1-5 seconds, brief spike in latency

# Envoy workflow when backends change:
# 1. Control plane detects Endpoints change
# 2. Pushes xDS update to Envoy via gRPC stream
# 3. Envoy applies the change in-memory
# 4. Zero reload, zero new processes
# Time: milliseconds, no impact on latency

Where you see Envoy in Kubernetes:

• Istio — Envoy as sidecar proxy in every pod (service mesh)
• Contour — Envoy-based Ingress Controller by VMware
• Emissary-Ingress — Envoy-based API Gateway (formerly Ambassador)
• Gateway API implementations — many use Envoy under the hood

Traefik

Traefik takes a different approach: it auto-discovers services from orchestrators (Kubernetes, Docker, Consul) and requires minimal configuration.

# Traefik IngressRoute (Traefik CRD — not standard Ingress)
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: api-route
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`api.example.com`) && PathPrefix(`/api`)
    kind: Rule
    services:
    - name: api-service
      port: 80
    middlewares:
    - name: rate-limit
  tls:
    certResolver: letsencrypt  # Automatic Let's Encrypt

Key Traefik advantages:

• Built-in Let's Encrypt integration (automatic certificate provisioning and renewal)
• Dashboard UI out of the box for debugging routes
• Simpler configuration for common use cases
• Supports both standard Ingress resources and its own CRDs (IngressRoute)

Comparison at a Glance

---

Part 3: The Kubernetes Ingress Resource — Anatomy and Lifecycle

An Ingress resource is a Kubernetes API object that declares HTTP routing rules. By itself, an Ingress does nothing — it requires an Ingress Controller to implement it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx       # Which controller handles this
  tls:
  - hosts:
    - app.example.com
    - api.example.com
    secretName: tls-secret      # K8s Secret with TLS cert + key
  rules:
  - host: app.example.com       # Match this Host header
    http:
      paths:
      - path: /api              # Match this path prefix
        pathType: Prefix
        backend:
          service:
            name: api-svc       # Route to this Service
            port:
              number: 80
      - path: /                 # Default: everything else
        pathType: Prefix
        backend:
          service:
            name: frontend-svc
            port:
              number: 80
  - host: api.example.com       # Different host, different rules
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80

The Full Lifecycle: Ingress Resource to Routed Traffic

Here is what happens from the moment you kubectl apply an Ingress to the moment traffic reaches your pod:

Step 1: Ingress resource created. kubectl apply -f ingress.yaml writes the Ingress object to the Kubernetes API server.

Step 2: Controller detects the change. The Ingress Controller (running as a Deployment in the cluster) watches the API server for Ingress resources. It sees the new one.

Step 3: Controller generates config. For NGINX, it generates a new nginx.conf. For Envoy, it pushes an xDS update. For Traefik, it updates its internal routing table.

Step 4: TLS secret loaded. If the Ingress specifies TLS, the controller reads the referenced Kubernetes Secret containing the certificate and private key.

Step 5: Proxy reloads/updates. NGINX reloads, Envoy applies dynamically, Traefik updates in-memory. The reverse proxy is now ready to handle requests matching the new rules.

Step 6: DNS propagation. The client resolves api.example.com to the load balancer IP (which you set up separately — Ingress does not create DNS records).

Step 7: Request arrives. Client sends HTTPS request to the load balancer IP. The cloud LB (L4) forwards the TCP connection to an Ingress Controller pod.

Step 8: TLS termination. The Ingress Controller terminates TLS, decrypts the request, and inspects the HTTP headers.

Step 9: Route matching. The controller matches the Host header and URL path to an Ingress rule, finds the target Service.

Step 10: Backend selection. The controller resolves the Service to pod IPs (either via Endpoints or by skipping the ClusterIP and going directly to pod IPs) and picks a backend.

Step 11: Request forwarded. The Ingress Controller opens a connection to the selected pod and forwards the HTTP request. The pod processes it and returns a response back through the same path.

IngressClass — Multiple Controllers in One Cluster

You can run multiple Ingress Controllers in the same cluster. Common pattern: one for public traffic, one for internal traffic.

# Public Ingress Controller (internet-facing)
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-public
spec:
  controller: k8s.io/ingress-nginx
  parameters:
    kind: ConfigMap
    name: nginx-public-config

---
# Internal Ingress Controller (VPC only)
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-internal
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  controller: k8s.io/ingress-nginx

---
# This Ingress uses the public controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-api
spec:
  ingressClassName: nginx-public   # ← routes through public LB
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80

---

Part 4: Connection Management — Timeouts, Keep-Alive, and Retries

The reverse proxy sits between clients and backends. Every connection setting matters. Get them wrong and you see intermittent 502/504 errors that are impossible to reproduce.

Timeout Configuration

# Critical NGINX Ingress timeouts
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # How long to wait for the backend to accept the connection
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"

    # How long to wait for the backend to send a response
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"

    # How long to wait for the backend to accept sent data
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"

    # Max request body size (affects file uploads)
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"

Keep-Alive and Connection Reuse

# Client-side keep-alive:
# The client maintains a persistent TCP connection to the proxy.
# Multiple HTTP requests reuse the same TCP connection.
# This avoids the overhead of TCP handshake + TLS handshake for each request.

# Backend-side keep-alive (upstream keep-alive):
# The proxy maintains persistent connections to backend pods.
# Without upstream keep-alive, the proxy opens a NEW TCP connection
# for every single request it forwards to a backend.
# With 1000 req/s, that is 1000 TCP handshakes per second — wasteful.

# NGINX Ingress: enable upstream keep-alive
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  upstream-keepalive-connections: "320"   # connections per upstream
  upstream-keepalive-timeout: "60"        # seconds before idle close
  keep-alive: "75"                        # client-side keep-alive

Circuit Breaking and Retries

Some Ingress Controllers support retry logic and circuit breaking:

# Envoy-based Ingress (Contour HTTPProxy)
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: api-proxy
spec:
  routes:
  - services:
    - name: api-service
      port: 80
    retryPolicy:
      count: 3                # Retry up to 3 times
      perTryTimeout: 2s       # Each try gets 2 seconds
      retriableStatusCodes:
      - 503                   # Only retry on 503 (service unavailable)
    timeoutPolicy:
      response: 30s           # Total response timeout

Rate Limiting at the Proxy Level

# NGINX Ingress rate limiting
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # 100 requests per minute per client IP
    nginx.ingress.kubernetes.io/limit-rps: "2"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "5"
    # Returns 429 Too Many Requests when exceeded
    nginx.ingress.kubernetes.io/limit-req-status-code: "429"

---

Part 5: The Gateway API — The Future of Kubernetes Ingress

The Ingress resource has known limitations: no support for TCP/UDP routing, reliance on annotations for advanced features (which vary by controller), and no separation of concerns between cluster operators and application developers.

The Gateway API is the successor to Ingress. It splits responsibility across three resources:

# 1. GatewayClass — managed by infrastructure provider
# "What kind of load balancer is available?"
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: external-lb
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

---
# 2. Gateway — managed by cluster operator
# "Create a load balancer with these listeners"
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gateway
  namespace: infra
spec:
  gatewayClassName: external-lb
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - name: wildcard-cert

---
# 3. HTTPRoute — managed by application developer
# "Route this traffic to my service"
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: apps
spec:
  parentRefs:
  - name: external-gateway
    namespace: infra
  hostnames:
  - api.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api-service
      port: 80

---

Key Concepts Summary

• A reverse proxy decouples clients from backends — it handles TLS, routing, load balancing, and connection management in one place
• NGINX Ingress Controller is the most deployed, uses file-based config with reload — simple and battle-tested
• Envoy uses dynamic xDS configuration with zero-downtime updates — preferred for high-churn environments and service meshes
• Traefik auto-discovers services and has built-in Let's Encrypt — simplest for smaller deployments
• An Ingress resource does nothing by itself — it requires an Ingress Controller to implement the routing rules
• Always set ingressClassName — without it, controllers may silently ignore your Ingress
• Connection timeouts (proxy-read-timeout, proxy-connect-timeout) are the most common source of intermittent 502/504 errors
• Upstream keep-alive prevents the proxy from opening a new TCP connection for every request — configure it for high-throughput services
• Gateway API is the successor to Ingress — GA since K8s 1.28, with better role separation and native TCP/UDP support

Common Mistakes

• Creating an Ingress without specifying ingressClassName and wondering why traffic does not route
• Setting proxy-read-timeout too low for long-running endpoints — causes 504 errors that only happen on slow requests
• Not configuring upstream keep-alive — the proxy creates and destroys a TCP connection for every request, wasting CPU and adding latency
• Retrying non-idempotent requests (POST) — causes duplicate records in the database
• Running a single Ingress Controller replica — no high availability, a single pod restart drops all traffic
• Forgetting to set proxy-body-size — file uploads fail silently with 413 Request Entity Too Large
• Assuming Ingress creates DNS records — it does not, you must configure DNS separately (or use external-dns)

---