The Identity Stack: From Bytes to Business Logic

A request arrives at your API. Six layers of identity machinery process it before any application code runs. By the time your handler executes, the request has been authenticated, authorized, audit-logged, rate-limited, traced, and possibly mutated. Every one of those layers can be the source of an identity bug. Knowing which layer does what is the prerequisite for debugging any of them.

Identity in modern systems is not one thing; it is a stack of cooperating layers. Each layer answers a different sub-question and hands off to the next. The stack is the same across most modern infrastructure (with details that vary), and understanding it as a stack is what lets you reason about where a problem lives.

This lesson is the layered model. From the TCP packet up to the business logic, what each layer does for identity, what information it has, and what its failure modes look like.

The problem

Production identity bugs almost never live in one layer. They live at the boundary between two layers, where the assumptions of one do not match the assumptions of another. Examples:

• TLS terminates at the ingress; the identity proven by the client cert is dropped at that point; the upstream service trusts a header instead, which is forgeable. (Layer mismatch: network identity vs application identity.)
• The API gateway authenticates the user; passes the user ID to backend services in a header; the services accept the header without verifying. (Layer mismatch: edge auth vs internal trust.)
• Kubernetes RBAC allows the user to create pods; the pod creates other pods using its inherited service-account token; the second-order pods inherit far more privilege than the user had. (Layer mismatch: human identity vs workload identity.)

To debug any of these you need to know which layer made which decision and why. Not knowing the stack means treating identity as an opaque "did it work or not" check, which gets you exactly nowhere when it does not work.

The motivating scenario at the top is the typical request lifecycle. Six layers process an inbound API call. Each is doing something with identity. Each can be wrong.

How it works

The seven layers of a typical identity stack, from network to application:

Three patterns to internalize:

Identity is established at lower layers and consumed at upper layers. TLS at layer 2 establishes peer identity. Layer 5 reads it and produces a principal. Layer 6 reads the principal and decides yes or no. Layer 7 trusts the decision. If any layer drops or alters the identity (a common bug at layer 4, the propagation step), upper layers cannot recover it.

Each layer has different vocabulary for "who you are." Layer 1: "this IP." Layer 2: "this client cert." Layer 4: "this user ID in this header." Layer 5: "this principal object." Different names for the same logical entity in different layer-specific terms. The translation between vocabularies is where bugs hide.

Failure modes are layer-specific. A misconfigured firewall at layer 1 looks like "request never arrived." A TLS misconfiguration at layer 2 looks like "TLS handshake failed." An OAuth bug at layer 5 looks like "401 Unauthorized." An RBAC bug at layer 6 looks like "403 Forbidden." Knowing the layer from the symptom is the first diagnostic step.

The seven-layer model is conceptual; in real systems some layers collapse together (a service mesh might handle layers 2-5 with one configuration). But the logical separation is always present, and it is the lens through which to read complex identity flows.

In practice

Three real-world layer interactions that bite production.

Interaction 1: The "trust the header" antipattern. An ingress controller authenticates users via OIDC. It then forwards the request to backend services with X-Auth-User: alice@example.com in the headers. The backend service reads the header and trusts it. An attacker who can connect to the backend directly (because of a network misconfiguration, a port-forward, or lateral movement) can set the header to anything. The authentication at layer 5 was never propagated cryptographically; only the result was propagated, in a forgeable way.

The fix: propagate the original token (or a derived signed assertion) all the way through, and have backend services validate it themselves. Or use a service mesh that does mTLS-based identity injection that backends cannot fake.

Interaction 2: The "ingress strips the cert" problem. A client presents an mTLS client cert to your ingress. The ingress terminates TLS and forwards the request to the backend over a new TLS connection (or plaintext). The client cert identity exists at the ingress's layer 2 but is gone by the time the backend processes the request. Backends that need client identity get nothing.

The fix: either pass the client cert through (TLS passthrough at the ingress) or have the ingress encode the verified client identity in a signed header that backends can verify (Envoy-style XFCC headers with a signature, or service mesh forwarded identity).

Interaction 3: The "authenticated equals authorized" assumption. A new endpoint is added; the developer assumes the framework's auth middleware handles authorization too. The middleware only authenticates (verifies the JWT). The endpoint has no per-action authorization check. Any authenticated user can hit it, including for actions they should not be able to perform. The Atlassian Confluence CVE-2022-26134 had this shape: layer 5 was fine; layer 6 was missing for a specific endpoint.

The fix: separate auth and authz in your code reviews. Every endpoint, in addition to being behind the auth middleware, must have an explicit authorization check. Codify with tooling: linters that flag handlers without an authz call.

Configuration examples

A request flow through an Istio service mesh, layer by layer:

Client request → cluster ingress (Istio gateway):
  Layer 2: ingress terminates TLS from client.
  Layer 3: gateway applies edge rules (rate limit, WAF).
  Layer 5: optional JWT validation at the gateway.

Gateway → upstream service (in mesh):
  Layer 2: mesh sidecar establishes mTLS to downstream sidecar.
  Layer 4: sidecar adds X-Forwarded-Client-Cert header with original
           verified identity, plus tracing context.
  Layer 5: downstream sidecar validates the AuthorizationPolicy that
           gates this service.

Downstream sidecar → application container (same pod):
  Layer 4: application reads its own request from sidecar over loopback.
           Headers preserved; identity available as the verified
           workload principal.
  Layer 6: application code reads the principal, applies its own
           per-resource authorization.
  Layer 7: business logic runs.

Five-plus layers cooperating per request. If any one is misconfigured, the request fails at that layer with a layer-specific error.

A debugging command per layer:

# Layer 1: network reachability
nc -zv backend-service.namespace.svc.cluster.local 8080

# Layer 2: TLS handshake details
openssl s_client -connect host:443 -servername host -showcerts < /dev/null 2>&1

# Layer 3: did the request reach the ingress
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller \
  | grep $REQUEST_ID

# Layer 4: header propagation
curl -H "X-Forwarded-User: foo" -v http://backend/admin
# Inspect what headers the backend actually receives

# Layer 5: token decoded and validated
echo $TOKEN | cut -d. -f2 | base64 -d | jq

# Layer 6: authorization decision in K8s
kubectl auth can-i create pods --as=$USER --namespace prod

# Layer 7: app-level audit
grep $REQUEST_ID /var/log/app/access.log

When debugging an identity issue, work top to bottom or bottom to top through this list. The first failed layer is the bug.

Common mistakes

• Treating identity as a single concern. "Auth is broken" is too vague. Which layer? What did the previous layer pass through? What did the next layer expect?
• Stripping identity at the edge without re-establishing it inside. mTLS terminates at the ingress; backends have no way to know who connected. Either pass through, sign forward, or rebuild identity from cookies.
• Trusting headers without cryptographic verification. Headers are forgeable by anyone who can connect to the backend. Always verify.
• Implementing authn/authz in business logic. The application should be the consumer of an identity decision, not the producer. Push identity to lower layers (middleware, mesh, gateway) so the application code is simple.
• Different layers using different identity vocabularies without mapping. Layer 5 says "user ID 12345." Layer 6 expects "email." Layer 7 expects "tenant_id." Without an explicit mapping, things misalign.
• Skipping per-layer audit. Each layer should log its decisions: which token it accepted, which policy fired, which rule denied. Without per-layer audit, debugging requires guessing which layer made which decision.
• Mixing dev shortcuts with prod assumptions. "In dev we trust X-User header for testing" is a footgun in prod. Either make the trust environment-conditional and explicit, or do not do it at all.