Kubernetes Security for DevOps Engineers

The API Server: Architecture and Security Surface

Your API server is exposed to the internet on port 6443. An attacker finds it through Shodan. What can they do, and what stops them?

The previous lesson covered the request flow at a logical level. This lesson is the API server itself: what it actually exposes, what is hardened by default, what is not, and the audit checklist that determines whether your API server can survive contact with the internet.

The exercise of imagining your API server as the public-facing service it sometimes accidentally becomes is the right framing for this lesson. Every default that "is fine because it is internal-only" stops being fine the moment that assumption breaks.

Attack path / threat explanation

A discovered API server exposes a series of endpoints, some intentionally public and some accidentally so. The standard attacker reconnaissance:

  1. /healthz and /version: confirm the target is a Kubernetes API server. Often returns the K8s version, which tells the attacker which CVEs apply.
  2. /api and /apis: enumerate the API groups and versions available. This works without authentication on many older clusters.
  3. /openapi/v2 or /openapi/v3: the full OpenAPI schema, which lists every resource and verb the cluster supports.
  4. /api/v1/namespaces: try unauthenticated access. If anonymous auth is on with permissive defaults, this returns the list.
  5. /api/v1/namespaces/kube-system/secrets: the holy grail. If reachable, the attacker has every cluster credential.

The attacker tries each unauthenticated, then with the most common credentials (default ServiceAccount tokens, well-known bootstrap tokens, leaked CI tokens). At any point a valid credential gives them the corresponding identity's full access.

The defenses are layered: network access control (do not let the attacker reach the API at all), authentication hardening (no anonymous, no shared credentials), authorization tightening (no wildcard cluster-admin-equivalent), and audit logging (so you see the attack happening).

KEY CONCEPT

The API server has dozens of endpoints. Most are intentionally exposed (the API itself) but several are easy to forget: /metrics, /healthz, /livez, /readyz, /openapi, /version. Each is a small information leak by itself; together they paint a complete picture of the cluster for an attacker. The audit work is knowing what each one exposes and locking down what you can.

How it works under the hood

The API server's architecture, at a security-relevant level:

API server: components and security surfaces

Secure port 6443 (HTTPS)
Aggregation layer
Webhook callbacks (admission, auth)
OpenAPI + discovery endpoints
Audit pipeline
etcd client (mTLS)

Hover components for details

The endpoints worth memorizing for a security audit:

EndpointWhat it returnsDefault auth
/versionK8s version stringUnauthenticated
/healthz, /livez, /readyzHealth checksUnauthenticated
/metricsPrometheus metrics including request rates, latenciesAuthenticated by default; sometimes opened up
/api, /apisAPI discovery (groups/versions)Unauthenticated by default
/openapi/v2, /openapi/v3Full schemaUnauthenticated by default
/api/v1/namespaces/{ns}/podsPod listingAuthenticated + RBAC
/api/v1/namespaces/{ns}/secretsSecret listingAuthenticated + RBAC
/api/v1/nodes/{node}/proxy/stats/summaryPer-node kubelet metrics through API server proxyAuthenticated + RBAC

The "Authenticated + RBAC" rows are where most attackers get blocked. The "Unauthenticated" rows are where information leaks happen even with perfect RBAC.

A specific concern: the /api/v1/namespaces/{ns}/secrets endpoint returns the secret data in base64 by default. base64 is not encryption. An attacker with read access to secrets has the underlying credentials in seconds.

Defense architecture

A layered hardening approach for the API server:

1. Network access first. Do not put the API server on the public internet. Use a private endpoint (managed K8s providers offer this; for self-managed, put it behind a VPN or bastion). If you must expose it (external automation, multi-cluster federation), use IP allow-lists at the network layer.

2. Authentication hardening. Disable anonymous authentication unless you have a specific reason. No shared cluster-admin credentials; use OIDC for humans. Bound short-lived tokens for ServiceAccounts.

3. Authorization minimization. Default RBAC should be restrictive. No wildcard verbs on wildcard resources for any role except cluster-admin (which itself should have minimal members). Audit RBAC quarterly.

4. Admission control as the policy layer. Pod Security Admission enabled cluster-wide. Custom policies (OPA Gatekeeper, Kyverno) for org-specific rules. Defense in depth: policies enforced even if RBAC misses something.

5. Audit logging always on. Audit policy at least at Metadata level (records who did what, when, against what resource). For high-stakes clusters, RequestResponse for sensitive operations. Logs shipped immutably to a SIEM.

6. Continuous CIS benchmarking. Run kube-bench periodically; it checks the API server against the CIS Kubernetes Benchmark and surfaces specific misconfigurations.

Configuration examples

A hardened API server flag set (for self-managed clusters):

kube-apiserver \
  --bind-address=10.0.0.10 \
  --secure-port=6443 \
  --anonymous-auth=false \
  --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota \
  --audit-log-path=/var/log/audit.log \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=10 \
  --audit-log-maxsize=100 \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --tls-min-version=VersionTLS13 \
  --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt \
  --request-timeout=300s \
  ...

The single line that makes the biggest difference: --anonymous-auth=false. Disables the system:anonymous user that older clusters allowed by default.

A minimal audit policy that captures security-relevant events:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Always log secrets, configmaps, RBAC, and admission webhook activity
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
      - group: "rbac.authorization.k8s.io"
        resources: ["*"]
      - group: "admissionregistration.k8s.io"
        resources: ["*"]

  # Log metadata for everything else at the namespace level
  - level: Metadata
    namespaces: ["kube-system", "kube-public"]

  # Skip noisy read-only requests from system components
  - level: None
    users: ["system:kube-controller-manager", "system:kube-scheduler"]
    verbs: ["get", "list", "watch"]

  # Default: log metadata for everything
  - level: Metadata

This policy catches what matters (secret access, RBAC changes, admission webhook config) without drowning in noise.

Common misconfigurations

  • Anonymous auth left at default. Some older distros default to enabled. Always verify with kubectl get --raw '/api/v1' --as=system:anonymous (returns 200 if anonymous works).
  • Audit policy too verbose or too sparse. Verbose policies fill disks; sparse policies miss security events. The example above is a balanced starting point.
  • TLS version below 1.2. TLS 1.0/1.1 deprecated. Configure --tls-min-version=VersionTLS12 (or 1.3 if your client base supports it).
  • --insecure-port set to a non-zero value. Pre-1.20 only; if you are on an older version, set this to 0 immediately.
  • API server load balancer accepting traffic from anywhere. Even if the API server itself is hardened, unrestricted network access widens every other attack surface.
  • No regular CIS benchmark run. Configurations drift over time; periodic re-scoring catches regressions.
  • Audit logs only on local disk. A compromise that touches the node erases the trail. Ship audit logs out of cluster.
INTERVIEW QUESTION

How would you harden a Kubernetes API server that's currently using default settings?