cgroups, Controlling Resources

A container with no memory limit consumed all the RAM on a node and the OOM killer started killing random processes. The container had no cgroup constraints. What went wrong at the kernel level?

Namespaces control what a process can see. Control groups, cgroups, control what a process can use. This is the second half of the container boundary, and the scenario above is what happens when it is missing. The container had no memory cgroup, so nothing capped its allocations. It consumed real host RAM until the kernel had none left, and the system-wide OOM killer then chose victims by oom_score across the entire node, including processes that had nothing to do with the container. A memory cgroup would have confined the blast radius to the container itself.

v1 vs v2

There are two cgroup implementations, and modern systems use v2.

• cgroups v1 used a separate hierarchy per controller. A process could sit in one cgroup for memory, a different one for cpu, and so on. Powerful but inconsistent, and the source of many subtle accounting bugs.
• cgroups v2 uses a single unified hierarchy: a process is in exactly one cgroup, and controllers are enabled per subtree. It is the default on current systemd distros, and Kubernetes has fully supported it for years. The interface is a tree of directories under /sys/fs/cgroup, each with control files.

The controllers that matter for containers:

How a memory limit is enforced, exactly

Set a limit and the runtime writes it into the container's cgroup:

docker run -d --name capped --memory=512m nginx
CID=$(docker inspect --format '{{.Id}}' capped)

# The limit is a single file in the container's cgroup
cat /sys/fs/cgroup/system.slice/docker-$CID.scope/memory.max
# 536870912          <- 512 * 1024 * 1024, enforced by the kernel

# Live usage accounting for the same cgroup
cat /sys/fs/cgroup/system.slice/docker-$CID.scope/memory.current
# 12873728

When the processes in that cgroup try to exceed memory.max, the kernel first attempts to reclaim memory (drop caches, swap if allowed). If it still cannot satisfy the allocation, the cgroup-scoped OOM killer fires and kills a process inside that cgroup, not a random host process. That is the crucial difference from the scenario: a limit turns a node-wide catastrophe into a contained, single-container event.

The killed process exits with signal SIGKILL, which surfaces as exit code 137 (128 + 9). That number in your logs almost always means a cgroup memory limit was hit.

CPU is a quota, not a share of cores

The cpu.max file holds two numbers, a quota and a period, in microseconds. 50000 100000 means "50ms of CPU time per 100ms window." A container that wants more is throttled: the kernel stops scheduling it until the next window opens.

cat /sys/fs/cgroup/system.slice/docker-$CID.scope/cpu.max
# 50000 100000        <- 0.5 CPU: 50ms of runtime every 100ms

This is why a container can show low average CPU yet suffer latency spikes: it burns its quota early in the window and is parked until the window rolls over. It is a per-window ceiling, not an average, and it is the same mechanism a Kubernetes CPU limit configures.

How a Kubernetes limit maps down

This is the interview question, and it is now just a translation table. Kubernetes does not invent resource control; it configures cgroups through the kubelet and the CRI runtime.

So when you write resources.limits.memory: 512Mi, the kubelet asks the runtime to set memory.max = 536870912 on that container's cgroup. When the container exceeds it, the cgroup OOM killer terminates the offending process, the container shows OOMKilled with exit 137, and the kubelet restarts it per the pod's restart policy. A CPU limit becomes cpu.max, and exceeding it throttles rather than kills. Requests, by contrast, mostly steer scheduling and relative CPU weight; they are not hard ceilings.

Common mistakes

• No memory limit on untrusted or memory-hungry workloads. This is the scenario: one container can OOM the whole node. Always set memory.max (a Kubernetes memory limit) for anything that could spike.
• Reading exit 137 as an application crash. It is a SIGKILL, almost always the cgroup OOM killer. Check memory.max versus memory.peak before blaming the code.
• Treating a CPU limit as an average. It is a per-window quota. A bursty service throttles even at low average utilization; sometimes the right move is a higher limit or none at all.
• Confusing requests with limits. Requests schedule; limits enforce. Setting only requests leaves the container able to consume the whole node.
• Forgetting the pids controller. A fork bomb inside a container is contained by pids.max, not by the memory or CPU limits.

Summary

cgroups are the kernel's resource-limiting and accounting boundary, the counterpart to namespaces: namespaces isolate what a process sees, cgroups cap what it uses. v2 is the modern unified hierarchy exposed under /sys/fs/cgroup. A memory limit becomes memory.max, and exceeding it triggers a cgroup-scoped OOM kill (exit 137), confining the damage to one container instead of the node. A CPU limit becomes cpu.max, a per-window quota that throttles rather than kills. Kubernetes limits are just these cgroup files set through the kubelet. Next we move from what a process can use to what it can do: capabilities.