Why GPU Requests Must Equal Limits in Kubernetes

You write what looks like an ordinary resource block:

resources:
  requests:
    nvidia.com/gpu: 2
  limits:
    nvidia.com/gpu: 4

For CPU and memory this is the normal shape, and it is how most teams are taught to write a pod spec. Here it is rejected. The same block with cpu or memory in place of nvidia.com/gpu would be fine.

The reason is one sentence long, and the rest of this post is what follows from it: a GPU is allocated to a container or it is not. There is no fractional share, no throttling, no swap and no overcommit. Every API rule below is that fact expressed in YAML.

The three rules, from the Kubernetes docs#

The upstream documentation is unusually direct about it. GPUs "are only supposed to be specified in the limits section", which it then spells out:

• "You can specify GPU limits without specifying requests, because Kubernetes will use the limit as the request value by default."
• "You can specify GPU in both limits and requests but these two values must be equal."
• "You cannot specify GPU requests without specifying limits."

So the canonical form is a limit on its own:

resources:
  limits:
    nvidia.com/gpu: 1    # request defaults to the same value

Set both and they must match. The API is explicit when they do not:

The Pod "mismatch" is invalid: spec.containers[0].resources.requests:
Invalid value: "2": must be equal to nvidia.com/gpu limit of 4

Why the CPU intuition does not transfer#

For CPU, requests and limits are two different things: the request is what the scheduler reserves and the limit is what the cgroup enforces at runtime. The gap between them is overcommit, and overcommit works because the kernel can timeslice a core between processes.

Nothing in that sentence is available for a GPU. A device plugin advertises a whole number of devices and the kubelet hands one to exactly one container. There is no mechanism that could give a container "two GPUs reserved, four at peak", so the API does not let you ask for it.

GPU memory is the same wall from the other side. There is no swap and no overcommit: a model that does not fit in VRAM does not run slowly, it fails. Picking the fraction of VRAM a server may claim is a measurement loop, not a default.

What a quota on nvidia.com/gpu actually counts#

Here is where it stops being an API curiosity and starts costing money.

A ResourceQuota is the innermost of two limits. The cloud decides whether you may have the hardware at all, which is a separate system from whether the capacity exists. This one decides who inside the cluster may allocate what you already have.

The key format is not optional. The quota docs state that "as overcommit is not allowed for extended resources, it makes no sense to specify both requests and limits for the same extended resource in a quota", and conclude: "for extended resources, only quota items with prefix requests. are allowed."

So this is the only valid shape:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-gpu
  namespace: ml-team
spec:
  hard:
    requests.nvidia.com/gpu: "8"

The wrong key is not rejected. That is the part worth knowing. On Kubernetes v1.37 all three of these are accepted and created without complaint:

  nvidia.com/gpu: "8"            created. status.used never appears.
  limits.nvidia.com/gpu: "8"     created. status.used never appears.
  requests.nvidia.com/gpu: "8"   created. status.used tracked and enforced.

Only the third does anything. The other two report their hard value forever and never populate used, because nothing in the quota controller matches them. They are quotas that exist, appear configured, and enforce nothing.

Nothing in the ordinary output distinguishes them. kubectl get resourcequota renders a dead quota as though it were fine, because a missing used is displayed as zero:

NAME      AGE   REQUEST
broken    6s    nvidia.com/gpu: 0/8
working   6s    requests.nvidia.com/gpu: 0/8

One of those enforces a limit and one is decoration. The tell is a missing used field in the object itself:

kubectl get resourcequota -A -o json | jq -r '
  .items[] | select((.status.used // {}) == {})
  | "\(.metadata.namespace)/\(.metadata.name) tracks nothing"'

When the key is right, exceeding it fails loudly at admission:

Error from server (Forbidden): pods "over" is forbidden: exceeded quota: team-gpu,
requested: requests.nvidia.com/gpu=100, used: requests.nvidia.com/gpu=2,
limited: requests.nvidia.com/gpu=8

Now the part that catches people. That quota counts GPUs allocated to pods in the namespace. It does not count GPUs doing work.

A namespace can sit at 8 of 8 with every one of those GPUs at zero percent utilisation, because somebody left a notebook pod running over the weekend. The quota is doing exactly what it was asked to do. It is an admission control mechanism, not a utilisation control, and it has no opinion about whether the allocation was deserved.

The practical consequence is that a GPU quota tells you about contention for scheduling slots, not about waste. If your teams are hitting quota while the fleet shows low utilisation, both facts are true at once and the quota is not the thing that will fix it.

Check it yourself#

What is allocated, per namespace:

kubectl get resourcequota -A \
  -o custom-columns='NS:.metadata.namespace,USED:.status.used.requests\.nvidia\.com/gpu,HARD:.status.hard.requests\.nvidia\.com/gpu'

What is actually allocated across the cluster, straight from the pod specs:

kubectl get pods -A -o json | jq -r '
  [ .items[]
    | select(.status.phase == "Running")
    | .spec.containers[].resources.limits["nvidia.com/gpu"] // empty
    | tonumber ] | add // 0'

Put that next to what the hardware reports. If allocation is high and DCGM_FI_DEV_GPU_UTIL is low across the fleet, you have an idle allocation problem, and no quota change will touch it.

The trap: MIG renames the resource#

This one is worth knowing before you enable it rather than after. If the partitioning decision is still open, MIG vs time-slicing is the one to read first.

A quota on requests.nvidia.com/gpu constrains pods that request nvidia.com/gpu. When you partition a node with MIG, the device plugin stops advertising that name on it and advertises the profile names instead:

spec:
  hard:
    requests.nvidia.com/mig-1g.10gb: "10"
    requests.nvidia.com/mig-2g.20gb: "4"
    requests.nvidia.com/mig-3g.40gb: "2"
    requests.nvidia.com/gpu: "2"            # full GPUs, still counted separately

Each profile is a separate extended resource with a separate quota key. Your existing requests.nvidia.com/gpu quota does not constrain them, and it does not error either. It keeps reporting a used figure, which quietly falls as workloads migrate onto partitions, while the namespace consumes partitions without limit.

The failure mode is a quota that still looks healthy on a dashboard and is no longer enforcing anything.

The fixes, cheapest first#

1. Write the limit alone. Drop requests from every GPU resource block. It cannot differ from the limit, so specifying it adds a way to be wrong and nothing else.

2. Use whole numbers, and do not reach for fractions. There is no API for half a GPU. Sharing a card is MIG or time-slicing, which are node-level configuration, not a pod-level request.

3. Write quota keys with the requests. prefix, and add a key per MIG profile you enable. Do this in the same change that enables MIG, not after.

4. Alert on quota that stops moving. A requests.nvidia.com/gpu used value that is flat while GPU workloads keep being scheduled is the signature of the MIG trap above.

5. Pair the quota with a utilisation signal. Quota answers "who may allocate". Only DCGM answers "is it being used", and getting that signal out of DCGM and into Prometheus is the prerequisite. Treating the first as a proxy for the second is how fleets end up fully allocated and half idle.

In production#

Write GPUs as a limit only, and treat every distinct resource name the device plugin can advertise as a separate thing to put a quota on. A quota is a statement about allocation, so the moment the resource name changes underneath it, the quota stops meaning what the dashboard says it means.

GPU scheduling, dedicated node pools, taints and tolerations, priority classes, MIG profile selection and the device plugin behaviour underneath all of this are covered in depth in Production GPU Infrastructure on Kubernetes.