The DevOpsBeast Blog
Field notes on Kubernetes, GPUs, Linux, and the rest of the production stack, from engineers who run real infrastructure.
On Azure, GPU availability is a three-dimensional problem, and quota is not the same thing as capacity. The constraint that most often forces an architecture redesign, and how to plan around it.
Read postThe device plugin API can only count. That single limitation is why your GPU scheduling runs on node labels and pre-partitioned MIG. Dynamic Resource Allocation went GA in Kubernetes 1.34, and in 1.36 the features that make it usable landed. Here is what changes and what to do about it.
Read postEvery post about GPU incidents starts with 'the dashboards looked fine.' That's the problem. nvidia-smi GPU utilization tells you a kernel ran, not whether the silicon is doing work. The metrics that actually matter, the DCGM + Prometheus stack that exposes them, and the queries and alerts that catch real GPU failures.
Read postMIG is hardware partitioning. Time-slicing is software multiplexing. They are not interchangeable. The production decision walk-through, the H100 profile math, the GPU Operator config, and the migration path most teams hit.
Read postThe model weights are 16GB. The KV cache is 20GB. The A100 has 80GB. nvidia-smi shows 50GB free. The next request OOMs. The CUDA memory allocator's fragmentation story most ML engineers never learn.
Read post