Advanced|26 hours|52 lessons

Designing Large-Scale GPU Clusters on Kubernetes

Training jobs and inference replicas make opposite demands on the same GPUs. This course is about designing the Kubernetes platform underneath both: the pools, the scheduling contract teams deploy through, weight distribution and cold start, capacity and elasticity, the control plane ceilings that bite before the GPUs do, the cluster boundary, and the security of the weights. Not a datacenter course and not a machine learning course.

Text-based, no videos

12 modules, 52 lessons

Lifetime access

What you'll learn

Design a GPU fleet that serves training jobs and inference replicas without either starving the other

Choose the cluster boundary deliberately, argued on blast radius rather than a published node limit

Build the scheduling contract a hundred engineers deploy through: requests, quota, priority, and admission

Architect GPU pools across a model portfolio without stranding capacity

Get model weights onto hundreds of nodes and know what that does to your cold start

Autoscale a fleet whose unit of capacity takes minutes to become useful

Recognise the control plane ceilings that bite long before the GPUs do, and size for them

Run gang scheduling for training jobs and multi-host serving replicas in one cluster

Recover fleet utilization by backfilling serving headroom with preemptible batch work

Protect model weights from exfiltration while keeping them usable to hundreds of engineers

Curriculum

12 modules · 52 lessons

The Shape of a GPU Cluster

Two workload classes on one fleet, what actually limits a cluster, and the scheduling unit each class forces on you.

5 lessons

Training and Inference on One Fleet30 minFREE What Actually Limits a GPU Cluster30 minFREE The Job Is the Unit, Not the Pod30 minFREE The Replica Is the Unit, Not the Pod30 minFREE Reading a Fleet's Resource Profile30 minFREE

Interconnect and Placement

The two things a platform engineer must know about the interconnect: what it implies for placement, and how to tell when placement is the problem.

2 lessons

What the Interconnect Hierarchy Means for Placement30 min When Placement Is the Problem30 min

Scheduling at Fleet Scale

Gang scheduling for both workload classes, topology-aware placement, mixed hardware generations, preemption, and quota across teams.

6 lessons

Gang Scheduling and Why Partial Allocation Is Worthless30 min Topology-Aware Placement30 min Gang Scheduling for Serving Workloads30 min Mixed Fleet Management: H100, H200, and B200 in One Cluster30 min Preemption Without Losing Work30 min Queueing, Quotas, and Fair Share Across Research Teams30 min

The Platform Contract: How Teams Get GPUs

The interface between the platform and everyone deploying onto it. The API you are actually building.

4 lessons

Designing the Request Interface30 min Quota, Priority, and the Tragedy of the Highest Class30 min Admission Policy and Guardrails30 min Tenancy and Isolation Across Teams30 min

Capacity, Cold Start, and Fleet Elasticity

Getting models onto the fleet, absorbing demand that arrives faster than nodes can warm, and keeping the fleet full.

5 lessons

Distributing Model Weights Across the Fleet30 min Cold Start as the Real Constraint30 min Capacity Planning Across Training and Serving30 min Power and Thermal as a Capacity Ceiling30 min Batch and Realtime Tiers on One Fleet30 min

Failure Is the Steady State

Failure budgets, sick node detection, stragglers, silent corruption, elasticity, and allocations that survive loss.

6 lessons

The Failure Budget of a Multi-Week Run30 min Detecting a Sick Node Before It Kills the Job30 min Straggler Detection and Isolation30 min Silent Data Corruption30 min Elastic Training and Reconfiguration30 min Failure Domains and Hot Spares30 min

Checkpointing and Data

The training-specific module: checkpoint economics, the mechanics that keep a checkpoint from stalling the step, and the input pipeline.

3 lessons

Checkpoint Economics: Frequency Against Recovery Cost30 min Distributed and Asynchronous Checkpointing30 min The Training Data Pipeline30 min

Observability for the Fleet

Measuring whether the fleet is producing useful work, across both workload classes.

4 lessons

The Metrics That Matter: Utilization, MFU, and Goodput30 min Instrumenting the Collective Layer30 min Profiling a Training Step30 min Fleet Observability and Silent Degradation30 min

The Control Plane at Fleet Scale

What breaks before the GPUs do, where the cluster boundary should fall, and running more than one.

3 lessons

What Breaks First, and It Is Not the GPUs30 min One Cluster or Many30 min Multi-Cluster and Multi-Region Fleets30 min

Securing Weights and Research Environments

Model weights as the asset, research environments that stay usable, exfiltration detection, and incident response.

5 lessons

Model Weights as the Crown Jewel30 min Isolating Research Environments Without Blocking Research30 min Checkpoint Access Control and Provenance30 min Detecting Exfiltration30 min Auditing Model Access at Scale, and Responding to a Weights Incident30 min

Reference Architecture: Choosing the Stack

Layer by layer through a production GPU platform, naming the real technology options at each stage and how to choose between them.

7 lessons

The Layers of a Production GPU Platform30 min Cluster Foundation and Node Provisioning30 min The GPU Access Layer: Device Plugin, Sharing Modes, and DRA30 min The Scheduling Layer: Volcano, Kueue, YuniKorn, and the Default Scheduler30 min The Workload Orchestration Layer: Training and Serving Frameworks30 min Storage and Data for a GPU Fleet30 min Assembling the Reference Architecture30 min

Capstones

Two full design walkthroughs: the fleet that serves both workload classes, and a weights exfiltration response.

2 lessons

Capstone: Design the Control Plane for a Ten Thousand GPU Cluster45 min Capstone: Design Detection and Containment for a Checkpoint Exfiltration45 min

About the Author

Sharon Sahadevan

AI Infrastructure Engineer

Building production GPU clusters on Kubernetes. H100s, large-scale model serving, and end-to-end ML infrastructure across Azure and AWS.

10+ years designing cloud-native platforms with deep expertise in Kubernetes orchestration, GitOps (Argo CD), Terraform, and MLOps pipelines for LLM deployment.

Author of KubeNatives, a weekly newsletter read by 3,000+ DevOps and ML engineers for production insights on K8s internals, GPU scheduling, and model-serving patterns.

Ready to master GPU infrastructure?

Start with the free preview lesson and see for yourself.