Advanced|18 hours|40 lessons

Large-Scale AI Training Infrastructure

Training infrastructure is not inference infrastructure. One job occupies the whole fleet, every rank waits for the slowest, and a single failure stops everything. This course covers the fabric, the scheduler, the failure machinery, the checkpoint economics, the power budget, and the security of the weights, for engineers responsible for a research organization's training capacity.

Text-based, no videos

9 modules, 40 lessons

Lifetime access

What you'll learn

Read a training cluster as a topology of nodes, rails, and islands, and place a job so the collectives land where the bandwidth is

Explain what NCCL is doing on the wire and choose between InfiniBand, RoCE, and Ethernet with the tradeoffs stated honestly

Design gang scheduling, quota, and preemption that keep a shared research fleet busy without deadlocking it

Compute a run's failure budget from fleet size and node MTBF, then size the recovery machinery to match

Detect a sick node, a straggler, and silent data corruption before any of them quietly ruins a six week run

Derive the optimal checkpoint interval and build sharded asynchronous checkpointing that does not stall the step

Diagnose a training job with MFU, goodput, and a profiler trace instead of a GPU utilization number that means nothing

Manage a mixed H100, H200, and B200 fleet without one generation starving the others

Treat power as a first-class constraint and respond to a job overrunning its budget before facilities calls

Protect model weights from exfiltration while keeping them usable to hundreds of researchers, and respond when it happens

Curriculum

9 modules · 40 lessons

The Shape of a Training Cluster

Why a training fleet breaks the instincts that inference and general Kubernetes work build, and how to read a run's resource profile.

5 lessons

Why Training Infrastructure Is Not Inference Infrastructure30 minFREE Anatomy of a Ten Thousand GPU Cluster30 minFREE The Job Is the Unit, Not the Pod30 minFREE Parallelism Strategies and What They Cost the Network30 minFREE Reading a Training Run's Resource Profile30 minFREE

The Fabric

Collective communication, lossless networks, topology, and the data path. The network is the machine.

5 lessons

Collective Communication: What NCCL Actually Does30 min InfiniBand, RoCE, and the Ethernet Question30 min Topology Awareness: Rails, Islands, and Placement30 min GPUDirect RDMA and the Data Path30 min Diagnosing a Slow Fabric30 min

Scheduling at Fleet Scale

Gang scheduling, topology-aware placement, quota across research teams, mixed hardware generations, and survivable preemption.

5 lessons

Gang Scheduling and Why Partial Allocation Is Worthless30 min Topology-Aware Placement30 min Queueing, Quotas, and Fair Share Across Research Teams30 min Mixed Fleet Management: H100, H200, and B200 in One Cluster30 min Preemption Without Losing Work30 min

Failure Is the Steady State

Failure budgets, sick node detection, stragglers, silent corruption, elasticity, and designing allocations that survive loss.

6 lessons

The Failure Budget of a Multi-Week Run30 min Detecting a Sick Node Before It Kills the Job30 min Straggler Detection and Isolation30 min Silent Data Corruption30 min Elastic Training and Reconfiguration30 min Failure Domains and Hot Spares30 min

Checkpointing and Data

Checkpoint economics and mechanics, the storage underneath them, the input pipeline, and what reproducibility really costs.

5 lessons

Checkpoint Economics: Frequency Against Recovery Cost30 min Distributed and Asynchronous Checkpointing30 min Storage for Checkpoints30 min The Training Data Pipeline30 min Reproducibility and Determinism30 min

Observability for Training

The metrics that describe a run honestly, instrumenting collectives, profiling a step, and catching silent degradation.

4 lessons

The Metrics That Matter: MFU, Step Time, and Goodput30 min Instrumenting the Collective Layer30 min Profiling a Training Step30 min Fleet Observability and Silent Degradation30 min

Power, Thermal, and Capacity

The physical constraint most Kubernetes engineers have never had to model, and the planning it forces.

3 lessons

Power as a First-Class Constraint30 min Detecting and Capping a Runaway Job30 min Capacity Planning for Research Demand30 min

Securing Weights and Research Environments

Model weights as the asset, research environments that stay usable, exfiltration detection, and incident response.

5 lessons

Model Weights as the Crown Jewel30 min Isolating Research Environments Without Blocking Research30 min Checkpoint Access Control and Provenance30 min Detecting Exfiltration30 min Auditing Model Access at Scale, and Responding to a Weights Incident30 min

Capstones

Two full design walkthroughs combining everything: the training control plane, and a weights exfiltration response.

2 lessons

Capstone: Design the Control Plane for a Ten Thousand GPU Training Cluster45 min Capstone: Design Detection and Containment for a Checkpoint Exfiltration45 min

About the Author

Sharon Sahadevan

AI Infrastructure Engineer

Building production GPU clusters on Kubernetes. H100s, large-scale model serving, and end-to-end ML infrastructure across Azure and AWS.

10+ years designing cloud-native platforms with deep expertise in Kubernetes orchestration, GitOps (Argo CD), Terraform, and MLOps pipelines for LLM deployment.

Author of KubeNatives, a weekly newsletter read by 3,000+ DevOps and ML engineers for production insights on K8s internals, GPU scheduling, and model-serving patterns.

Ready to master this topic?

Start with the free preview lesson and see for yourself.