Advanced|16.5 hours|32 lessons

Production LLM Inference on Kubernetes

Deep production knowledge for engineers running LLM inference on self-managed Kubernetes. vLLM optimization, gateway architecture, observability, debugging, and cost modeling, all from real H100 production deployments. Lifetime updates included.

Text-based, no videos

8 modules, 32 lessons

Lifetime access

What you'll learn

The gateway vs engine two-layer architecture that every production inference stack needs

vLLM configuration: KV cache math, prefill vs decode, quantization trade-offs

Gateway architecture on Kubernetes: streaming, routing, multi-tenancy, failover

Metrics that actually matter, separating gateway and engine signals

Tail latency debugging, where the 5% of requests that are 10x slow actually come from

Throughput degradation, why your p99 drifts up over days and how to fix it

The production debugging playbook for LLM inference incidents

True cost per token, the math that turns rough guesses into real budgets

Concurrent traffic economics, why doubling QPS doesn't double cost

A cost optimization playbook that actually ships savings, not slideware

Continuous batching and the admission decisions that trade throughput against per-request latency

Operating SGLang and Triton in production, and what consolidating on one engine costs

Curriculum

8 modules · 32 lessons

The Production Inference Stack

The two-layer architecture that every production LLM stack converges on, the lifecycle of a single request, and the metrics that actually predict production health.

3 lessons

Gateway vs Engine: The Two-Layer Architecture30 minFREE The Inference Request Lifecycle30 minFREE Metrics That Actually Matter30 minFREE

Single-GPU Optimization with vLLM

vLLM configuration, prefill vs decode dynamics, KV cache management, quantization depth, and the batching and admission behaviour that decides throughput against latency on one GPU.

8 lessons

vLLM Configuration Deep Dive30 min Prefill vs Decode Stages30 min KV Cache Management30 min Quantization in Production30 min Weight-Only and Weight-and-Activation Quantization35 min Quantization Format Support Across Engines30 min Continuous Batching in Practice35 min Preemption, Queueing, and Backpressure40 min

Serving Models That Do Not Fit One GPU

Sizing, sharding, multi-host replicas, disaggregated prefill and decode, and cache beyond a single engine.

5 lessons

When One GPU Is Not Enough30 min Sharding a Model Across GPUs and Hosts30 min Multi-Host Serving on Kubernetes30 min Disaggregated Prefill and Decode30 min Distributed and Shared KV Cache30 min

Gateway Architecture on Kubernetes

Why you need a gateway, streaming APIs done right, routing and failover patterns, and multi-tenant gateway design on Kubernetes.

4 lessons

Why You Need a Gateway30 min Streaming APIs Done Right30 min Routing and Failover30 min Multi-Tenant Gateway Design30 min

Serving at Fleet Scale

Routing on state rather than health, the decode-side throughput lever most platforms never pull, and the SLOs that make any of it measurable.

3 lessons

Cache-Aware and Load-Aware Routing30 min Speculative Decoding in Production30 min Designing SLOs for Streaming Inference30 min

Operating Engines Beyond vLLM

Running SGLang and Triton in production once the engine choice is already made: the operational differences, the tuning surface, and what consolidation costs.

2 lessons

Running SGLang in Production35 min Running Triton for Multi-Framework Serving30 min

Observability & Debugging

How to actually see what your inference stack is doing: separating gateway and engine signals, debugging tail latency, chasing throughput regressions, and a playbook for the 3am pages.

4 lessons

Separating Gateway vs Engine Metrics30 min Tail Latency Debugging30 min Throughput Degradation Over Time30 min The Production Debugging Playbook30 min

Cost Modeling at Scale

The math that turns 'feels expensive' into a real budget. Cost per token, concurrency economics, and a playbook of optimizations that actually cut the bill.

3 lessons

The True Cost Per Token30 min Concurrent Traffic Economics30 min Cost Optimization Playbook30 min

About the Author

Sharon Sahadevan

AI Infrastructure Engineer

Building production GPU clusters on Kubernetes. H100s, large-scale model serving, and end-to-end ML infrastructure across Azure and AWS.

10+ years designing cloud-native platforms with deep expertise in Kubernetes orchestration, GitOps (Argo CD), Terraform, and MLOps pipelines for LLM deployment.

Author of KubeNatives, a weekly newsletter read by 3,000+ DevOps and ML engineers for production insights on K8s internals, GPU scheduling, and model-serving patterns.

Ready to master this topic?

Start with the free preview lesson and see for yourself.