The Inference Request Lifecycle

Most engineers who debug LLM inference have a model of how it works that is roughly correct at 30,000 feet and wildly inaccurate in the middle. They know there's "tokenization," "the model runs," and "tokens come back." When something goes wrong — tail latency spikes, throughput collapses, a request never returns — that mental model is useless for localizing the cause.

This lesson walks the actual path a single HTTP request takes from client socket to streaming response. If you understand this end-to-end, every debugging session in the rest of the course clicks into place.

---

The 30-second version

A request goes through these stages in order:

We'll walk each stage and pin down where time and failures actually go.

---

Stage 1: HTTP arrives

Client opens a TCP connection (usually HTTPS), sends headers and body. By the time the gateway sees it, the cloud load balancer has:

• Terminated TLS.
• Applied any geo-routing or simple WAF rules.
• Picked a gateway replica to forward to.

What can go wrong here: TLS handshake failure, connection limit on the LB, slow client body upload (you'd see read timeout in the gateway).

Timing: typically under 10ms on warm connections. Cold TLS handshakes can add 100-300ms.

---

Stage 2: Gateway validates and routes

The gateway receives the request. In roughly this order:

1. Decode headers: authenticate (API key, JWT, mTLS).
2. Parse body: JSON schema validation on the OpenAI-style request.
3. Identify tenant: usually from the API key.
4. Count tokens: the gateway either tokenizes or estimates. This is critical — quotas and routing often depend on token count.
5. Check quotas: request rate, token rate, concurrent requests.
6. Pick upstream: which engine pool handles this model? Which healthy replica?
7. Open upstream connection: usually a persistent HTTP/2 connection pool to the engine.
8. Forward request: send the body to the engine.

What can go wrong here:

• Auth fails → 401.
• Tenant over quota → 429.
• Unknown model → 400.
• No healthy upstream → 503.
• Upstream connection refused → 502, probably retry.

Timing: typically 1-5ms if everything is warm. Cold path (no pre-tokenized prompt, quota check against a remote Redis) can add 10-50ms.

---

Stage 3: Engine accepts and tokenizes

The request lands on an engine replica. The engine's HTTP handler:

1. Deserializes the request.
2. Tokenizes: turns the prompt string into a list of token IDs. This is CPU work, done in a tokenizer that's bound to the model.
3. Constructs a request object with the token list, sampling parameters (temperature, top-p, max tokens), and stop conditions.
4. Hands off to the scheduler.

Timing: tokenization is usually ~1-3ms for a few hundred tokens, up to ~30ms for very long prompts. Slower than you'd think — tokenizers are not trivially fast.

What can go wrong:

• Prompt exceeds max_model_len → 400.
• Invalid sampling params → 400.
• Tokenizer is slow on unusual characters → latency spike.

---

Stage 4: Scheduler queues and batches

This is where the real production complexity lives. The engine's scheduler is deciding which requests to run on the next GPU step.

The batch model

At any instant, the engine has a "running batch" of requests currently being decoded. It also has a waiting queue of new requests that haven't started yet.

On each scheduling step, the scheduler:

1. Looks at the running batch — how much KV cache does it need right now?
2. Looks at the waiting queue — can it admit one or more new requests given available KV cache blocks?
3. Decides: prefill or decode? The engine alternates between running prefill on new requests (compute-heavy) and running decode on existing ones (memory-heavy).
4. Updates the batch: adds admitted requests, evicts or swaps requests if pressured for memory.

If your request joins the running batch immediately, it goes to prefill next. If the batch is full or KV cache is exhausted, it waits in the queue.

Queueing — the first latency gotcha

Under load, queue time can dominate TTFT. A request that took 50ms of actual GPU time can easily spend 400ms waiting in the scheduler queue when concurrency is high.

The queue isn't FIFO — it's priority-weighted. Long prompts may be delayed in favor of short prompts, because admitting a long prompt requires a long prefill that stalls the batch. Schedulers are clever, and their cleverness is sometimes what you're debugging.

What can go wrong:

• Queue builds up faster than it drains → TTFT climbs → user sees latency spike.
• Long prompt blocks batch → short-prompt tail latency spikes.
• KV cache pressure triggers swap → throughput drops.

---

Stage 5: Prefill

Prefill is the first forward pass of the model on the input prompt. It:

1. Runs the prompt tokens through the model in a single pass — all tokens attend to all earlier tokens.
2. Produces the KV cache — the key/value tensors for every layer, at every prompt position. This is what makes future decode steps fast.
3. Generates the first output token.

Prefill is compute-bound — it's matrix math over the full prompt length. Roughly, time scales linearly with prompt length (ignoring quadratic attention costs at extreme context lengths).

Timing:

• Short prompt (100 tokens) on 70B model: ~50-100ms
• Long prompt (4000 tokens): ~400-800ms
• Very long prompt (32k tokens): multiple seconds — a meaningful fraction of your SLO

Prefill dominates TTFT — the time to first token, which is what the user perceives as "responsiveness."

---

Stage 6: Decode loop

After prefill, the request enters the decode loop. Each iteration:

1. Takes the latest output token and runs a single forward pass.
2. Consults the KV cache for all previous positions.
3. Produces the next output token.
4. Emits the token downstream (streaming).

Unlike prefill, decode is memory-bound — almost all the work is fetching KV cache tensors from HBM and running a tiny amount of matmul. This is why decode is way less compute-efficient than prefill.

The critical property: continuous batching

Decode steps are where continuous batching pays off. The engine runs many requests together in one decode step — each producing one token per step — because the matmul overhead is tiny compared to the memory traffic. Adding more requests to the batch costs almost no extra time.

This is why throughput scales sublinearly with concurrency: decode is cheap to stack.

Timing per decode step:

• Small batch (1 request), 7B model: 10-20ms per token.
• Large batch (64 requests), 7B model: 15-25ms per step → 64 tokens produced in ~20ms.

When a request leaves the batch

A request finishes the decode loop when any of these happens:

• Stop token — model emits the end-of-sequence token or whatever the stop condition is.
• Max tokens reached — hit the request's max_tokens limit.
• User cancels — client closed the connection, gateway propagates cancel.
• Timeout — engine-enforced cap on generation time.

On finish, the scheduler frees the KV cache blocks. That frees slots for queued requests to enter the batch.

---

Stage 7: Stream to client

As tokens come out of the decode loop, the engine streams them back:

1. Engine emits a token to its HTTP response stream.
2. Gateway forwards the token to the client stream (usually SSE or chunked HTTP).
3. Client receives and displays / processes.

Streaming adds its own complexity:

• Backpressure: if the client is slow, the engine has to wait or buffer. Slow clients can effectively pin KV cache slots, hurting other users.
• Cancellation propagation: when the client disconnects, the gateway should cancel upstream so the engine can free KV cache.
• Timeouts: there are at least three — connect timeout, TTFT timeout, inter-token timeout. Each wants different values.

Backpressure in practice

Decoding produces tokens faster than most networks can deliver them, especially on mobile. If the engine doesn't pause, its write buffer fills up, and eventually it blocks the decode loop for that request.

A well-designed gateway + engine pair uses flow control: the engine's write to the gateway blocks when the gateway's write to the client blocks. The request naturally slows. Its KV cache stays pinned, but the GPU isn't wasting cycles generating tokens no one is reading.

---

A realistic timing budget

For a 70B model on 8× H100, handling a chat request with a 500-token prompt and 200-token response:

Stage	Typical time
HTTP arrival	under 5ms
Gateway processing	2-5ms
Engine HTTP + tokenize	2-5ms
Scheduler queue	0-200ms (depends on load)
Prefill (500 tokens)	80-150ms
Decode (200 tokens at 20ms each)	~4s
Stream overhead	0-10ms per token

TTFT ≈ HTTP + gateway + tokenize + queue + prefill ≈ 90ms to 400ms

Total request time ≈ TTFT + decode ≈ 4-4.5s

Now consider what moves when things go wrong:

• Queue swells under load → TTFT doubles. Model throughput unchanged; you're just waiting.
• KV cache pressure → requests get swapped out → decode rate drops, TPOT goes up.
• Long prompt in batch → other requests' per-step time goes up → every concurrent user feels it.
• Slow client → request stays in batch longer → fewer slots available for others.

All of these are diagnosable only if you know which stage is which. Which is what Module 4 is about.

---

The gateway's view vs the engine's view

A fundamental asymmetry worth internalizing:

The gateway sees: wall-clock request time, HTTP status, streaming start, client disconnect.

The engine sees: queue time, prefill time, decode steps, KV cache events.

Neither side has the full picture alone. The gateway can tell you "request took 4.2s"; only the engine can tell you "1.8s of that was queue time because KV cache was full." Combining them is the whole point of Module 4.

---

Cancellation: the most-missed lifecycle bit

Most teams forget that cancellation is part of the lifecycle. A client disconnects — what happens?

Without proper cancellation propagation:

• Gateway notices disconnect, closes its upstream connection to the engine.
• Engine's HTTP handler notices the connection died, eventually.
• The request stays in the running batch, pinning KV cache until it times out.
• You waste GPU time generating tokens no one is reading.

With proper cancellation propagation:

• Gateway propagates cancel signal via HTTP/2 stream reset (or explicit DELETE).
• Engine handler receives cancellation, asks scheduler to abort the request.
• Scheduler removes the request from the batch, frees KV cache.
• Next scheduling step can admit a new request into the freed slot.

This is a throughput feature, not just a cleanup nicety. Long-running requests with flaky clients can chew through your KV cache capacity if cancellation doesn't work. We'll cover this in the streaming APIs lesson.

---

Quiz

---

What to take away

• A request passes through seven distinct stages: HTTP, gateway validate+route, engine accept+tokenize, scheduler queue, prefill, decode loop, stream response. Each has its own timing and failure modes.
• TTFT = everything before the first decode step. Prefill dominates at high prompt lengths; queue time dominates at high load.
• TPOT = per-step time in the decode loop. Continuous batching makes per-step time stay flat as batch grows — up to KV cache limits.
• Prefill is compute-bound, decode is memory-bound. Long prompts hurt prefill and thus hurt TTFT for everyone in the batch.
• Cancellation is part of the lifecycle. Without propagation, client disconnects leak KV cache and hurt throughput.
• Every request should carry a correlation ID so gateway logs and engine logs reconstruct the full story.

Next lesson: which metrics actually matter on each side of the split — and which ones are lying to you.