The Retrieval Request Path
A user asks a question and gets an answer four seconds later. Two hundred milliseconds of that was the model. The rest happened before the model saw anything, in a path most teams cannot draw.
The Problem at Scale
RAG is usually drawn as three boxes: retrieve, augment, generate. That picture is fine for explaining the idea and useless for operating it, because it collapses six independent components into one arrow.
Here is the path as it actually exists, with an owner for each hop.
Hop 1: the query arrives. Possibly rewritten first, expanded, or decomposed into several queries. Each of those is a model call, and a team that added query rewriting has quietly put a second inference service on the critical path.
Hop 2: the query is embedded. A forward pass through an embedding model on a GPU. This is inference, with the batching and queueing physics of any inference service, and at request time it is a batch of one, which is the least efficient shape that workload has.
Hop 3: the vector search. The embedded query is compared against an index holding every vector in your corpus. This is the hop people mean when they say retrieval, and it is often not the slowest one.
Hop 4: filtering and permission. The candidate set is narrowed to what this user is allowed to see. Whether this happens before or during the search rather than after it decides whether your recall survives, which is a whole lesson later in this course.
Hop 5: reranking. Optional, increasingly common, and a second model on a second GPU with a second queue. It takes the top fifty from hop 3 and reorders them properly.
Hop 6: assembly. The surviving chunks are packed into a prompt under a token budget, which means something is dropped, and what gets dropped is a decision nobody usually owns.
Only then does generation start.
Retrieval is not one hop, it is a pipeline with at least two model inferences in it before the model everyone is thinking about runs at all. The embedding call and the reranker are inference services with all the properties Module 2 of Production LLM Inference on Kubernetes describes, and they are on the synchronous path of every request. Teams that draw RAG as three boxes end up optimising the box they can see.
How It Works
Where the time actually goes
The intuition that vector search dominates is usually wrong, and it costs teams months.
At a corpus of a few million vectors on a warm in-memory index, the search itself is frequently the smallest term in the retrieval half. A graph index traverses a few hundred nodes and returns, and that is fast. What is not fast:
The embedding call at batch one. A GPU forward pass with no batching to amortise it. The model is small compared to a language model, and the per-call overhead is not proportional to model size.
Reranking. A cross-encoder scoring fifty candidate pairs is fifty forward passes worth of work, not one, and it is a larger model than the embedder in many stacks.
The network hops between all of it. Four services, each with a connection, a queue and a p99.
So the retrieval half of a RAG request is frequently dominated by the two inference calls surrounding the vector search rather than by the search. That reorders the optimisation list completely, and it is why a team that spends a quarter tuning index parameters can come away with nothing.
The measurement that settles it is per-hop timing, and it is the first thing to build. Without it, every conversation about retrieval latency is speculation about which of six components is responsible.
Why the path is synchronous and what that implies
Every hop above happens before the first token is generated, which means the entire retrieval pipeline sits inside time to first token. A user waiting for a streaming response sees nothing at all until retrieval has finished.
That has an uncomfortable consequence. Streaming makes generation feel fast by showing progress, and retrieval gets none of that benefit, because there is nothing to show. A slow retrieval tier is felt more sharply than slow generation of the same duration, and it is the part of the system that is usually least instrumented.
It also means retrieval failures and generation failures land in different places. A retrieval timeout happens before any bytes are sent, so it can still be an honest HTTP error. A generation failure happens mid-stream, after a 200, which is the problem Module 7 of the LLM Inference course covers. Retrieval is the last part of the path where you can still fail cleanly.
Building and Operating It
Instrument per hop before anything else. The goal is a single trace where the six components are separable.
# The minimum viable breakdown, as span durations on one request
# rag.query_rewrite (optional, model call)
# rag.embed (model call, batch of one)
# rag.vector_search (the index)
# rag.filter (permission and metadata)
# rag.rerank (optional, model call)
# rag.assemble (token budget packing)
# llm.generate (everything after)If you take one thing from this lesson into your own stack, make it this. Almost every team debugging RAG latency is missing exactly this breakdown, and almost every one of them has a theory about which hop is slow that turns out to be wrong.
Establish the ratio that decides where effort belongs:
# Retrieval as a fraction of time to first token
histogram_quantile(0.95, sum by (le) (rate(rag_retrieval_duration_seconds_bucket[5m])))
/
histogram_quantile(0.95, sum by (le) (rate(llm_time_to_first_token_seconds_bucket[5m])))
A number near one means retrieval is essentially all of your latency before generation starts, and tuning the model will not help.
Check that the embedding call is not being made one at a time when it does not have to be. Query embedding at request time is genuinely a batch of one, and there is nothing to do about that. Ingestion embedding is not, and the two workloads are frequently served by the same deployment sized for neither.
Give the retrieval pipeline its own trace span and its own error budget, separate from generation. They fail for different reasons, they are owned by different parts of the system, and blending them into a single request duration hides the one you can actually fix. Most RAG stacks have excellent visibility into the model and almost none into the four services in front of it.
Tradeoffs and Decision Framework
| Hop | Typical cost | What it buys | When to remove it |
|---|---|---|---|
| Query rewriting | A full model call | Better recall on vague queries | Latency budget is tight and queries are already specific |
| Query embedding | A GPU forward pass at batch one | The search is impossible without it | Never, but it can be cached |
| Vector search | Often the smallest term | The candidates | Never |
| Filtering | Small if done right, catastrophic to recall if done wrong | Correctness and tenancy | Never, but where it runs is a real decision |
| Reranking | A second model, often larger than the embedder | Materially better ordering | Latency budget is tight and top-k ordering is good enough |
| Assembly | Negligible compute, large consequence | Fits the context budget | Never |
Two questions decide most of the shape. What is your latency budget for the whole retrieval half, because that number determines whether you can afford query rewriting and reranking at all. And which hops are model calls, because those are the ones with queues, GPUs, and failure modes that have nothing to do with your index.
Default: instrument all six hops before optimising any of them, and expect the two model calls surrounding the search to dominate rather than the search itself.
Failure Modes and Common Mistakes
Treating RAG as three boxes. Six components with different owners, different failure modes and different scaling behaviour, drawn as one arrow.
Optimising the index because it is the interesting part. Frequently the smallest term in the budget. The per-hop breakdown reorders the list, and without it the interesting component gets the attention.
Forgetting the embedder is inference. It is a GPU workload with a queue on the synchronous path, and it is often deployed as though it were stateless and cheap.
One deployment serving both query and ingestion embedding. Batch of one against batch of thousands, sized for neither, and the backfill starves the request path.
No per-hop timing. Every latency conversation becomes speculation.
Assuming retrieval time is hidden by streaming. It is not. It is entirely inside time to first token, before the user sees anything at all.
A team is investigating RAG latency. p95 for the whole request is four seconds against a three second target. They have spent two months tuning vector index parameters with little improvement. What is the most likely explanation?
Trace a RAG request from the user question to the tokens the model generates. Name each hop and what it costs.