How Retrieval Fails
Nothing errored. Every service returned 200. The model answered confidently and the answer was wrong, because the three chunks it was handed did not contain the fact it needed.
The Problem at Scale
Every other tier in your stack fails loudly. A database returns an error, a service times out, a pod crashes and something goes red. Retrieval does not work this way, and that difference is the reason retrieval quality problems live for months.
A vector search always returns results. Ask for the ten nearest vectors and you get ten, because there are always ten nearest vectors. The index has no concept of whether they are relevant. It returns the closest things it has, and if the corpus does not contain the answer, the ten closest irrelevant things come back with the same shape, the same status code and the same latency as ten correct ones.
Then the model is handed those ten chunks and asked to answer. It does what it was built to do: produce a fluent answer grounded in the context provided. If the context is wrong, the answer is fluent, confident and wrong.
The retrieval tier has no error state for its primary failure mode. There is no status code for "the right document was not in the top ten". Every layer reports success, the user gets a confident answer, and the only entity in the entire path capable of noticing is the person reading it. This is why retrieval quality needs deliberate measurement rather than monitoring, and why the metrics lesson that follows this one matters more than it looks.
How It Works
The failure taxonomy
Retrieval failures divide into causes that look identical from outside and need completely different fixes.
The document is not in the corpus. Ingestion missed it, it failed to parse, it was filtered, or it has not been processed yet. The retrieval tier is working perfectly and searching a corpus that does not contain the answer. Nothing about tuning the index will help.
The document is in the corpus but not in the index. Embedded but not committed, or in a shard that is unavailable, or deleted by a bug. Detectable, and only if you check.
The document is in the index but ranks below the cutoff. It is there, the query embedding simply does not land close enough to it. This is the genuine retrieval quality failure and it is the hardest to see, because everything is functioning as designed.
The document ranked well and was filtered out. A permission filter, a metadata filter, or a recall collapse from the filtering itself, which Module 2 covers. Right answer found and discarded.
The document was retrieved and dropped at assembly. It ranked fourth, the token budget fit three, and nobody owns the truncation policy.
The document was retrieved, included, and the model ignored it. Now it is a generation problem, not a retrieval one, and telling these apart quickly is a lesson in Module 7.
Chunking split the answer. The fact spans a boundary, so no single chunk contains it and every chunk is individually a poor match. The corpus contains the answer and the index cannot represent it.
Seven distinct causes. One observable symptom.
Why scale makes each one worse
At a small corpus most of these are rare enough to ignore. At millions of documents each becomes systematic.
Ingestion gaps become invisible. Missing four documents out of four hundred is noticeable. Missing forty thousand out of ten million is a rounding error nobody can see, and it is the same failure.
The cutoff gets harder. Returning the top ten from a hundred thousand candidates is a much easier problem than the top ten from fifty million. More near neighbours compete, and the margin between the correct chunk and a plausible wrong one narrows.
Filters get more selective. More tenants, more metadata, more permission rules, each one narrowing the candidate pool and pushing toward the regime where filtered search degrades sharply.
Freshness lag matters more. A corpus large enough to take hours to rebuild is a corpus where some portion is always stale, and the stale portion is invisible.
Building and Operating It
The single most valuable thing you can build is the ability to answer one question quickly: was the right document retrievable at all?
That question separates the first three failure classes, which need different teams, and it is answerable directly:
# 1. Is the document in the corpus at all?
# Query the source of truth, not the index.
# 2. Is it in the index?
# Fetch by document id, bypassing vector search entirely.
# Every store supports this and almost nobody uses it for debugging.
# 3. Would it be found?
# Embed the user query, search, and look for the document id
# in the top 100 rather than the top 10.Step three is the one that distinguishes a ranking problem from an absence problem, and the answer changes who investigates. Present at rank 47 means the index is fine and the ranking is not. Absent from the top 100 means the query embedding lands nowhere near it, which is a chunking or embedding question. Not in the index at all means ingestion.
Instrument the drop points, since three of the seven failure classes are silent losses in your own pipeline:
# Candidates lost at each stage, which is where filtered and truncated
# results disappear without anything recording it
rag_candidates_after_search
rag_candidates_after_filter
rag_chunks_after_assembly
A large gap between the first two is the filtering problem. A large gap between the last two means the token budget is discarding retrieved context and nobody chose the policy that decides which.
Log the retrieved document identifiers with every request, alongside the response. It costs almost nothing and it converts the most common support ticket in a RAG system, which is a user reporting a wrong answer, from a day of investigation into a lookup. Without it you cannot reconstruct what the model was given, and any explanation of why it answered as it did is a guess.
Tradeoffs and Decision Framework
| Failure | Who owns it | Cheapest signal |
|---|---|---|
| Not in the corpus | Ingestion | Source of truth count against indexed count |
| In corpus, not in index | Ingestion or the store | Fetch by id |
| In index, ranks too low | Retrieval quality | Search top 100 for a known id |
| Filtered out | Filtering and permissions | Candidate count before and after filter |
| Dropped at assembly | Prompt assembly | Chunks retrieved against chunks included |
| Model ignored it | Generation | The context was correct and the answer is not |
| Split by chunking | Chunking configuration | No single chunk scores well for a known answer |
The framework is simply this: do not debug retrieval quality until you know which of these seven you have, because five of them are not ranking problems and will not respond to anything you do to the index.
Default: build the three step check into a debugging tool on day one, and log retrieved document identifiers with every request. Everything else in this course assumes you can tell absence from misranking.
Failure Modes and Common Mistakes
Assuming a returned result set means retrieval worked. It always returns results. Ten nearest always exist.
Debugging ranking when the document was never ingested. Five of the seven causes are not ranking, and tuning the index for them wastes weeks.
No record of what was retrieved. The wrong answer cannot be reconstructed and the investigation becomes speculation.
Monitoring only latency and error rate. Neither moves during the primary failure mode.
Treating a wrong answer as a hallucination by default. Frequently the model was given wrong context and did exactly what it was asked. This misattribution sends the investigation to the wrong team.
Ignoring silent drop points. Filtering and assembly both discard candidates with nothing recording it.
A user reports that a RAG system gave a confidently wrong answer. Latency was normal, every service returned 200, and no errors were logged. What should the first diagnostic step be?
Why is a retrieval failure harder to detect than an inference failure, and what would you instrument to catch it?