Metrics for a Retrieval Tier
Your dashboard shows query latency, index size and queries per second, all green. Not one of them would move if retrieval quality halved overnight.
The Problem at Scale
The previous lesson established that retrieval fails without producing an error. This one is about what to measure instead, and the honest starting point is that the metrics most teams have are the metrics the vector store happens to export.
Those are operational metrics: latency, throughput, memory, index size, uptime. They are necessary and they answer exactly one question, which is whether the service is running. They are structurally incapable of answering whether it is working.
Retrieval needs four families of signal, and most stacks have one.
Operational. Is it up and is it fast. Every store gives you this.
Quality. Is it finding the right things. Almost nobody has this, and it is the reason this course has an entire module on measuring it without labels.
Freshness. Is the index current with the source of truth. Usually absent entirely, which means staleness is invisible.
Capacity. How close is the index to the memory ceiling that decides whether it stays on one node. Available and rarely watched until a rebuild fails.
Operational metrics tell you the retrieval tier is running. They cannot tell you it is working, because the primary failure mode returns results with normal latency and a 200. A dashboard built only from what the vector store exports will be entirely green through a complete collapse in retrieval quality. Quality and freshness have to be instrumented deliberately, by you, because nothing in the stack produces them on its own.
How It Works
The four families, and what actually belongs in each
Operational, and the one nuance worth knowing. Latency percentiles per hop from the first lesson, throughput, error rate, memory. The nuance is that retrieval latency is bimodal at scale in a way most services are not: a query whose candidates sit in one region of the graph is fast, and a query that traverses widely is slow, on identical hardware against an identical index. A p50 that looks excellent and a p99 eight times higher is normal rather than alarming, which makes the percentile spread more informative than any single number. Module 8 covers where that tail comes from.
Quality, and the proxies available without labels. True recall needs ground truth you do not have. What you can measure continuously:
The score distribution of returned results. If the similarity score of the top hit is drifting downward across your traffic, the index is finding worse matches than it used to, and that is measurable without knowing what the right answer was.
The gap between the top result and the tenth. A healthy retrieval has a clear leader. A flat distribution means nothing matched well and the model is being handed ten mediocre candidates.
Zero and low result rates after filtering, which catch the silent drop points.
Click or citation feedback where your product produces it, which is the closest thing to a label that arrives for free.
Freshness, which is the family that is simply missing. Ingestion lag, meaning the age of the oldest document not yet indexed. Indexed count against source of truth count, whose divergence is the ingestion gap from the previous lesson. Time since last successful full rebuild.
Capacity, read against the arithmetic. Index memory as a fraction of the node budget, vector count growth rate, and time to rebuild, which is the number that determines your recovery position and is usually only discovered during recovery.
The three that predict incidents
If the dashboard can only hold a few, these are the ones that move before something breaks rather than after.
Ingestion lag. Rising lag means the index is drifting away from reality, and it degrades answer quality with no operational symptom whatsoever.
Index memory against the node ceiling. This is the arithmetic lesson made continuous. It predicts the sharding project, and it predicts the rebuild that will fail for lack of headroom.
Top result score distribution. The only continuously available proxy for quality that needs no labels and no human, and it moves when an embedding pipeline breaks, when a bad batch is ingested, or when traffic shifts to a domain the corpus does not cover.
Building and Operating It
Instrument what the store does not give you, because that is the half that matters.
# Freshness: the age of the oldest document waiting to be indexed.
# Almost no stack has this and it is the earliest honest signal
# that the index no longer describes reality.
max(rag_ingestion_lag_seconds)
# The ingestion gap: what the source of truth holds against what is indexed.
# A widening divergence is documents silently failing to arrive.
rag_source_document_count - rag_indexed_document_count
# Quality proxy: is the best match getting worse over time?
histogram_quantile(0.5, sum by (le) (rate(rag_top_score_bucket[1h])))
# Silent drops, from the previous lesson
rag_candidates_after_search - rag_candidates_after_filter
# Capacity: the arithmetic lesson, watched continuously
rag_index_memory_bytes / rag_node_memory_budget_bytes
Slice everything by tenant or corpus segment. A fleet average blends a well covered domain at high scores with a poorly covered one near the floor and describes neither, which is the same lesson the DNS and prefix cache material teaches in the sibling courses.
Alert on the ones that predict rather than the ones that report:
# Freshness breach: the index is meaningfully behind reality
max(rag_ingestion_lag_seconds) > 3600
# Capacity: act before the rebuild that cannot fit
rag_index_memory_bytes / rag_node_memory_budget_bytes > 0.75
# Quality drift: the top score distribution has shifted down
histogram_quantile(0.5, sum by (le) (rate(rag_top_score_bucket[1h])))
< 0.9 * histogram_quantile(0.5, sum by (le) (rate(rag_top_score_bucket[1h] offset 7d)))
Do not alert on absolute similarity scores. They are not comparable across embedding models, across index configurations or across query types, so a threshold that is correct today is meaningless after a re-embedding and misleading between two tenants with different query styles. Alert on the change in the distribution against its own recent history. The absolute number tells you nothing; the movement tells you something happened.
Tradeoffs and Decision Framework
| Family | Cost to instrument | What it catches | Typical state |
|---|---|---|---|
| Operational | Free, the store exports it | The service being down or slow | Present, and over-trusted |
| Freshness | Low, you emit two counters | Silent staleness and ingestion gaps | Almost always absent |
| Quality proxies | Low, one histogram | Degradation with no operational symptom | Almost always absent |
| Quality, measured properly | High, needs an evaluation set | Real recall regression | Module 7, and worth it |
| Capacity | Low, one ratio | The sharding project and the failed rebuild | Available, rarely watched |
Two questions decide where to start. Would anything on your current dashboard move if retrieval quality halved, which for most teams is a clear no and settles the priority. And can you tell how stale the index is right now, which is usually unanswerable and is the cheapest gap to close.
Default: add freshness and the top score distribution first. They are the two cheapest signals that catch failures nothing else in the stack reports, and between them they cover the two most common silent degradations.
Failure Modes and Common Mistakes
A dashboard made only of what the store exports. Entirely operational, entirely green through a quality collapse.
No freshness metric. Staleness is undetectable, so the system serves confidently outdated answers with every signal healthy.
Alerting on absolute score thresholds. Not comparable across models, configurations or query types, and meaningless after any re-embedding.
Fleet averages. A well covered segment and an uncovered one average to a number describing neither.
Watching capacity only during an incident. The memory ratio predicts both the sharding project and the rebuild that will not fit, months ahead.
Treating latency percentile spread as a problem. Retrieval latency is legitimately bimodal at scale. The spread is information rather than a fault.
A retrieval tier dashboard shows query latency, throughput, index size and memory, all healthy and stable. An embedding pipeline bug has been writing zero vectors for one document category for two weeks. Why has nothing alerted, and what would have caught it?
What would you put on a dashboard for a retrieval tier, and which of those metrics tells you quality has degraded?