What Metrics Alone Cannot Tell You
The dashboard is excellent. Latency percentiles, tokens per second, error rate, cost per request, all flat for three weeks. Support has fourteen tickets saying the assistant has got worse, and the dashboard has no opinion about that at all.
The Problem at Scale
Metrics are the right first instrument for almost every system you have operated. For an LLM system they are necessary and structurally unable to answer the questions you will actually be asked, for three reasons that compound.
the failure is not an error a wrong answer is a 200
the population is not homogeneous requests differ 100x in cost
individual events matter aggregates exist to hide those
None of these is a gap in your dashboard. They are properties of the workload, and no additional metric fixes them.
In ordinary systems failure correlates with status codes, so error rate is a good proxy for whether users are suffering. In an LLM system the most important failure mode is a successful response containing the wrong content, which is indistinguishable from success at every layer that metrics observe. Error rate cannot see quality, and quality is what people complain about, which is why a flat dashboard and unhappy users is the normal condition rather than a contradiction.
How It Works
The failure is not an error
A hallucinated citation, an ignored instruction, a refusal to answer something reasonable, a response in the wrong language, a summary that inverts the meaning of the source. Every one of those is:
HTTP 200
normal latency
normal token counts
zero errors
zero exceptions
There is nothing in the shape of the response that distinguishes a good answer from a confidently wrong one. That is not true of a database query or an API call, where wrongness usually announces itself somewhere.
So the entire class of failure that matters most to your users is invisible to the instrument you would normally reach for first.
The population is not homogeneous
A p95 latency over all requests is only meaningful if requests are comparable. In an LLM system they differ enormously:
a 40 token intent classification ~200 ms
a 2,000 token question with retrieval ~3 s
a 12,000 token document summarisation ~40 s
A single p95 across that mixture describes none of them. Worse, it moves for reasons that have nothing to do with performance: shift the traffic mix slightly toward summarisation and your p95 rises with no system change at all, which produces investigations into a regression that never happened.
You have to segment before you aggregate, and the segmentation keys are model, prompt version, task type and tenant. Which is the cardinality problem in Module 5, arriving immediately: the segmentation that makes metrics meaningful is the same thing that makes them expensive.
Individual events matter
An aggregate is a device for making individual events invisible, which is usually what you want. One slow request out of a million is noise.
Here it frequently is not. One catastrophically wrong answer to an important customer is an incident, and it is one event. A metric system is the wrong instrument for a question about one event, and no percentile will ever contain it.
That is the structural argument for traces being mandatory rather than a nice addition: you will be asked about a specific response, by name, and only a trace can answer.
The trap inside quality metrics
The response to all this is usually to add evaluation and put the score on the dashboard. Worth doing, and it introduces a specific statistical trap.
Quality distributions are frequently bimodal. A system does not degrade by every answer getting slightly worse. It degrades by a subset of answers collapsing while the rest stay fine.
90% of answers score 0.95
10% of answers score 0.10
mean 0.865 <- looks healthy
degrade to:
80% of answers score 0.95
20% of answers score 0.10
mean 0.78 <- a small move on the dashboard
The failing population doubled and the headline moved by less than a tenth. A mean is the wrong statistic for a bimodal distribution, and it is the statistic everybody picks.
Track the share below a threshold rather than the mean. That number doubled in the example above, which is what happened.
What metrics are genuinely good for
Being fair to them, because the conclusion is not to skip them:
capacity and cost tokens/sec, spend, utilisation
availability and latency against an SLO, per segment
detecting that something changed, without saying what
the trigger for looking at traces
That last line is the useful framing and it is the next lesson. Metrics are an excellent detector and a poor explainer, and in this domain the gap between those two roles is wider than anywhere else you have worked.
Building and Operating It
Segment before aggregating, or the number describes nobody.
# A single p95 across all requests is a fiction. These are the four
# dimensions that make it mean something, and Module 5 is about
# affording them.
histogram_quantile(0.95,
sum by (le, model, task_type) (rate(llm_request_duration_seconds_bucket[5m])))
Watch the distribution rather than the mean, for anything about quality.
# Share of scored responses below the acceptable threshold. This is
# the number that doubles when a subset collapses, which a mean
# barely registers.
sum(rate(llm_eval_score_bucket{le="0.5"}[1h]))
/ sum(rate(llm_eval_score_count[1h]))
Detect the traffic mix shift that moves your latency without a regression.
# If this moves, your latency percentiles move too, and nothing is
# wrong. Put it next to the latency panel so the two are read together.
sum by (task_type) (rate(llm_requests_total[1h]))
/ ignoring(task_type) group_left sum(rate(llm_requests_total[1h]))
And make the metric lead to a trace, which is the whole point.
# Exemplars attach trace identifiers to metric samples, so a spike on
# a dashboard becomes a click through to a specific request rather
# than the start of a search. Module 8 uses this constantly.
histogram_quantile(0.99,
sum by (le) (rate(llm_request_duration_seconds_bucket[5m])))
# with exemplar storage enabled on the histogram
A team investigated a rising p95 for most of a week. Nothing had been deployed, the models were unchanged, GPU utilisation was flat, and the inference engine metrics were healthy. The cause was a new customer whose workload was document summarisation rather than short questions, so the traffic mix had shifted toward requests that were legitimately ten times longer. Latency per task type had not moved at all. The dashboard had one latency panel for all traffic and no panel showing the mix, so a change in who was using the product presented as a performance regression, and the only way anybody found out was by opening individual traces and noticing they were all from the same tenant.
Tradeoffs and Decision Framework
| Question | Metrics | Traces | Evaluations |
|---|---|---|---|
| Is it up? | Yes | No | No |
| Is it slow? | Yes, if segmented | Yes, and where | No |
| What does it cost? | Yes | Per request | No |
| Why was this answer wrong? | No | Yes | Partly |
| Is quality falling? | Only as a distribution | No | Yes |
| What happened to this one request? | No | Yes | No |
Three questions when the dashboard is flat and users are unhappy. Is the failure mode an error at all, since a wrong answer is a 200 and error rate cannot see it. Is the aggregate mixing incomparable populations, because an unsegmented percentile describes nobody. And has the traffic mix moved, as that changes latency without anything being wrong.
Default: segment every aggregate by model and task type at minimum, track quality as the share below a threshold rather than as a mean, put the traffic mix panel next to the latency panel, and attach exemplars so a metric leads to a trace.
Failure Modes and Common Mistakes
Using error rate as a proxy for user experience. The failure that matters returns 200.
One latency percentile across all traffic. It describes none of the populations and moves when the mix moves.
Averaging quality scores. Distributions are bimodal, so a subset collapsing barely moves a mean.
Adding more metrics in response to an unanswerable question. The question was about one event, and no aggregate contains one event.
Investigating a latency regression before checking the traffic mix. A new customer with longer requests looks identical to a regression.
Metrics with no path to a trace. The detection works and the explanation still starts from nothing.
Latency p95 has risen steadily for a week. Nothing was deployed, models are unchanged, GPU utilisation is flat and inference engine metrics are healthy. What should you check before investigating performance?
Your dashboards are flat and users say the product has got worse. How do you approach that?