Which Signal Answers Which Question
Four systems are in place: dashboards, a trace store, an evaluation harness, and thumbs up and down on every response. At half past two in the morning, with a customer escalation open, nobody can say which one to open first.
The Problem at Scale
There are four signals available for an LLM system and they answer different questions. Teams tend to fail in one of two directions: building one and expecting it to cover everything, or building all four and having no order of operations when it matters.
metrics is something wrong, and how widespread
traces what actually happened in this request
evaluations is quality different from before
feedback did a human think it was bad
Each is authoritative for its own question and misleading outside it, and the ways they mislead are specific enough to name.
Build traces before metrics. That is the opposite of the order you would use for any other system, and it follows from the previous lessons: the failure mode that matters returns a 200, so metrics cannot see it, while a trace containing the prompt and the completion can. Metrics remain necessary and they are the detector rather than the explanation, and in this domain you will be asked to explain a specific response far sooner than you will be asked about an aggregate.
How It Works
What each is authoritative for
Metrics. Availability, latency, throughput, cost, and the presence of a change. Excellent at telling you that something moved and how many requests are affected.
Traces. What happened in one request: the prompt, the retrieved context, the model call, the completion, the timing of each hop. The only signal that answers why a specific answer was wrong.
Evaluations. Whether output quality differs from a baseline, measured against something. The only signal that quantifies quality rather than describing symptoms.
Feedback. Whether a human considered a response bad. The only ground truth in the entire system, and the most biased.
Where each one misleads
This is the part worth memorising, because every one of these has caused an incorrect conclusion somewhere.
Metrics mislead by aggregation. An unsegmented percentile over incomparable populations, and a mean over a bimodal quality distribution. Both from the previous lesson.
Traces mislead by sampling and by singularity. You are looking at the traces you kept, which is a biased sample unless you designed the sampling with that in mind, which is Module 4. And more subtly: a trace is one instance of a non-deterministic process. A single terrible trace does not establish that the system is broken, because the same input would plausibly have produced a fine answer. One trace proves the failure is possible, not that it is prevalent.
Evaluations mislead through the judge and the dataset. An LLM judge has its own biases, preferring longer answers, particular formats, its own family of models. And the evaluation set drifts away from production traffic the moment traffic changes, so a stable score can mean quality held or can mean you are measuring last quarter's questions.
Feedback misleads by selection, severely. Almost nobody submits feedback. Those who do are overwhelmingly reporting failures, because a good answer produces no impulse to click anything. So a thumbs down rate is not a quality rate and is not even proportional to one, and its absolute level says more about how visible the button is than about the product.
The order to reach for them
At half past two in the morning, in this order:
1. metrics is this real, and how many requests
2. traces what happened in the failing ones
3. evaluations is quality actually different, or is this
one bad instance of a normal distribution
4. feedback is the user impact what I think it is
The anti-pattern is starting at four. One escalated ticket becomes a theory, the theory becomes the investigation, and the investigation confirms it because a search for supporting traces will always find some. Feedback tells you a human was unhappy. It is very poor evidence about scale, and scale is what decides whether this is an incident.
The minimum viable setup
If you have nothing, build in this order. It is not the familiar order and the reasoning is in the callout above.
Traces with payloads, sampled, with short retention. Nothing else answers the question you will be asked first. Short retention keeps the volume arithmetic from Module 1 lesson two manageable while you learn what you actually need.
A small number of segmented metrics, with exemplars pointing into those traces. Segmented by model and task type at minimum. Exemplars are what turn a spike into a click rather than a search.
Feedback capture on responses. Cheap to add, and it is the only ground truth you will ever have. Capture it linked to a trace identifier or it is an anecdote.
Evaluations last. They are the most work, they need a dataset, and the best source of a dataset is production traces you have already collected, which is Module 7. Building evals before you have traces means inventing the test set, and an invented test set drifts from reality immediately.
Teams reverse this constantly, usually building dashboards first because that is what they have always done first, and then discovering that the dashboard cannot answer any of the questions the product generates.
Building and Operating It
Capture feedback against a trace, or it is unusable.
# A thumbs down that is not linked to a trace is an anecdote. Linked,
# it is the start of an investigation and a candidate for a dataset.
langfuse.score(
trace_id=trace_id, # the request this refers to
name="user_feedback",
value=0, # 0 down, 1 up
comment=user_comment, # optional, and the most useful field
)Make metrics lead into traces rather than stopping at a number.
# Exemplars attach a trace id to histogram samples, which is what
# turns "p99 spiked at 02:14" into a specific request you can open.
# Without this the detection works and the explanation starts blank.
storage:
exemplars:
max_exemplars: 100000Weight feedback by how little of it there is.
# Thumbs down as a share of RESPONSES, next to the share of responses
# that got any feedback at all. The second number is usually under a
# percent, which is the context the first number needs.
sum(rate(llm_feedback_total{value="down"}[1d])) / sum(rate(llm_responses_total[1d]))
sum(rate(llm_feedback_total[1d])) / sum(rate(llm_responses_total[1d]))
And check whether your evaluation set still resembles production.
# Distribution of task types in the eval set against live traffic.
# Divergence here means a stable score is measuring the wrong thing,
# which is the quiet failure of every evaluation programme.
curl -s -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
"$LANGFUSE_HOST/api/public/datasets/$DATASET/items?limit=500" \
| jq -r '[.data[].metadata.task_type] | group_by(.) | map({(.[0]): length}) | add'When somebody asks whether a bad answer they were shown means the system is broken, the honest answer needs two signals rather than one. The trace establishes that the failure is possible and shows exactly how it happened. Only an evaluation over a population establishes whether it is prevalent. A single trace is compelling, it is one draw from a non-deterministic process, and treating it as evidence of scale is how a team spends a week on something that happens once in ten thousand requests while a real regression sits unexamined.
Tradeoffs and Decision Framework
| Signal | Authoritative for | Misleads through | Cost |
|---|---|---|---|
| Metrics | Availability, latency, cost, scale of change | Aggregation over incomparable populations | Low |
| Traces | What happened in one request | Sampling bias, and one instance of a random process | High, see Module 1 lesson two |
| Evaluations | Whether quality changed | Judge bias, and dataset drift | High, and it is a second inference workload |
| Feedback | That a human was unhappy | Severe selection bias | Very low |
| Build order | Why |
|---|---|
| 1. Traces with payloads | The only answer to the first question you will be asked |
| 2. Segmented metrics with exemplars | Detection, and a path into the traces |
| 3. Feedback linked to trace ids | The only ground truth, and nearly free |
| 4. Evaluations | Most work, and the dataset comes from the traces |
Three questions before concluding anything. Which signal is authoritative for the question being asked, since three of the four will offer an opinion regardless. Is this one instance or a population, because a trace cannot tell you and an evaluation can. And how much feedback exists at all, as the rate is usually under a percent and the level says more about the button than the product.
Default: traces first with payloads and short retention, then a small segmented metric set with exemplars into them, feedback captured against trace identifiers, and evaluations last built from datasets drawn out of production traces.
Failure Modes and Common Mistakes
Building dashboards first. It is the familiar order and it answers none of the questions this product generates.
Starting an investigation from one ticket. Feedback establishes unhappiness, not scale, and a search for supporting traces always finds some.
Treating one bad trace as a regression. It is one draw from a non-deterministic process.
Reading a thumbs down rate as a quality rate. Almost nobody gives feedback and those who do are reporting failures.
Trusting a stable evaluation score. The set drifts from production traffic, so stability can mean nothing changed or that you are measuring last quarter.
Feedback not linked to a trace. It is an anecdote you cannot investigate.
Metrics with no exemplars. Detection works and every explanation starts from an empty search box.
A customer escalates one badly wrong answer. The trace clearly shows retrieval returned irrelevant chunks and the model answered from them. What can you conclude, and what would you do next?
You are starting from nothing on an LLM platform. Which observability signal would you build first, and why?