The Payload Is the Telemetry
Tracing was added to the inference service the same way it was added to every other service. It worked beautifully for nine days. On the tenth the trace store stopped accepting writes, and the team discovered they had been ingesting more telemetry than production traffic.
The Problem at Scale
In ordinary observability a span is metadata about work. Route, status, duration, a handful of attributes. A few hundred bytes, and the body of the request is deliberately not in there, because bodies are large and rarely needed.
In LLM observability the span contains the work. The prompt and the completion are the payload, and they are the reason you instrumented anything.
an HTTP span ~200 to 600 bytes of attributes
an LLM span a 2,000 token prompt and a 500 token
completion, roughly 10 KB of text
Twenty to fifty times heavier, per span, and text that does not compress the way repeated metric labels do because every one is different.
The instinct from every other system you have instrumented is to keep the metadata and drop the body. Here the body is the point, and that inversion is what the rest of this course follows from.
In ordinary observability the request body is optional and usually omitted. In LLM observability it is the only thing that answers the question you actually have, which is why a given answer was wrong. A trace recording that a call took 3.2 seconds and produced 412 output tokens tells you nothing about that, so the usual advice to keep the envelope and discard the contents inverts completely, and every cost, retention and sampling decision in this course follows from it.
How It Works
Where the bytes come from
A single LLM call in a real application is not one span. It is a chain, and the payload appears more than once in it.
request received small
retrieval query embedded small
vector search, 20 chunks returned the chunks, several KB
prompt assembled the assembled prompt, several KB
model call prompt AND completion
post processing, guardrail check often the text again
Instrument that naively and the prompt appears three or four times in one trace, because each stage records its input and its output and the output of one stage is the input of the next. A six span trace can therefore carry four copies of a document that only ever existed once.
This is the first place the volume arithmetic in the next lesson goes wrong, and it is a design decision rather than an accident: record the payload at the boundary where it changes, and reference it elsewhere.
Why you cannot simply not store it
Consider the questions you have about an LLM system:
why was this answer wrong? needs the prompt
did retrieval return the right chunks? needs the chunks
did the model ignore the instruction? needs prompt and completion
why did this request cost so much? needs the token counts, and
usually the prompt to see why
is quality degrading? needs samples of both
Every one needs the content. The duration and the status code answer none of them, which is why an LLM platform with metrics and no payloads has monitoring rather than observability, in exactly the sense the free Observability Fundamentals course draws that distinction.
The output is not reproducible
The second difference, and the one that raises the stakes on everything else.
In ordinary debugging a trace is a pointer. You find the slow request, you replay it, you attach a debugger. The trace is evidence that something happened and the real investigation is the reproduction.
An LLM call is not reproducible. Same prompt, same model, same parameters, different output. Temperature, sampling, routing between replicas, a provider changing a model behind a stable name.
So the trace is not a record of a reproducible event. It is the only copy. A dropped trace is not an inconvenience that costs you one data point, it is the permanent loss of the only evidence that a specific bad answer was ever produced. That is why Module 4 treats silent drops as seriously as it does, and why sampling here is harder than sampling HTTP traffic.
What this does to the rest of the design
Four consequences, each a later module:
Volume decides the architecture, because the payload dominates and the multiplication is large. Next lesson.
Sampling is genuinely hard, because the traces worth keeping are the ones with bad outputs and you do not know an output is bad at the moment you decide whether to keep it.
Retention is a privacy question, not only a cost one, because you are storing customer text rather than durations.
Cardinality lives in the payload, so the usual label discipline is necessary and insufficient.
Building and Operating It
Measure your own span size before believing any estimate, including the ones above.
# Average bytes per span, from a sample of real traces. This is the
# only number that predicts your storage, and it varies by more than
# an order of magnitude between applications.
curl -s -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
"$LANGFUSE_HOST/api/public/traces?limit=50" \
| jq '[.data[] | tojson | length] as $sizes
| {traces: ($sizes|length),
mean_bytes: (($sizes|add) / ($sizes|length) | floor),
max_bytes: ($sizes|max)}'Compare it against an ordinary service, which is the comparison that makes the case internally.
# Bytes ingested per span, LLM services against everything else.
# Expect one to two orders of magnitude between them.
sum by (service) (rate(telemetry_bytes_ingested_total[1h]))
/ sum by (service) (rate(telemetry_spans_ingested_total[1h]))
Find the payload you are storing more than once.
# Spans within one trace whose recorded input equals the previous
# span's output. Each match is a copy you are paying for twice.
curl -s -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
"$LANGFUSE_HOST/api/public/traces/$TRACE_ID" \
| jq -r '[.observations[] | {name, in: (.input|tojson|length), out: (.output|tojson|length)}]
| . as $o | range(1; length) as $i
| select($o[$i].in == $o[$i-1].out)
| "DUPLICATED PAYLOAD: \($o[$i-1].name) -> \($o[$i].name) (\($o[$i].in) bytes)"'Record the payload once, at the boundary where it changes, and reference it from the spans that merely pass it along. A six span trace that stores the assembled prompt at assembly, at the model call, and again at the guardrail check is storing the same several kilobytes three times, and the two redundant copies answer no question the first one does not. This single decision routinely takes a third off ingest volume and costs nothing in debuggability, which makes it the cheapest thing in this course.
Tradeoffs and Decision Framework
| Ordinary observability | LLM observability | |
|---|---|---|
| The span is | Metadata about work | The work itself |
| Typical size | Hundreds of bytes | Kilobytes to tens of kilobytes |
| The body is | Optional, usually dropped | The reason you instrumented |
| A trace is | A pointer to a reproducible event | The only copy |
| Dropping one costs | A data point | The only evidence it happened |
| Cardinality lives in | Labels | Labels and the payload |
Three questions before instrumenting an LLM system. How large is a span here, measured rather than assumed, since it varies enormously between applications. How many times does the payload appear in one trace, because naive instrumentation records it at every stage. And what question does each recorded field answer, as anything answering none of them is volume without value.
Default: record payloads at the boundary where they change and reference them elsewhere, measure mean span size against an ordinary service before sizing anything, and treat a dropped trace as lost evidence rather than a lost sample.
Failure Modes and Common Mistakes
Instrumenting an LLM service like an HTTP service. The defaults assume the body is optional, and here it is the point.
Recording the payload at every stage. One document becomes four copies in a single trace.
Keeping metrics and dropping payloads to control cost. You have kept the part that answers none of your questions.
Treating a dropped trace as a lost sample. The output is not reproducible, so it is the only copy.
Sizing from a published figure. Span size varies by more than an order of magnitude between applications and yours is one query away.
Assuming you can sample the interesting traces. You do not know an answer was bad at the moment you decide whether to keep it.
A team instruments their LLM service with the same tracing configuration they use for HTTP services, keeping span metadata and dropping request bodies to control cost. What is the consequence?
How is observability for an LLM system different from observability for an ordinary service?