The Volume Arithmetic

The decision to self host the trace store took one meeting and was framed as adding a Helm chart. Nobody multiplied anything. Six weeks later it was a four terabyte problem with a ClickHouse cluster nobody had budgeted for.

---

The Problem at Scale

Self hosting an LLM trace store is a capacity decision dressed as a tooling decision, and the arithmetic takes about ten minutes.

  requests/day  x  payload-carrying spans  x  bytes per span   =  ingest/day
  ingest/day    x  retention days                              =  stored

Four inputs. Three of them you control, and the fourth is your traffic. The reason this gets skipped is that trace stores for ordinary services have never needed the calculation, because a few hundred bytes a span does not accumulate into anything you have to think about.

Here it does, and the previous lesson showed why: the payload is the telemetry.

---

How It Works

The worked case

One million requests a day, a six span chain, and the two instrumentation styles from the previous lesson.

Naive: record the payload wherever it appears.

  6 spans, 4 of them carrying the prompt or completion
  4 x ~10 KB  +  2 x ~0.5 KB   =  ~41 KB per trace

  1,000,000 x 41 KB            =  ~41 GB / day
  x 90 days retention          =  ~3.7 TB

Disciplined: record the payload once, at the boundary where it changes.

  6 spans, 1 carrying the payload, 5 carrying references
  1 x ~10 KB  +  5 x ~0.5 KB   =  ~12.5 KB per trace

  1,000,000 x 12.5 KB          =  ~12.5 GB / day
  x 90 days retention          =  ~1.1 TB

Same traffic. Same questions answerable. Three and a half times the infrastructure. That is the single highest leverage decision in this course and it is made, usually without discussion, in the first week of instrumenting.

You store it twice

The term almost every estimate omits, and it follows from how the system is actually built.

A self hosted Langfuse deployment persists incoming events to blob storage and traces, observations and scores to ClickHouse. Those are two copies with different purposes: the blob copy is the durable record of what arrived, and the ClickHouse copy is what queries run against.

So your ingest figure lands in two places:

  12.5 GB/day  ->  blob storage      raw events, durable
               ->  ClickHouse        queryable, compressed

Plan for both. An estimate that sizes only the query store is short by roughly the raw volume, and blob is the cheaper of the two, which is the good news.

Compression, which you must measure

ClickHouse is a column store and compresses well. How well depends entirely on what you put in it, and prompt text is close to the worst case for a column store, because the value of the compression comes from repetition and every prompt is different.

Repeated metric labels compress by very large factors. Natural language compresses by a factor of a few. Which end of that range you land on decides whether your ClickHouse footprint is a disk you already have or a cluster you do not.

This is a number to measure rather than look up, for the same reason span size was in the previous lesson: it varies by more than enough to change the decision, and it is one query away once you have any data at all.

Which levers actually move it

Ranked by effect, which is not the order people try them in:

  payload duplication      3 to 4x     the instrumentation decision
  retention                linear      90 days against 14 is 6.4x
  sampling                 linear      and see Module 4, it is hard here
  truncation               sublinear    and it costs debuggability
  request volume           not a lever  this is your product working

The first one is free and is usually available. The second is a policy decision that Module 5 splits by signal, because the metadata and the payloads do not need the same retention. The third is genuinely difficult here and has its own module. The fourth trades away the thing you instrumented for.

Nobody should be reducing request volume to control observability cost, and it is worth saying because at the point where the bill is alarming somebody always suggests sampling the product rather than the telemetry.

What the number decides

Once you have it, the self hosted against managed question becomes answerable rather than philosophical.

  ~1 TB at rest, modest growth      self hosting is comfortable
  ~10 TB, growing monthly           a real ClickHouse operation
  hundreds of GB/day ingest         a dedicated team problem

The other input is what your data cannot leave. If prompts contain regulated content, from Module 6, then managed is constrained by where the vendor stores it and the arithmetic is a secondary consideration.

---

Building and Operating It

Measure the four inputs. All of them are available today.

# 1. requests per day, from the application rather than the trace store
# 2. spans per trace, and how many carry a payload
curl -s -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  "$LANGFUSE_HOST/api/public/traces?limit=100" \
  | jq '[.data[] | {spans: (.observations // [] | length)}] as $t
      | {traces: ($t|length), mean_spans: (([$t[].spans]|add) / ($t|length))}'

# 3. bytes per span, split by whether it carries a payload. The two
# populations are orders of magnitude apart, so a single mean is
# misleading and the split is what you project from.
curl -s -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  "$LANGFUSE_HOST/api/public/traces?limit=100" \
  | jq '[.data[].observations[]? | (tojson|length)] as $s
      | {n: ($s|length),
         p50: ($s|sort|.[($s|length)/2|floor]),
         p95: ($s|sort|.[($s|length)*95/100|floor]),
         max: ($s|max)}'

Then project, and keep the projection where the decision gets made.

#!/usr/bin/env python3
# The ten minutes that should precede the self hosting decision.
requests_per_day = 1_000_000
payload_spans    = 1        # after recording once, from lesson one
small_spans      = 5
payload_bytes    = 10_240
small_bytes      = 512
retention_days   = 90

per_trace = payload_spans * payload_bytes + small_spans * small_bytes
ingest    = requests_per_day * per_trace
print(f"per trace     {per_trace/1024:,.1f} KB")
print(f"ingest/day    {ingest/1e9:,.1f} GB")
print(f"blob at rest  {ingest*retention_days/1e12:,.2f} TB   (uncompressed)")
print(f"clickhouse    measure YOUR compression ratio and divide")
print(f"1 year        {ingest*365/1e12:,.2f} TB of ingest")

Measure your own compression rather than assuming one.

-- Run this once you have real data. Text compresses far less than
-- repeated labels, and which factor you get decides the hardware.
SELECT
  table,
  formatReadableSize(sum(data_uncompressed_bytes)) AS raw,
  formatReadableSize(sum(data_compressed_bytes))   AS stored,
  round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 2) AS ratio
FROM system.parts
WHERE active AND database = 'default'
GROUP BY table
ORDER BY sum(data_uncompressed_bytes) DESC;

And watch the growth rate, which is the number that turns a comfortable deployment into an uncomfortable one.

# Ingest bytes per day, week over week. Telemetry volume tracks
# product adoption, so this grows faster than your traffic does
# whenever prompts get longer or chains get deeper.
sum(increase(langfuse_ingest_bytes_total[7d]))
  / sum(increase(langfuse_ingest_bytes_total[7d] offset 7d))

---

Tradeoffs and Decision Framework

Lever	Effect	Cost
Record the payload once	3 to 4x	None. Do this first
Shorter payload retention	Linear	Debugging window, Module 5
Sampling	Linear	You may drop the trace that mattered, Module 4
Truncation	Sublinear	The part of the prompt you needed
Fewer requests	Linear	This is your product. Not a lever

Three questions before committing to self hosting. What is the ingest per day, from measured span sizes rather than an estimate. How many times does the payload appear per trace, because that is a factor of three or four before you tune anything else. And what is your compression ratio, measured, since prompt text is close to the worst case for a column store.

Default: do the multiplication before choosing a platform, record the payload once, size blob and the query store separately because you store it twice, measure compression rather than assuming it, and project from measured span size split by whether a span carries a payload.

---

Failure Modes and Common Mistakes

Treating self hosting as adding a Helm chart. It is a multi terabyte stateful system with four datastores.

Sizing only the query store. Incoming events are persisted to blob as well, so the estimate is short by roughly the raw volume.

Assuming a compression ratio. Prompt text compresses by a factor of a few, not a factor of twenty, and the difference decides the hardware.

Projecting from a single mean span size. Payload spans and metadata spans are orders of magnitude apart and the mean describes neither.

Projecting from request growth alone. Context length grows independently, so the estimate is systematically optimistic.

Reaching for sampling first. Recording the payload once is free, larger, and available immediately.

Suggesting fewer requests. That is the product working, and somebody proposes it every time the bill is discussed.