Self-Hosting Langfuse on Kubernetes: ClickHouse Sets Your Ceiling

The first thing people get wrong about self-hosting Langfuse is thinking of it as an application with a database. It is four datastores and two services, they fail in different ways, and only two of them can actually lose your data.

The second thing people get wrong is which one to size carefully. Almost everyone reaches for Postgres, because that is the database they know. Postgres is the one component here whose size has nothing to do with how many traces you ingest.

Here is the architecture, what each piece costs you when it is down, and the two sizing numbers that bite.

What a self-hosted deployment actually contains#

As of Langfuse v4, a production deployment is:

Component	Role
langfuse-web	UI, API, accepts incoming events
langfuse-worker	Processes events asynchronously
Postgres	Projects, users, API keys, prompt and dataset definitions
ClickHouse	Traces, observations, scores
Redis or Valkey	Cache, and the ingest queue
Blob storage	Raw events, multimodal attachments, large exports

Both services are stateless, so both scale horizontally. Everything interesting is in how the four stores differ.

The write path, and which two components lose data#

An event does not go straight into ClickHouse. It goes:

  SDK
   |
   v
  langfuse-web        accepts the event
   |
   +--> blob          the raw event, durably
   +--> Redis         queued for processing
                       |
                       v
                      langfuse-worker
                       |
                       v
                      ClickHouse       queryable

Read that as a failure table rather than a diagram:

  web down           nothing is accepted at all
  blob down          the event cannot be persisted, ingest fails
  Redis down         the queue is gone, queued events are lost
  worker down        events accumulate, nothing lost, visibility lags
  ClickHouse down    nothing queryable, events accumulate, recoverable
  Postgres down      no login, no queries, no project config

Two of those lose data. Four do not.

That distinction matters more than it looks, because it cuts against instinct. ClickHouse is the component holding all your traces, and ClickHouse being down loses nothing: events pile up in the queue and drain when it returns. Redis looks like a cache, and the word people attach to caches is "disposable", but it is on the write path. Losing Redis loses whatever was queued in it.

If you treat Redis as a cache you can restart freely, you will eventually restart it during a ClickHouse incident, which is exactly when it is holding the most data.

ClickHouse sets your ceiling, and it runs out of things that are not disk#

ClickHouse is usually the component the team has least experience with, and it is the one that decides how far you can scale. Disk is the obvious limit and rarely the first one you hit.

Data lands as immutable parts. A background process merges small parts into larger ones. That design is what makes the store fast to write and fast to scan, and it has one operational consequence: merges have a throughput limit. Write frequently in small batches and parts accumulate faster than they merge. Past one threshold inserts are deliberately delayed; past a second they are rejected with Too many parts, often alongside Merges are processing significantly slower than inserts.

Note the shape of that. You get a warning period where writes slow down before they fail, which means a monitored parts count turns a future outage into a Tuesday afternoon.

Do not trust the thresholds you read, including here#

Numbers for parts_to_delay_insert and parts_to_throw_insert circulate widely and disagree with each other. You will find 150 and 300 in older material and 1000 and 3000 in newer material, presented with equal confidence and usually with no version attached.

Check your own:

SELECT name, value, description
FROM system.merge_tree_settings
WHERE name IN ('parts_to_delay_insert', 'parts_to_throw_insert', 'max_parts_in_total');

The mechanism is what is stable. Delay first, then reject. The specific integers are a version detail and a blog post is the wrong place to learn them.

The error says ClickHouse, the fix is upstream#

This is the part worth internalising. Too many parts is raised by ClickHouse, so people tune ClickHouse. ClickHouse's own documentation is blunt that this is the wrong move: raising parts_to_throw_insert or max_parts_in_total "does not address the cause of excessive part creation," and higher limits add filesystem and metadata overhead and hurt query performance while merely delaying the exception.

The cause is almost always insert frequency, not insert volume. Many small writes produce many parts. Few large writes produce few. ClickHouse recommends batching at least 1,000 rows per insert, ideally 10,000 to 100,000, at roughly one insert per second, or using asynchronous inserts with wait_for_async_insert = 1 when client-side batching is impractical so that acknowledgements still mean the write landed.

In a Langfuse deployment that batching is the worker's behaviour, not something you set on the database. Which means the fix for a ClickHouse error lives in the ingest pipeline, one component upstream of where the error appeared.

The same logic rules out the other reflex. Teams who hit this often conclude they need to shard ClickHouse. Sharding is a large step up in operational complexity, it brings distributed queries, rebalancing and a considerably harder backup story, and it does not fix the parts limit at all.

Sizing the four stores from one number#

Everything downstream comes from bytes ingested per day. Take a worked case: one million requests a day at roughly 12.5 KB per trace, so about 12.5 GB/day.

The four stores are driven by genuinely different things:

  ClickHouse disk   ingest x retention x compression
  blob              ingest x raw retention, uncompressed
  Postgres          not driven by ingest at all
  Redis             how long an outage you want to survive

ClickHouse. Compression happens per column, and your columns are not alike. Model names repeat and compress enormously. Timestamps are monotonic and compress well. Prompt and completion text are unique natural language and compress badly, and they are where your volume is. So your footprint is dominated by two or three payload columns, and storage projections should be made per column rather than as one blended ratio.

  12.5 GB/day x 90 days retention   = 1,125 GB raw
  / compression, say 4x             = ~280 GB stored
  + headroom for merges             = ~450 GB provisioned

Treat the 4x as an assumption to measure, not a constant. The merge headroom is the line people drop, and dropping it is how the previous section's failure arrives: merges rewrite parts into larger ones, so they need free space to consolidate into. A store running near full does not merely risk filling up, it stops merging, and a store that stops merging walks straight into the parts limit.

Blob. Raw and uncompressed, on its own retention policy. Thirty days rather than ninety is usually right, because blob exists to make reprocessing possible and a one month reprocessing window is ample:

  12.5 GB/day x 30 days  = ~375 GB

Postgres. Grows with projects, users, prompts and dataset definitions. Tens of gigabytes for a long time. Size it small, back it up properly, stop thinking about it.

Two ClickHouse properties worth knowing before you need them#

The sorting key is the index. ClickHouse does not keep a general purpose index on every column. The table's sorting key decides physical order, and queries aligned with that order read a small range while queries that are not aligned scan. You do not usually pick this schema, since the platform ships it, but you should know what it is, because it tells you which of your queries are affordable. The query that exhausts memory is typically a filter on an unsorted high cardinality field across a long time range, and it is usually a person exploring in the UI rather than a dashboard.

Retention is a TTL, so it is schema. Expiry is enforced by the store, not by a cleanup job you write and forget. That is better in every operational sense, with one consequence: changing a retention policy is a schema change, with the review and rollout that implies, rather than editing a cron entry.

The Redis default buys you about ninety seconds#

This is the number that surprises people.

Redis holds accepted events waiting to be written to ClickHouse. Its capacity is therefore not a performance tuning knob. It is how long an analytical store outage you can absorb without losing traces.

At 1,000 events a second and 12.5 KB each, the queue grows at about 12.5 MB/sec:

  survive 5 minutes    ~3.8 GB
  survive 30 minutes   ~21 GB
  survive 2 hours      ~90 GB

A 2 GB default gives you roughly ninety seconds.

Ninety seconds is less than a ClickHouse pod reschedule. It is far less than a version upgrade, a disk resize, or any incident involving a human being reading a dashboard first. If you took the default and your ClickHouse has ever been down for two minutes, you have already lost traces, and nothing in the system was obliged to tell you.

Pick the outage length you want to ride out, multiply, and provision for that.

Scale the worker on queue depth, not CPU#

The last one, because it follows directly. The two services scale on different signals:

  langfuse-web       ingest request rate + query load
  langfuse-worker    queue depth

CPU is the wrong autoscaling signal for the worker, for the same reason it is wrong for a CI runner fleet: utilisation saturates and then tells you nothing about how far behind you are. A worker fleet at 100 percent CPU with an empty queue is correctly sized. The same fleet at 100 percent CPU with a growing backlog is not. CPU reports an identical number in both cases.

Queue depth distinguishes them, and queue depth is also the metric that tells you how much of that Redis sizing you are currently consuming.

Quick reference#

Question	Answer
What can lose data?	Redis and blob. Not ClickHouse, not the worker.
What sets the ceiling?	ClickHouse, usually via the parts limit before disk.
Too many parts, first move?	Look at insert frequency upstream, not ClickHouse settings.
ClickHouse disk	ingest x retention / compression, plus merge headroom.
Redis size	ingest rate x event size x outage you want to survive.
Postgres size	Small. Not a function of ingest.
Worker autoscaling	Queue depth.
Run it or buy it?	Blob always managed. Postgres usually managed. ClickHouse is the real decision.

The pattern underneath#

Three of the four mistakes here share a shape: a component was sized or treated according to what it resembles rather than what it does in this system. Redis resembles a cache and is a durable queue. Postgres resembles the main database and holds almost nothing. ClickHouse resembles the thing that breaks and is usually reporting a problem created one hop upstream.

Self-hosting a trace store is not hard because any individual piece is hard. It is hard because the piece you have the strongest instincts about is the one those instincts fit worst.

---

The full capacity arithmetic, retention tiering for prompt data, redaction and deletion requests against a trace store, running evaluation as infrastructure, and debugging a slow request across six candidate layers are covered in LLM Observability in Production. The general question of which stateful systems belong inside your cluster and which you should buy is the subject of Running Databases and Stateful Systems on Kubernetes, and the operational side of running the models themselves is in LLM Operations. Related reading: Prometheus Cardinality Explosion for the same class of problem in a metrics store, and Prompt Economics for where the token volume being traced here comes from in the first place.