The Arithmetic of a Corpus

Someone asks what it costs to make ten million documents searchable. The honest answer is a calculation nobody in the room has done, and it decides the architecture before a line of code is written.

---

The Problem at Scale

This is the anchor lesson of the course. Everything after it assumes you can do this arithmetic, because the difference between a retrieval system that works and one that quietly cannot be operated is usually a number somebody never calculated.

The chain is short and every link multiplies:

documents  ->  chunks  ->  vectors  ->  bytes  ->  memory  ->  nodes

Work it for ten million documents.

Documents to chunks. A document is not the unit of retrieval. It is split, and the chunk count is what matters. At roughly five chunks per document, which is modest for anything longer than a page:

10,000,000 documents  x  5 chunks  =  50,000,000 chunks

Chunks to vectors. One vector per chunk. Fifty million vectors.

Vectors to bytes. This is where the number becomes real. A vector is an array of floats, and its size is dimensions times bytes per dimension:

1024 dimensions  x  4 bytes (float32)  =  4,096 bytes per vector
50,000,000  x  4,096  =  204,800,000,000 bytes  =  ~205 GB

Two hundred and five gigabytes, for a corpus that a lot of organisations would describe as medium. That is before the index structure, before the metadata, before any working memory for queries.

---

How It Works

The graph overhead, and why it is not the problem

It is worth doing this term explicitly, because people assume the index structure is the expensive part and then tune the wrong thing.

A graph index gives each vector a bounded number of links to neighbours. With a maximum of sixteen connections per node and roughly double that at the base layer, and a link stored as a four byte identifier:

~32 links  x  4 bytes  =  ~128 bytes per vector of graph
50,000,000  x  128 bytes  =  ~6.4 GB

total, float32:  204.8 GB vectors + 6.4 GB graph  =  ~211 GB

Against 205 GB of vectors, the graph is about three percent. Raising the connection count to thirty two doubles the graph term and still leaves it small next to the vectors.

This is a property of high dimensional float32 vectors specifically. At 1024 dimensions a vector is 4 KB and a link is 4 bytes, so the vectors win by orders of magnitude per node. The intuition that graph overhead matters comes from low dimensional examples where it genuinely does, and published estimates of an HNSW index costing one and a half to two times the raw vectors come from exactly those, where a vector is a few hundred bytes rather than four kilobytes.

Hold onto the three percent lightly, because the next section changes it.

The two levers that actually move the number

Bytes per dimension. Quantizing from float32 to int8 is a straight four times reduction:

50,000,000  x  1024  x  1 byte   =  ~51 GB vectors
                    plus graph   =  ~6.4 GB
total, int8                      =  ~58 GB

The same corpus, from 211 GB to 58 GB, by changing one thing. That is the difference between needing several nodes and fitting on one, which is why Module 2 treats quantization as mandatory at scale rather than as an optimisation.

Notice what happened to the graph term. It did not move, because the number of links is set by M rather than by how the vectors are stored. But the vectors shrank around it, so the graph went from about three percent of the index to about eleven percent. Quantization does not shrink the graph, so it raises the graph's relative share and puts a floor under how small the index can get. At extreme compression the graph stops being a rounding error and becomes a real term, which is worth knowing before assuming the next compression step will pay like the last one did.

Number of vectors. Chunk size sets this directly and linearly. Halving the chunk size doubles the chunk count, which doubles the vectors, the memory, the index build time and the embedding bill. Chunking is usually discussed as a retrieval quality decision, and it is also the largest single input to your infrastructure cost.

From memory to nodes

Now turn bytes into machines, which is the step that produces the architecture.

A node cannot use all its memory for vectors. The index needs working memory for queries, the process needs headroom, and the operating system needs its share. Budgeting around seventy percent of node memory for the index is a reasonable planning figure to start from and to verify against your own store.

float32, 211 GB  ->  needs ~301 GB usable  ->  does not fit a 256 GB node
int8,     58 GB  ->  needs  ~83 GB usable  ->  fits comfortably, with room to grow

That one calculation is the whole architecture decision. Unquantized, this corpus is a sharded multi node deployment with everything Module 3 covers: scatter-gather, top-k merge, recall loss at the boundary, and a rebuild that has to be coordinated. Quantized, it is one node with a replica for availability, which is a dramatically simpler system to operate.

Now scale it to fifty million documents and neither option is a single node:

250,000,000 chunks, int8  =  ~288 GB  ->  sharded regardless

Which tells you the useful thing: quantization buys you roughly one order of magnitude of corpus growth before the architecture has to change again.

---

Building and Operating It

Do this arithmetic on your own corpus before choosing anything. The inputs are four numbers and most teams can get them in an afternoon.

def index_memory_gb(documents, chunks_per_doc, dims, bytes_per_dim, links=32):
    vectors = documents * chunks_per_doc
    vector_bytes = vectors * dims * bytes_per_dim
    graph_bytes = vectors * links * 4
    return (vector_bytes + graph_bytes) / 1e9

# The corpus above, both ways
print(index_memory_gb(10_000_000, 5, 1024, 4))   # ~211 GB, float32
print(index_memory_gb(10_000_000, 5, 1024, 1))   # ~58 GB,  int8

Measure the four inputs rather than assuming them:

# Chunks per document: the input people guess wrong most often
#   Sample a few thousand real documents through your actual chunker
#   and take the mean. Guessing this at 3 when it is 8 is a 2.7x error
#   in every downstream number.

# Dimensions: a property of the embedding model, and not negotiable
#   without re-embedding everything, which is Module 5.

Then plan for growth explicitly, because the number that matters is not today's:

corpus growth 8% per month  ->  ~2.5x in twelve months

A design that fits exactly today is a design that shards in seven months, and the sharding decision is much cheaper made deliberately than made under pressure.

---

Tradeoffs and Decision Framework

Lever	Effect on memory	Effect on quality	Reversible?
Quantize float32 to int8	Four times smaller	Small recall loss, recoverable with rescoring	Yes, rebuild the index
Quantize further	Up to thirty times smaller	Larger recall loss, needs rescoring to stay usable	Yes, rebuild
Larger chunks	Linearly fewer vectors	Coarser retrieval, more irrelevant text per hit	Requires full re-embedding
Lower dimensions	Linear in the dimension count	Depends entirely on the model	Requires full re-embedding
More nodes	No effect on total, splits it	Sharding costs recall, see Module 3	Yes, but operationally expensive

Note the rightmost column, because it is the one that gets ignored. Quantization is a rebuild. Chunk size and dimensions are a full re-embedding, which at this corpus is a multi day job covered in Module 5. The cheap levers and the expensive levers look similar on a whiteboard and are nothing alike in practice.

Three questions settle the initial design. How many vectors will you have in twelve months, not today. Can you quantize, which is a recall question you answer by measuring rather than by preference. And does the result fit one node, because single node against sharded is the largest architectural fork in this course and everything in Module 3 follows from crossing it.

Default: do the arithmetic with a twelve month corpus, assume int8 quantization with rescoring, and treat fitting on one node as worth real effort to achieve.

---

Failure Modes and Common Mistakes

Never doing the calculation. The most common failure, and the one that produces an architecture discovered rather than chosen.

Guessing chunks per document. It multiplies every downstream number and it is measurable in an afternoon.

Sizing for today's corpus. Growth turns a comfortable design into a sharding project on a timeline you did not pick.

Assuming graph overhead is significant. At realistic dimensions it is a few percent, and tuning it does not solve a memory problem.

Budgeting one hundred percent of node memory. Query working memory, process headroom and the operating system are not optional, and an index sized to exactly fill a node does not run.

Treating chunk size as purely a quality decision. It is the largest single input to vector count, and therefore to memory, build time and embedding cost.