Scalability

An interviewer says 'design a system for 10 million users.' Before you draw a single box, you need to reason about what scaling actually means, vertical, horizontal, and where the limits are.

---

The Concept Explained

Scalability is the property that lets a system absorb more work by being given more resources. That definition contains two things people routinely skip.

The first is "more work." Work is not one number. Ten million users is not a load figure, it is a vanity figure. The load figures that matter are requests per second, writes per second, bytes stored, fan-out per write, and how unevenly those spread across keys and across time. A system serving ten million users who each read once a day is a completely different machine from one serving ten thousand users who each hold an open socket and write twenty times a second.

The second is "being given more resources." Scalability is not performance. A fast system handles more load on the hardware it already has. A scalable system handles more load when you give it more hardware. These are independent properties, and confusing them is one of the most common failures in a design interview. A single well-tuned Postgres instance is often faster than a distributed database at moderate load and completely unable to grow past one machine. That is a fast system that does not scale.

The second answer is far more common than engineers expect, and it has a cause worth understanding precisely.

---

How It Works

There are exactly two ways to add resources, and they behave very differently. Vertical scaling means one machine made bigger: more CPU, more memory, faster disk. Horizontal scaling means more machines working together behind a load balancer.

Vertical scaling is where everybody starts, because it requires no design work at all. Buy the bigger instance. It works remarkably well and for far longer than distributed systems enthusiasts like to admit. Its problem is that it terminates. There is a largest instance your cloud sells, and when you reach it there is no next step, only a rewrite under time pressure.

Horizontal scaling has no such wall, but it only works if the work can be split. That condition is the whole subject.

Why Adding Machines Stops Helping

Naively, ten machines should do ten times the work of one. They never do, for two reasons that compound.

Contention. Some fraction of every request touches something shared: a lock, a hot row, a single writer. That fraction cannot proceed in parallel no matter how many machines you add. This is Amdahl's argument, and it sets a ceiling. If 5% of the work is serialized, you cannot exceed twenty times the throughput of one machine, however much hardware you buy.

Coherency. Nodes must agree with each other, and agreement gets more expensive as participants are added. Cache invalidation, replica synchronization, and consensus rounds all grow with cluster size. Unlike contention, this is not merely a ceiling. It is a penalty that keeps growing, which means throughput can peak and then decline as you add machines.

Reads and Writes Scale Differently

This asymmetry drives most real architectures.

Reads are easy to scale because they can be copied. Add read replicas, add caches, add a CDN. Every copy answers independently, and copies do not need to consult each other to serve a question. Read capacity tracks the number of replicas closely.

Writes are hard to scale because they must be reconciled. Two nodes accepting writes to the same key have to agree eventually, and agreement costs a round trip at best and a consensus protocol at worst. The standard escape is sharding: split the keyspace so any given key has exactly one writer, so writes to different keys never interact. That turns one hard problem into many easy ones, at the cost of cheap transactions and any query that spans shards.

---

System Design Implications

Take the interview prompt directly: 1,000 requests per second today, 100,000 required. That is 100 times growth. Here is how to reason about it out loud.

Start with the split. At 100,000 requests per second, what is the read to write ratio? A 95:5 split means 95,000 reads and 5,000 writes. The reads are a caching and replication problem, which is tractable. The 5,000 writes are the real design. Interviewers are checking whether you separate these before you start drawing boxes.

Then use Little's Law. Concurrency equals throughput multiplied by latency. At 100,000 requests per second with 50ms average service time, roughly 5,000 requests are in flight at any instant. That single number tells you how many threads, connections, and sockets the system has to hold, and it is very often the thing that breaks first.

Then walk the bottleneck. Bottlenecks move as you scale, and naming the sequence is what separates a senior answer from a staff one.

Notice that the first two steps are operational and the last two are architectural. Most systems can reach roughly ten times their current load with no design change at all. It is the second order of magnitude that forces the sharding conversation, and that is where interviewers want your time to go.

Finally, say your assumptions out loud. If you assume 100,000 requests per second, a 95:5 read to write ratio, and 2KB per record, state it. An interviewer cannot evaluate reasoning they cannot see, and a stated assumption that turns out to be wrong is a much better signal than a number that appeared from nowhere.

---

Tradeoffs and Decision Framework

Dimension	Vertical scaling	Horizontal scaling
Mechanism	More CPU, memory, faster disk	Add nodes behind a balancer
Time to implement	Hours	Weeks to months
Application changes	None	Statelessness or sharding required
Upper bound	Largest available instance	Coordination cost, not hardware
Cost per unit of capacity	Rises sharply at the top	Roughly flat
Effect on availability	None, still one failure domain	Improves, node loss is survivable
Downtime to scale	Often a restart	None when done well
Operational complexity	Low	High: placement, rebalancing, partial failure
Best suited to	Databases, early stage, stateful work	Web tiers, stateless compute

The decision rule is short. Scale vertically until one of three things is true: you are within sight of the largest instance available, the single machine has become an availability risk you cannot accept, or the cost curve has bent past the point where more small machines are cheaper. Until then, vertical scaling is the correct engineering answer, and saying so in an interview is a strength rather than a lack of ambition.

For horizontal scaling the sequence is always the same: make the tier stateless first, push state into a store built to hold it, and only then shard that store. Teams that try to shard before removing state from the application layer end up with sticky sessions, which is sharding with none of the benefits.

---

Common Mistakes

Treating scalability and performance as one thing. Tuning a single node raises the line, it does not change the curve, and it does nothing about the ceiling you are about to hit.

Reaching for microservices as a scaling answer. Service boundaries are an organizational and deployment tool. Splitting a monolith into services that all hit the same database moves the bottleneck nowhere.

Assuming linear scaling. Contention caps you and coordination cost can actively reverse you. Any capacity plan built on "ten machines equals ten times" is wrong before it is finished.

Scaling the stateless tier and calling it done. The app tier is the easy half. The interviewer is waiting for the write path.

Ignoring connection limits. Databases usually fall over from connection exhaustion long before they run out of CPU, and no amount of vertical scaling fixes a pooling problem.

Sharding too early. Sharding costs you cross-shard transactions, joins, and simple aggregate queries, permanently. Do it when the single writer is genuinely the wall, not in anticipation of it.

Skipping the load estimate. Without requests per second, a read to write ratio, and a payload size, every decision that follows is arbitrary, and the interviewer can tell.

---