Latency vs Throughput vs Bandwidth
These three get conflated constantly, and confusing them in an interview signals a gap. They're distinct, they trade off against each other, and understanding them shapes performance design.
The Concept Explained
Three words, three genuinely different quantities, and a great deal of imprecise conversation because they all feel like "speed."
Latency is the time one operation takes, measured from request to response. Units are time: milliseconds, microseconds. It is a property of a single request.
Throughput is how many operations complete per unit time. Units are operations per time: requests per second, writes per second. It is a property of the system in aggregate.
Bandwidth is the maximum rate at which data can move across a link. Units are data per time: gigabits per second. It is a property of the channel, a ceiling rather than a measurement of what is happening.
The classic analogy holds up well enough to be worth keeping. A pipe carrying water has a width, which is bandwidth, the most it could ever carry. It has a length, which is latency, how long any given molecule takes to traverse it. The volume actually arriving per second is throughput, and it can be far below what the width allows if the pipe is only half full.
Bandwidth is capacity, latency is delay, throughput is delivered rate. Bandwidth is what you bought, latency is what one user feels, and throughput is what the system is actually achieving. Widening a pipe does nothing to shorten it.
That last point is the one that trips people. Latency has a floor set by physics. Light in fibre covers roughly 200,000 kilometres per second, and London to Sydney is about 17,000 kilometres in a straight line, so a round trip cannot go below roughly 170 milliseconds no matter how much bandwidth you provision. Real routes are not straight lines, so the achievable figure is higher still. No amount of money makes a packet arrive before the speed of light allows.
How It Works
The three are linked by one relationship worth memorizing, because it turns vague performance conversations into arithmetic. Little's Law states that the number of requests in flight equals throughput multiplied by latency.
If a service handles 10,000 requests per second and each takes 50 milliseconds, then 500 requests are in progress at any instant. That number determines thread pool sizes, connection counts, and memory footprint, and it is usually the constraint that fails first under load.
The law also explains why latency and throughput are not independent. Push throughput up while concurrency is capped and latency must rise, because requests queue. Push it far enough and latency rises without bound while throughput stops improving at all. This is the shape every loaded system makes: flat latency, then a knee, then a vertical wall.
Why Improving One Hurts the Other
Batching is the cleanest example, and the one to reach for in an interview.
Writing each record to disk individually means each write pays the full round trip: syscall, seek, fsync. Batching a hundred writes and flushing them together amortizes that cost across all hundred, so throughput can improve by an order of magnitude. But every record now waits for the batch to fill before it is written at all. Throughput went up. Latency for any individual write went up too.
The same trade appears everywhere. Nagle's algorithm coalesces small TCP segments to use the network efficiently, at the cost of delaying the first byte. Connection pooling reduces handshake overhead and adds queueing time when the pool is exhausted. Compression cuts bytes on the wire and adds CPU time at both ends. In each case you are buying aggregate efficiency with individual delay.
Tail Latency Is the Number That Matters
Averages are close to useless for user-facing systems, and quoting one is a reliable way to look inexperienced.
Percentiles are the right vocabulary. The p50 is the median, the experience of a typical request. The p99 is the slowest 1%, and the p999 the slowest 0.1%. The gap between p50 and p99 is usually large, because latency distributions have long right tails driven by garbage collection pauses, cache misses, lock contention, and retries.
The reason to care about the tail rather than the median is fan-out. If serving one user request requires calling 100 internal services, and each has a p99 of one second, then the probability that at least one of them is slow is 1 minus 0.99 to the power of 100, which is roughly 63%. Nearly two thirds of user requests hit at least one tail event. The p99 of an individual service becomes the common case for the user.
This is why tail latency is treated as a first-class metric at scale rather than an edge case. In a fan-out architecture, one service's rare slow path becomes the median user experience. Optimizing the p50 of a service that is called 100 times per request accomplishes almost nothing.
System Design Implications
Optimizing for Latency vs Optimizing for Throughput
Latency-optimized
Every request answered as fast as possible
Throughput-optimized
Maximum work completed per unit cost
The practical consequence is that a system cannot be tuned for both at once, and pretending otherwise produces designs that do neither well. Decide which one the workload actually needs and say so.
For latency, attack the components separately. Total latency is propagation delay plus transmission time plus processing plus queueing. Propagation is bounded by geography, so you fix it by moving data closer: CDNs, regional replicas, edge caching. Transmission is bounded by payload size, so you fix it by sending less. Processing is your code. Queueing is a capacity problem disguised as a code problem, and it is very often the dominant term under load.
For throughput, attack the per-operation overhead. Batch, pipeline, compress, and parallelize. Accept that each of these costs individual latency and confirm that is acceptable for the workload.
For the tail specifically, the techniques are different from either. Hedged requests, where a duplicate is sent to a second replica if the first has not responded by the p95 and the loser is cancelled, cut the tail dramatically for a few percent extra load. Load shedding under pressure protects the requests you do serve. Bounded queues turn unbounded latency growth into fast, honest errors. Reducing fan-out width helps more than optimizing any single dependency.
When you quote a latency number in an interview, always attach a percentile and a load level. "20 milliseconds" invites the question "at what percentile, under what load?" Saying "p99 of 20 milliseconds at 5,000 requests per second" answers it before it is asked and signals that you have measured real systems.
Tradeoffs and Decision Framework
| Technique | Effect on latency | Effect on throughput | When it is right |
|---|---|---|---|
| Batching writes | Worse | Much better | Ingestion, analytics, logging |
| Larger connection pool | Better under load | Better to a point | Until the database hits its limit |
| Compression | Worse (CPU), better (transfer) | Better on constrained links | Large payloads, slow networks |
| Caching | Much better | Better | Read-heavy, tolerant of staleness |
| Deep queues | Much worse | Slightly better | Batch work, never interactive |
| Load shedding | Better for served requests | Lower total | Protecting an interactive SLO |
| Hedged requests | Much better at the tail | Slightly worse | Wide fan-out, replicas available |
| Running at high utilization | Worse | Better | Batch, never latency-sensitive |
The framework: classify the workload as interactive or batch first, because the two want opposite settings on nearly every knob. For interactive work, set a tail latency target rather than an average, keep utilization deliberately below capacity so queues stay short, and shed load rather than letting queues grow. For batch work, drive utilization high, batch aggressively, and measure only completion time for the whole job.
Then check the bandwidth question separately, because it is a different kind of constraint. Bandwidth is a ceiling you either have or do not. If you are nowhere near it, adding more changes nothing at all, and it is remarkable how often "the network is slow" turns out to mean a latency or queueing problem on a link running at 3% utilization.
Common Mistakes
Using the three words interchangeably. They have different units. If a sentence would not survive having units attached, it is imprecise.
Quoting averages for user-facing latency. The mean hides the tail, and the tail is what users actually experience in any fan-out system.
Assuming more bandwidth reduces latency. It reduces transmission time for large payloads and does nothing at all for propagation delay or queueing.
Ignoring fan-out amplification. A p99 of one second across 100 dependencies means most user requests hit a slow path. The per-service number is not the user-facing number.
Running interactive systems at high utilization. Queueing delay grows sharply as utilization approaches capacity. Spare capacity is what buys predictable latency, and it is not waste.
Unbounded queues. They convert an overload into unbounded latency, so requests are still being processed long after the user has given up. Bounded queues fail honestly and fast.
Optimizing the wrong term. Shaving milliseconds off processing while queueing contributes hundreds is effort spent in the wrong place. Measure the breakdown before optimizing.
Define latency, throughput, and bandwidth precisely. Give an example where improving throughput hurts latency.