Reliability
A system can be available but unreliable, returning wrong answers, losing data, corrupting state. Reliability is a distinct property and interviewers probe whether you understand the difference.
The Concept Explained
Availability asks whether the system responded. Reliability asks whether the response was correct, and whether the system did what it promised with the data you gave it.
They are separate properties and they fail separately. A service that returns HTTP 200 with a stale, empty, or wrong body is perfectly available and completely unreliable. Every dashboard is green. Every health check passes. Users are being lied to at full speed, which is worse than an outage, because an outage is at least honest about its state.
The reason this distinction is an interview favourite is that it exposes whether a candidate thinks about correctness at all, or only about uptime. Uptime is easy to measure, so it is what most monitoring measures, and so it becomes what many engineers optimize by default. The failures that damage a business most are usually not downtime. They are silent corruption, lost writes, and duplicated side effects, all of which happen while availability metrics look excellent.
Availability is "did it answer?" Reliability is "was the answer right, and is my data still intact?" A system can score perfectly on the first while failing catastrophically on the second, and the second failure is the one that is hard to detect and hard to undo.
Reliability also has a time dimension that availability lacks. Availability is a ratio you can compute for any window. Reliability is about the probability that the system performs correctly over a period, which is why it degrades with load, with time, and with the accumulation of small unhandled edge cases.
How It Works
Reliability is built from a small set of properties, and it is worth being able to name them separately because they fail separately.
Available but Unreliable vs Reliable but Unavailable
Available, unreliable
Answers every request, answers wrongly
Unavailable, reliable
Refuses to answer rather than answer wrongly
The right-hand column is a deliberate design choice, and a good one for anything holding money or state. Refusing service is recoverable. Corrupting data is often not.
The Components of Reliability
Fault tolerance is the ability to keep producing correct results while some component is broken. Note the word correct. A system that stays up during a failure but starts serving stale reads has traded reliability for availability, which may be right or wrong depending on what the data is.
Durability is the promise that acknowledged writes survive. This is the property users assume without being told and the one whose violation is least forgivable. If you return success on a write, that write must survive the failure of the node that accepted it, which means it must have been replicated or committed to stable storage before you answered.
Correctness under concurrency is where most subtle unreliability lives. Two requests arriving at the same moment, a retry arriving after a timeout, a message delivered twice. Each is individually handled and collectively catastrophic if the operations are not idempotent.
Graceful degradation is the ability to shed functionality rather than accuracy. A recommendation panel that disappears when its service is down is graceful. A recommendation panel that shows another user's recommendations is not degradation, it is a correctness failure wearing degradation's clothes.
Why Retries Are a Reliability Problem
The classic trap: a request times out, so the client retries. The original request had already succeeded, the response was simply lost on the way back. The retry now applies the operation a second time. Availability tooling created a reliability failure.
This is why idempotency is not an optimization but a precondition for safe retries, and why any interview answer that reaches for retries should mention it in the same breath. Without idempotency, every retry mechanism you add is a duplicate-execution mechanism.
Redundancy improves availability but does nothing for reliability on its own, and can actively hurt it. Adding replicas means more copies that can disagree. Without a defined way to resolve that disagreement, you have simply increased the number of places a wrong answer can come from.
System Design Implications
The design question is always the same: when this component fails, would you rather be wrong or be down? Answer it per data type, not per system, because the answer genuinely differs.
For a bank balance, an inventory count, or a permission check, being wrong is unacceptable and being down is merely bad. These paths should fail closed: refuse the operation, return an error, make the failure loud. For a recommendation feed, a view counter, or a "people also bought" panel, being briefly wrong is fine and being down is a worse user experience. These should fail open: serve stale, serve empty, serve a default.
Getting this backwards is a serious design error, and interviewers construct scenarios specifically to catch it. Serving a stale permission check because the auth service is slow is exactly the failure mode that turns a minor outage into a security incident.
The mechanisms that buy reliability are worth naming explicitly:
Write acknowledgement discipline. Do not return success until the write is durable to the standard you have promised. If you acknowledge after writing to one node's memory, you have promised durability you cannot deliver.
Idempotency keys. Give every mutating operation a client-supplied identifier so a duplicate arrival is recognized and discarded rather than applied twice. This is what makes retries safe, and retries are what make availability recoverable.
End-to-end checksums. Verify data at the boundary it is read, not only where it was written. Silent corruption between the two is exactly the failure mode that availability monitoring cannot see.
Failing closed on the paths that matter. When the system cannot establish the truth, returning an error is a correct answer. Guessing is not.
Reconciliation. Accept that divergence will happen and build the process that detects and repairs it. Any system holding money has one of these, usually running nightly, and its existence is an admission that reliability is maintained rather than achieved once.
Tradeoffs and Decision Framework
| Choice | Favours | Cost |
|---|---|---|
| Fail closed on error | Reliability | Lower availability, visible errors |
| Fail open, serve stale | Availability | Users act on wrong data |
| Synchronous replication before ack | Durability | Higher write latency |
| Asynchronous replication | Write latency | Acknowledged writes can be lost on failover |
| Aggressive client retries | Recovery from transient faults | Duplicate execution without idempotency |
| Strong validation at boundaries | Catches corruption early | Throughput cost, more rejected requests |
The framework has three steps. Classify each data path by what a wrong answer costs. Choose fail-open or fail-closed per path from that cost, not as a system-wide default. Then make sure your monitoring can actually see the reliability failures you have chosen to risk, because the defining property of unreliability is that uptime dashboards do not show it.
A strong interview move is to name the correctness metric alongside the availability metric. "99.9% availability, and separately, zero acknowledged writes lost" tells the interviewer you understand these are different promises requiring different mechanisms. Most candidates only offer the first.
Common Mistakes
Using the terms interchangeably. They are different properties with different mechanisms, and conflating them is the exact thing the question is designed to detect.
Adding retries without idempotency. This converts a transient failure into duplicate execution, which is a reliability failure created by an availability mechanism.
Acknowledging writes before they are durable. A success response the system cannot honour after a node loss is the most damaging lie a system can tell.
Failing open on paths that hold truth. Serving stale permissions, stale balances, or stale inventory to preserve uptime trades a small outage for a large incident.
Assuming replicas improve correctness. More copies means more opportunities to disagree unless the reconciliation rule is defined.
Monitoring only uptime. If nothing measures correctness, silent corruption runs for as long as it takes a customer to complain.
Calling any degradation graceful. Removing a feature is graceful. Showing wrong data in place of right data is a correctness failure with better marketing.
Explain the difference between availability and reliability. Give an example of a system that is highly available but unreliable.