Distributed Systems Design

Availability

A stakeholder asks for 'five nines.' Do they know that's 5 minutes of downtime per year, and what it costs to get there? Availability is a number you must be able to reason about precisely.


The Concept Explained

Availability is the fraction of time a system is able to serve requests. It is a ratio, usually written as a percentage, and it is the one system property that non-engineers will quote at you with total confidence and no sense of what it costs.

The useful form is uptime divided by total time, but that hides the two quantities that actually drive the number: how often the system breaks, and how long it stays broken. Availability improves either by breaking less often or by recovering faster. In practice, recovering faster is almost always the cheaper lever, and it is the one engineers under-invest in because it feels less like real engineering than preventing the failure outright.

That reframing matters in interviews. If someone asks you to design for higher availability and you only talk about redundancy, you have addressed half the equation. Automated failover, fast health checking, small blast radius, and quick rollback all attack recovery time, and they usually buy more availability per unit of effort than another replica does.

KEY CONCEPT

Availability is a conversation about time, not about hardware. Two systems with identical redundancy can differ by an order of magnitude in availability purely because one detects and routes around failure in seconds and the other needs a human to wake up and run a runbook.

The other thing to establish early is what "available" even means for the system in question. Available to whom, doing what? A checkout service that returns cached product pages but cannot take payment is up by one definition and completely down by the definition that matters to the business. Define the critical user journey first, then measure availability against that.


How It Works

The nines are the standard shorthand, and you should know this table without reaching for a calculator. A year is 525,600 minutes, so the arithmetic is straightforward once you have the anchor.

AvailabilityDowntime per yearDowntime per 30 daysRecovery posture this demands
99% ("two nines")3.65 days7.2 hoursManual recovery during business hours
99.9% ("three nines")8.76 hours43.2 minutesOn-call rota, manual failover
99.99% ("four nines")52.6 minutes4.3 minutesAutomated failover, no human in the loop
99.999% ("five nines")5.26 minutes26 secondsAutomatic, plus redundancy across regions

Read the right-hand column carefully, because it is the part interviewers are testing. At three nines you can afford a human to notice a page, open a laptop, and run a documented procedure. At four nines you cannot: 52 minutes is the entire annual budget, and a single incident where someone takes twenty minutes to respond consumes nearly half of it. Four nines is not a hardware decision, it is the decision to remove humans from the recovery path.

Availability Composes, and It Composes Badly

The number that surprises people is what happens when you chain dependencies.

If a request must pass through three services in sequence, and each is independently available 99.9% of the time, the end-to-end availability is 0.999 multiplied by itself three times, which is 99.7%. Your three carefully engineered three-nines services have produced a user-facing system with 26 hours of downtime a year.

This is why availability targets have to be set on the critical path as a whole, not per service, and it is why reducing the number of synchronous dependencies is one of the highest-leverage availability moves available. Every service you remove from the synchronous path multiplies back in your favour.

Redundancy composes in the opposite direction, and just as sharply. Two independent replicas that are each available 99% of the time give a combined availability of 99.99%, because both must fail simultaneously for the system to be down. Two replicas at 99.9% each give 99.9999%.

WARNING

That redundancy calculation assumes the failures are independent, and in real systems they very often are not. Two replicas in the same rack share a top-of-rack switch. Two availability zones share a region's control plane. Both replicas running the same code fail identically on the same poison input. Correlated failure is the reason real systems never reach the availability their redundancy maths predicts.


System Design Implications

Designing for 99.99% is a concrete exercise, and it is worth being able to walk through it in order.

What Each Availability Tier Actually Requires

99% : single instance, good monitoring

One server, backups, and someone who notices when it is down. Three and a half days of annual downtime is a lot, but for internal tools this is often the right amount of engineering. Cost is minimal.

99.9% : redundant instances, manual failover

Multiple app instances behind a load balancer, a database with a standby replica, health checks that pull bad instances out. Failover is human-triggered. On-call is now required. This is where most business systems should sit.

99.99% : multi-AZ, automated failover

Redundancy across availability zones so a single datacenter loss is survivable. Failover must be automatic because the annual budget is 52 minutes. Requires leader election, automated promotion, and connection draining that works without supervision.

99.999% : multi-region, active-active

Requests served from more than one region simultaneously, with global routing that sheds a failed region in seconds. Now you own cross-region data replication and every consistency problem it brings. The cost is not the hardware, it is the permanent engineering burden.

Hover to expand each layer

Working through 99.99% specifically, the design has four parts, and they map exactly onto the two levers.

Remove single points of failure. Multiple app instances, a database with automatic promotion, availability zone diversity so a datacenter loss is a degradation rather than an outage. This attacks failure frequency.

Make failover automatic and fast. Health checks that detect failure in seconds, not minutes. Automatic promotion of a standby without human approval. Connection draining and client retry so in-flight requests survive the transition. This attacks recovery time, and it is where the budget is really won or lost.

Shrink the blast radius. Cell-based architecture, per-tenant isolation, and bulkheads so that one failure takes down a fraction of traffic rather than all of it. Partial outages consume the error budget proportionally, so a failure that affects 5% of users costs you 5% of the downtime a total outage would.

Cut synchronous dependencies. Every service on the critical path multiplies into the availability figure. Making a dependency asynchronous, cached, or optional removes it from that product entirely. This is usually the cheapest large win available and it is the one candidates most often miss.

The cost framing is what turns this from a checklist into a senior answer. Each nine is roughly an order of magnitude more expensive than the last, and past four nines the dominant cost stops being infrastructure and becomes engineering time: the permanent burden of running active-active, of testing failover continuously, of maintaining the discipline that keeps correlated failures out.

PRO TIP

When a stakeholder asks for five nines, the productive response is not to argue about feasibility. It is to ask what a minute of downtime costs, and what the current number actually is. Most organizations asking for five nines are at three, and the honest gap between those is a budget conversation rather than an architecture one.


Tradeoffs and Decision Framework

DecisionBuys youCosts you
Add redundant instancesSurvives single-node lossLinear infrastructure cost, more to operate
Multi-AZ deploymentSurvives a datacenter failureCross-AZ traffic charges, some added latency
Multi-region active-activeSurvives a region failureCross-region consistency, large permanent complexity
Automated failoverCuts recovery time hugelyRisk of false-positive failover and split brain
Aggressive health check timeoutsFaster detectionHealthy nodes evicted during load spikes
Fewer synchronous dependenciesDirectly multiplies availability upAsync complexity, eventual consistency

The framework: pick the target from the cost of downtime, not from ambition. Work out what an hour of outage costs the business, compare that to the annual cost of the next nine, and stop where the curve crosses. For most systems that lands at three nines, which is an entirely respectable and defensible answer in an interview when you can show the reasoning.

Then, whatever target you pick, measure against the critical user journey rather than infrastructure uptime. A fleet reporting 100% instance health while checkout fails is a system that is down, no matter what the dashboard says.


Common Mistakes

Quoting nines without translating them to time. "Four nines" means nothing until you say 52 minutes a year, and the reaction in the room usually changes once you do.

Only designing against failure frequency. Redundancy without fast automatic recovery leaves most of the available improvement on the table.

Multiplying availability the wrong way. Chained dependencies multiply downward and redundant components multiply upward. Getting the direction wrong makes the whole estimate meaningless.

Assuming failures are independent. Shared racks, shared control planes, and shared code make real correlated failure far more likely than the redundancy maths suggests.

Measuring infrastructure instead of the user journey. Instance health checks pass constantly during outages that are obvious to every customer.

Treating planned maintenance as free. If your SLA does not exclude it, a maintenance window spends the same error budget an incident does.

Ignoring the failover path itself. Automated failover that has never been exercised is a hypothesis. The most expensive outages come from a standby that was not ready when it was finally called on.


INTERVIEW QUESTION

What does 99.99% availability actually mean in downtime per year? Design a system to achieve it, and explain what each nine costs you.