Production Systems Engineering

Deployment Strategies Overview

You have new code ready and users depending on the current version. How you get from old to new, without downtime, without breaking things, and with a way back, is one of the most important operational skills there is.


The Concept Explained

A build has passed review and tests, and the artifact is sitting in a registry. Several thousand people are meanwhile using the version currently running. Getting from one to the other is a production operation, not a build step, and it is where most user-visible incidents are created.

The simplest way is to stop what is running and start the new thing. Between the stop and the first healthy new instance, nothing serves traffic. That interval is not the second it takes to launch a process; it is process start, dependency connections, cache warming, and whatever readiness gate the platform applies, which on a real service is closer to a minute. Every request arriving in that window fails.

Then the worse half. Suppose the new version is broken in a way the tests did not catch, which is the normal case for production faults: a configuration value that only exists in the live environment, an assumption about a schema, a dependency that behaves differently under real concurrency. You discover this with 100% of traffic already on the broken version, and getting back means running the same stop-and-start operation in reverse, which is a second outage of the same length.

Two properties caused that damage. The change was total, so every user was exposed to it. And the change was slow to undo, so exposure lasted as long as a full deployment. Every strategy that follows is an attempt to weaken one or both.

Three moves are available. Overlap: run old and new at the same time so there is never a gap with nothing serving. Retain: keep the old version alive and idle instead of destroying it, so undo becomes a traffic switch rather than a rebuild. Split: send a fraction of traffic to the new version first, so a fault is found by a few users instead of all of them.

Those three moves generate the whole family. Rolling deploys overlap. Blue-green retains. Canary splits, and usually retains as well. They are not four unrelated techniques to memorize.

KEY CONCEPT

The four strategies are one spectrum, not a menu. Moving along it you spend capacity, time, and operational complexity, and what you buy in return is a smaller blast radius and a faster path back. Recreate spends nothing and exposes everyone with no cheap undo; canary spends the most and exposes a controlled slice with an undo measured in seconds. Choosing a strategy is therefore not a matter of picking the most advanced one available, but of deciding how much a bad release would cost you and buying exactly that much protection.


How It Works

The Deployment Spectrum: What Each Step Costs and What It Buys

Recreate

Stop every old instance, then start the new ones. Costs a full outage for the length of a restart cycle and offers no undo faster than a second deployment. Buys absolute simplicity and the guarantee that only one version is ever live, which matters when two versions genuinely cannot coexist.

Rolling

Replace instances in batches so capacity never drops to zero. Costs a period where both versions serve real traffic simultaneously, and a rollback that is itself another rolling deployment. Buys zero downtime for roughly the cost of one extra batch of capacity, which is why it is almost every platform default.

Blue-green

Stand up a complete second environment on the new version, verify it, then move all traffic across at once. Costs double infrastructure for the overlap window and forces you to solve shared state and schema compatibility explicitly. Buys a rollback that is a single traffic switch taking seconds, and a full environment you can test before any user reaches it.

Canary

Route a small share of live traffic to the new version, watch it, then widen. Costs traffic-control machinery, meaningful metrics, and a slower release that a human must supervise. Buys the only thing the others cannot give you: a fault is discovered by a small measured fraction of users rather than by all of them.

Canary with automated analysis

The same split, with promotion and abort decided by comparing metrics between the two versions rather than by a person watching a dashboard. Costs real investment in signal quality and a tolerance for occasional false aborts. Buys consistency and speed of reaction at three in the morning, when human judgement is at its worst.

Hover to expand each layer

The spectrum is usually drawn as a straight line, but the jump from rolling to blue-green is different in kind from the others. Recreate, rolling, and canary all change which instances exist; blue-green changes which environment is authoritative. That is why blue-green is the only one whose rollback does not involve moving code at all, and also why it is the one that forces uncomfortable questions about shared databases, caches, and queues to surface before you deploy rather than during.

Every Strategy Except Recreate Runs Two Versions At Once

This is the property practitioners underestimate, and it is not an implementation detail of any single strategy. If there is no downtime, then for some window old and new are both serving live traffic. That window is minutes for rolling, seconds for blue-green, and potentially hours for a canary bake.

The consequence is that both versions must be compatible with each other and with a single shared data store. A new version that adds a required column, renames a field in a cached object, or changes the format of a message on a queue will produce failures that look like random errors rather than deployment errors, because they only affect requests that happen to cross the version boundary. This is what forces the expand-and-contract discipline: change schemas in additive, backward-compatible steps, and only remove the old shape once no running version depends on it.

WARNING

A fast rollback rolls back code, not data. If the new version wrote rows the old version cannot read, or consumed messages the old version would have handled differently, flipping traffic back leaves you running old code over new data. Blue-green makes this especially easy to overlook because the switch feels instantaneous and reversible, which it is at the traffic layer and is not at the storage layer. The strategy determines how fast you can undo the deployment; it says nothing about whether undoing it restores a working system.

Rollback Speed Is a Design Property, Not a Button

Every strategy has a nominal rollback path, and the useful question is how long the path takes and how much of it is automatic. Recreate rolls back at the speed of a deployment. Rolling rolls back at the speed of another rolling deployment, which is the full duration again. Blue-green and canary roll back at the speed of a routing change.

That difference sets your realistic recovery time far more than the deployment mechanism does. A team that deploys in ninety seconds but needs ten minutes to reverse has a ten-minute worst case, and it is the worst case that customers experience.


Production Implications

Choose by blast radius, not by sophistication. The question is what a bad release costs in the first five minutes. A payment path or an authentication service justifies canary; a batch report generator with no live users does not, and running one there is complexity nobody will maintain.

Statefulness pulls you left on the spectrum. Services holding long-lived connections, in-memory session state, or a leader election are harder to run in two versions at once. Sometimes recreate during a maintenance window is genuinely the correct answer, and saying so is a sign of judgement rather than a lack of it.

Canary needs traffic volume to work. Detecting a rise in error rate on 1% of traffic requires enough requests in that 1% for the comparison to mean anything. A service handling a few requests a minute cannot canary meaningfully, and a canary that cannot detect anything is a slow deployment with extra steps.

Blue-green is a capacity decision before it is a deployment decision. It requires the ability to run two full environments at the same time. That is straightforward with elastic infrastructure and expensive or impossible with fixed capacity, and the answer to that question decides whether the strategy is available at all.

Backward compatibility is the prerequisite, not the refinement. Any zero-downtime strategy assumes two versions can coexist. Teams that skip expand-and-contract schema changes get intermittent errors they blame on the deployment tooling.

The strategy is only as good as the signal. Canary and automated rollback depend on knowing within minutes that the new version is worse. Without per-version error rate and latency, you have a slower deployment with the same detection time as recreate.

Rollback must be rehearsed. Roll back deliberately on a low-stakes release and time it, because the number you measure is your real recovery time.

PRO TIP

Answer this by naming the axis before the strategies. Say that the four sit on a spectrum trading capacity and complexity for blast radius and rollback speed, then walk recreate, rolling, blue-green, canary in that order, one sentence each on cost and benefit. Then pivot to selection criteria: stakes, statefulness, traffic volume, and whether you can afford double capacity. Most candidates list the four correctly and stop there, which sounds like recall. The distinguishing move is stating that every zero-downtime strategy implies two versions serving at once, so backward compatibility is a precondition, and that a fast rollback returns the code but not the data.


Tradeoffs and Decision Framework

AxisRecreateRollingBlue-greenCanary
DowntimeFull restart windowNoneNone at the switchNone
Extra capacityNoneOne batchA second environmentA small overlap
Rollback speedAnother deploymentAnother full rollSeconds, a route changeSeconds, withdraw the slice
Blast radiusEveryoneEveryone, progressivelyEveryone, after the switchThe exposed slice only
Versions live at onceOneTwo, for minutesTwo, brieflyTwo, for the whole bake
Operational complexityLowestLow, usually built inModerateHighest

The framework in four questions. What does five minutes of the new version being wrong actually cost, since that sets how much you should spend on containment? Can two versions of this service safely run at once, because if they cannot, everything except recreate is off the table until the data model changes? Can you afford a second full environment for the length of a release? And do you have enough traffic and enough per-version signal for a small slice to tell you anything?

Default to rolling. It is zero-downtime, cheap, and built into most orchestration, and for the large majority of services it is proportionate. Move to blue-green when rollback speed is the binding constraint and you can pay for the capacity. Move to canary when the cost of exposing everyone is high enough to justify supervising the release, which in practice means payments, authentication, checkout, and the handful of paths where a bad five minutes is a real incident.


Common Mistakes

Treating the strategies as a maturity ladder. Canary on an internal tool with forty users is complexity that will rot. Match the strategy to the stakes.

Zero-downtime deployment with incompatible schema changes. Adding a required column mid-rollout produces errors that look random because they only hit requests served by one version.

Assuming rollback undoes everything. Traffic returns to the old code instantly; data written by the new version stays. Plan the data path separately.

Canary with no per-version metrics. Splitting traffic without comparing error rate and latency by version gives you a slow deployment and no earlier detection.

Canary slices too small to be significant. A slice with too few requests cannot distinguish a real regression from noise, so nothing is learned before promotion.

Blue-green without a plan for shared state. Two environments over one database, cache, or queue means the switch is only clean at the traffic layer.

Rolling forward to fix a bad release by habit. Reverting to a version known to work is faster and more predictable than shipping an untested fix into an active incident.


INTERVIEW QUESTION

Walk me through the main deployment strategies. For a payment system versus an internal dashboard, which would you choose for each and why?