CI/CD Pipelines
Code goes from a developer's laptop to production through a pipeline. Designing that pipeline (build, test, security scan, deploy) safely and quickly is core production engineering.
The Concept Explained
A developer merges a three-line change to a timeout default. Between that merge and the moment those bytes serve real traffic, a list of things has to become true: the code compiles, the tests pass, no dependency carries a known critical vulnerability, the thing being deployed is the thing that was tested, and something has decided it is allowed into production.
Done by hand, that list is done inconsistently. Someone builds on a laptop with a different toolchain. Someone skips the integration suite because the change is small. The pipeline makes the list mechanical, identical for every change, and impossible to skip quietly.
The naive version is one script that runs on every merge, does everything in sequence, and deploys at the end. It works, and it degrades in two predictable ways.
The first is feedback time. As the suite grows the script takes forty minutes, so a syntax error in a test fixture is reported forty minutes after the author moved on. Authors respond by batching changes. Feedback latency does not merely annoy developers; it changes the size of the changes they ship, and larger batches are harder to debug and riskier to reverse.
The second is subtler. If each environment builds from source, staging and production run different binaries. A resolver picked up a new patch release overnight, or a base image moved. Every test that passed did so against an artifact that no longer exists, and you have deployed something nobody validated while believing you did.
Both problems have one shape of answer. Build the artifact exactly once and promote those identical bytes through every stage, so testing means testing the thing you will run. Then order the stages so the cheapest checks most likely to fail run first, and failure arrives in minutes rather than at the end.
A pipeline is not a sequence of things that must happen before deploying. It is a machine for accumulating evidence about one specific artifact. That framing settles both design questions: the artifact is built once and promoted by digest, because a rebuild is a different artifact and silently invalidates every test that ran against the first; and stages are ordered cheapest-and-most-likely-to-fail first, because the value of a check is the evidence it produces divided by the time it costs everyone waiting behind it.
How It Works
From Commit to Production, and What Is Handed Between Stages
Click each step to explore
What the diagram cannot show is that the thing flowing along those arrows is a digest, not a branch. Two pipelines drawn identically can differ entirely on that point, and it is the difference between "we tested this release" and "we tested a build from the same commit, probably."
Where a Gate Earns Its Latency
A gate is worth its cost when it can stop something the automation cannot see: coordinating with an external party, a schema migration whose backfill needs a window, the first deployment of a system with no operational history.
A gate is theatre when the approver holds no information the pipeline lacks. An approval clicked by whoever is on shift, on a change they did not write and cannot evaluate, adds hours of latency and produces no evidence. Worse, it launders responsibility: the change was approved, so nobody owns the outcome. If the approver cannot describe what would make them click no, the gate is a delay, not a control. A slow gate also pushes batch size the wrong way, since an expensive approval encourages bundling.
A suite with a few percent flake rate destroys the pipeline as a control, gradually enough that nobody notices the moment it happens. Engineers learn that red usually means re-run, so they re-run, and a real regression is eventually dismissed the same way. The dashboard is green and the gate filters nothing. Quarantine a flaky test out of the blocking path and fix it as its own work item, because a check that is routinely overridden is worse than no check: it costs time and produces false confidence.
The Pipeline Is Itself a Production System
The pipeline holds credentials that can write to production, and it runs code from your repository with those credentials attached. That makes it a high-value target, and it means a pipeline outage is an outage of your ability to ship a fix.
Production Implications
Bound the time to first signal. Pick a target for commit to fast-check result, ten minutes or so, and treat exceeding it as a defect in the pipeline. Parallelize and shard rather than letting the fast path erode.
Address artifacts by digest, not by tag. A mutable tag can be repointed, reintroducing the ambiguity that building once was meant to remove. Promotion is a reference to an immutable digest.
Configuration travels separately from the artifact. The same bytes run in staging and production, so anything environment-specific is injected at deploy time. An artifact with configuration baked in cannot be promoted, only rebuilt.
Pin dependencies and record provenance. A lockfile and a pinned base image make a build reproducible enough that today's artifact is explicable tomorrow. Recording source commit, dependency versions, and builder identity is what lets you answer, during an incident, whether a vulnerable library is running anywhere.
Credentials should be short-lived, scoped, and issued per job. A static deploy key is a permanent production write capability sitting in a system that executes code from every branch. Prefer credentials issued per run and scoped to one environment, with production credentials unavailable to runs triggered from unmerged branches.
Pipeline definitions are reviewed code. They are as privileged as production configuration and should change through the same review, in the same repository.
The deploy step is pluggable and the pipeline stops at the handoff. Whether the rollout is progressive or all at once is a property of the service. The pipeline owns handing over the right artifact and recording what came back.
Structure the answer as stages, then artifact, then gates, then time budget. Most candidates list the stages competently and stop. Two things separate a strong answer: stating that the artifact is built once and promoted by digest, because a rebuild invalidates every prior test; and giving an explicit feedback-time target plus the ordering principle that produces it. Close on gating by naming one gate you would keep and one you would remove.
Tradeoffs and Decision Framework
| Dimension | Speed-weighted pipeline | Safety-weighted pipeline |
|---|---|---|
| Commit to production | Minutes to an hour | Hours to days |
| Blocking validation | Fast checks only | Full suite plus scans |
| Gate on production | Automated policy | Human approval |
| Typical batch size | One change | Many, bundled per approval |
| Failure discovered by | Production monitoring | Pre-production testing |
| Recovery expectation | Roll back or forward fast | Avoid the bad deploy at all |
| Dominant cost | Occasional user-visible defect | Slower recovery, larger batches |
The framework in four questions. What does a bad change actually cost here, since a payments ledger and an internal dashboard do not deserve the same gauntlet? How fast can you detect and reverse a bad deploy, because rollback capability is what buys the right to move quickly? Which of your gates could name the change it would stop? And is the slow validation slow because it is thorough or because nobody has parallelized it, which is the common case and is fixable rather than a tradeoff?
Default to automated gates, a fast blocking path, and thorough validation that runs against the promoted artifact without blocking every commit. Add human approval only where the approver holds information the pipeline cannot have. Leave that default when a change is genuinely irreversible, such as a destructive migration, where no amount of rollback speed helps.
Common Mistakes
Rebuilding per environment. Staging and production run different binaries, and every test result refers to an artifact that no longer exists.
Promoting by mutable tag. A tag that can be repointed reintroduces the ambiguity that building once removed.
Everything in one blocking stage. A forty-minute wall in front of every commit pushes engineers toward larger batches, the opposite of what the pipeline was for.
Tolerated flaky tests. Once re-running red is normal, the suite has stopped being a gate and become a tax.
Approval gates nobody can fail. An approver with no information the pipeline lacks adds latency and diffuses ownership without adding control.
Long-lived static deploy credentials. A permanent production write capability living in a system that runs code from every branch is the most attractive target you own.
Treating the pipeline as unowned infrastructure. It is on the critical path for shipping fixes, so its availability matters during incidents, not just on ordinary days.
Design a CI/CD pipeline that takes code from commit to production. What stages does it have, where do you gate, and how do you balance speed against safety?