Why Pipelines Rot
Nobody made this pipeline slow. Every step was added by somebody sensible, for a real reason, after a real incident. It is now 47 minutes and everyone agrees that is a problem, and no one can name the step to delete.
The Problem at Scale
Pipelines get worse over time in a way that most infrastructure does not. A cluster that is left alone stays roughly as it was. A pipeline that is left alone gets slower every month, and the reason is that it is not left alone: it is touched constantly, and every touch adds.
The forces are asymmetric, which is what makes this a ratchet rather than a drift.
Adding a check is attributable and safe. Somebody investigated an incident, found a gap, and closed it. That is good work and it is visible.
Removing a check is attributable and dangerous. If the thing it caught ever recurs, the removal is the named cause. Nobody is ever thanked for the incident that did not happen because the pipeline was fast.
So checks accumulate at the rate incidents occur, and are removed at approximately zero. Each addition costs 40 seconds and nobody objects to 40 seconds. Sixty of them is 40 minutes.
The ratchet is an incentive problem rather than a technical one, so a technical fix will not hold. The only mechanism that reliably works is a stated latency budget: the pipeline has a target, additions that breach it must be paid for by removing or accelerating something else, and the tradeoff is made explicitly by somebody with the standing to make it. Without a budget, every individual decision is correct and the aggregate is indefensible.
How It Works
The six forces
Accumulating checks. The ratchet above. The dominant force in most organisations, and the only one that is purely social.
Test suite growth. Tests are added at roughly the rate features are, and deleted at roughly never. Suite runtime therefore grows with the codebase, on hardware that is not getting faster at the same rate. This is not a defect, it is the expected behaviour of a healthy project, which is why the fix is selection and sharding rather than restraint.
Cache decay. A cache key that was correct when written stops matching as the code moves. Nothing fails when this happens, the pipeline just gets slower, and because it degrades gradually there is no moment anyone can point at. Module 2 covers why hit rate must be a first class metric for exactly this reason.
Flake accumulation. Each individually tolerable flaky test lowers the probability that a run passes, and Module 3 shows how brutal that arithmetic gets. Retries mask it, retries cost time, and the trust erosion is what actually hurts.
Setup creep. Another tool installed, another dependency added, another package to restore. Paid once per job, and multiplied by however many jobs the graph has.
Ownership diffusion. A pipeline is everybody's dependency and nobody's product. It is modified by dozens of people who each own one step, and nobody owns the whole. Systems with no owner do not get faster.
Why the slow ones feel unfixable
Because they are locally optimal. Every step defends itself:
"can we drop the license scan?" -> it is required by legal
"can we cut integration tests?" -> that is the suite that
caught the outage in March
"can we skip the security scan on PRs?"-> it is a compliance control
"can we stop building all four arches?"-> a customer runs on arm
Every answer is true. None of them is an argument that the current pipeline is the right one, because none of them was ever weighed against the cost of the whole. A defence of a step is not a defence of the total.
This is what a budget changes. It stops asking whether a step is useful, which it always is, and starts asking whether it is worth more than the other thing that would have to leave.
The measurement that makes rot visible
Rot is a trend, so a single measurement cannot show it. Record from the start:
per week, on pull requests only:
p90 time to first signal
p90 time to all required checks
cache hit rate, per cache
re-run rate (runs re-executed without a code change)
count of required status checks
test count and total suite runtime
The last two are the ratchet made visible. A chart of required check count over 18 months is usually the most persuasive artifact in the whole conversation, because it goes up and to the right in a way nobody intended and nobody can defend.
Building and Operating It
Track the ratchet explicitly.
# The number that only ever goes up. Record it weekly; the trend is
# the argument.
gh api "repos/$REPO/branches/main/protection/required_status_checks" \
| jq '.contexts | length'Find the additions that are no longer earning their place.
# Checks that have not failed in 90 days. Not proof that a check is
# useless, since prevention is invisible, but it is the right list to
# review rather than reviewing nothing.
gh run list --limit 500 --json conclusion,name,createdAt \
| jq -r 'group_by(.name)[]
| {name: .[0].name,
runs: length,
failures: ([.[] | select(.conclusion=="failure")] | length)}
| select(.failures == 0)
| "\(.name): \(.runs) runs, 0 failures"'State the budget somewhere that has teeth.
# pipeline-budget.yml, reviewed quarterly by a named owner.
budget:
p90_time_to_first_signal: 5m
p90_time_to_required_green: 20m
max_required_checks: 12
policy:
# The rule that converts an unbounded ratchet into a negotiation.
- any addition that breaches a budget must be accompanied by a
removal or an acceleration in the same change
- checks added after an incident carry a review date of 6 months
- quarantined tests expire, they are not permanent (Module 3)Give every check added in the aftermath of an incident an explicit review date, recorded at the moment it is added rather than later. Incident-driven additions are the most defensible ones at the time and the least examined afterwards, because the incident that justified them is precisely what nobody wants to reopen. A review date makes the reexamination routine instead of adversarial, and it is essentially free to attach while the incident is still being written up.
Tradeoffs and Decision Framework
| Force | Nature | Countermeasure |
|---|---|---|
| Accumulating checks | Social, asymmetric incentives | A budget with a named owner |
| Test growth | Healthy and unavoidable | Selection and sharding, Module 3 |
| Cache decay | Silent, gradual | Hit rate as a monitored metric, Module 2 |
| Flake accumulation | Compounding, trust destroying | Detection and expiring quarantine, Module 3 |
| Setup creep | Per job, multiplied by graph width | Prebuilt images, Module 4 |
| No ownership | Structural | Name an owner, publish the budget |
Two questions decide whether a pipeline will rot. Is there a stated budget, since without one every addition is individually justified and the total is nobody's decision. And is there an owner, because a budget with no one accountable for it is a document.
Default: publish a latency budget with a named owner, require every breaching addition to be paid for in the same change, attach review dates to incident-driven checks, and chart the required check count where people can see it.
Failure Modes and Common Mistakes
Arguing about individual steps. Every step wins that argument, which is why the argument is the wrong one.
Treating rot as a technical problem. It is an incentive structure, and refactoring the YAML does not change it.
No baseline. Rot is only visible as a trend, so a team that starts measuring during the complaint has one point and no history.
Permanent additions. A check added after an incident with no review date is a permanent addition by default.
Assuming a check that never fails is useless. It may be preventing the behaviour it checks for, so put it on a review list rather than deleting it on the strength of the counter.
Nobody owning the whole. Everyone owning their step is exactly the condition under which the total gets worse.
A pipeline has grown from 12 to 47 minutes over two years. Every stage was added deliberately, each one is defensible, and stakeholders reject every proposed removal with a legitimate reason. What is the most effective intervention?
Why do CI pipelines get slower over time, and what would you put in place to stop it?