The Pipeline Is a Feedback System

A team asks you to make their pipeline faster. It takes 47 minutes. You look at it and every stage is doing something reasonable, nothing is obviously wasteful, and you cannot find the part to delete.

---

The Problem at Scale

The reason that pipeline is hard to fix is that nobody has said what it is for. Asked directly, most teams answer that it builds and deploys the code, which is true and useless, because it describes the mechanism rather than the purpose.

A pipeline exists to answer one question: is this change safe to ship. Everything it produces along the way, the compiled binaries, the images, the test reports, exists to support that answer. The artifacts are a by-product. The output is a verdict.

Once you accept that, the governing metric follows. If the product is an answer, the thing to optimise is how long the answer takes to arrive, because an answer that arrives after the engineer has moved on to something else has already lost most of its value.

---

How It Works

Feedback latency, defined precisely

Not the wall clock of the pipeline. The time from pushing a change to the arrival of the first signal that would change what you do next.

The distinction matters because pipelines have several signals and they are not equally valuable:

First signal. Lint, type check, compile, unit tests. The earliest point at which the pipeline can say no. If this takes 90 seconds, an engineer will wait for it.

Full signal. Every check the pipeline runs. The point at which the change is provably eligible to merge.

Deployable. The artifact exists, is signed, and is ready for delivery to take over.

A pipeline optimised for the third number and ignoring the first is the common failure. It is efficient and it feels terrible, because the engineer sits for 20 minutes to find out about a missing import.

The attention threshold

Feedback latency matters because human attention is not continuous.

Under roughly ten minutes, an engineer waits. They keep the change in their head, they read the result, they fix it, they push again. The loop stays closed.

Past that, they start something else. The result arrives when they are somewhere else, and now the fix requires reloading context they have already discarded. The cost of a failure is not the pipeline time, it is the reload, and the reload cost is a step function rather than a gradual one.

This is why the difference between an 8 minute pipeline and a 25 minute pipeline is far larger than the ratio suggests, and why shaving 25 minutes down to 20 often changes nothing at all.

The batch size loop

Slow feedback does not stay a latency problem. It becomes a correctness problem through a loop that closes on itself.

slow pipeline
  -> engineers push less often, batching more into each push
    -> each run contains more independent changes
      -> a failure is harder to attribute to a cause
        -> debugging is slower, so people avoid pushing
          -> even larger batches
            -> even slower and less attributable

This is the mechanism behind most pipelines that are described as terrible without anyone being able to say which part is bad. No single stage is at fault. The system has settled into a state where every change is large, every failure is ambiguous, and everybody has adapted around it.

Merge queues, from Module 5, are one answer to the batching end of this loop. Making the pipeline fast is the other, and it is the one that treats the cause.

Trust as a component of latency

A signal you do not believe has infinite latency, because it does not change what you do.

A pipeline where 20 percent of failures are flake is not a pipeline with a flake problem. It is a pipeline whose red result carries almost no information, because the rational response to a failure is to re-run it rather than to investigate. That is Module 3, and it belongs here too, because trust is not separate from speed: it is a term in the same equation.

The practical version: measure the time to a signal that is acted upon. A red result that gets retried did not deliver a signal.

---

Building and Operating It

Instrument the three numbers separately, because they have different fixes.

# For a merged pull request, the three timestamps that matter.
# Everything in this lesson is derived from these.
gh run list --branch "$BRANCH" --json \
  createdAt,startedAt,updatedAt,conclusion,name --limit 50

per change, measured from the push:
  t_first    first stage that can fail        target < 5 min
  t_full     all required checks complete     target < 20 min
  t_ready    artifact built and publishable   whatever it costs

  t_first is the one that decides whether people wait.

Then measure whether the signal was believed:

# Runs that were re-run without a code change. Each one is a signal
# that was produced and not trusted, which is latency you are paying
# for and not receiving.
gh run list --json databaseId,headSha,conclusion,attempt --limit 200 \
  | jq '[.[] | select(.attempt > 1)] | length'

---

Tradeoffs and Decision Framework

If you optimise	You get	You lose
Total pipeline duration	Lower compute cost per run	Nothing about how it feels to use
Time to first signal	Engineers stay in the loop	Some duplicated work across stages
Trust in the signal	Failures get investigated	Time spent on flake instead of features
Throughput of runs	More changes processed per hour	Usually latency, since queues grow

Three questions frame any pipeline conversation. What is the first thing that can tell me no, and how long does it take, because that number sets the working experience. Is a red result believed, since an untrusted signal has no value regardless of when it arrives. And what is the batch size, which tells you whether the system has already adapted around its own slowness.

Default: measure time to first signal at the ninetieth percentile on pull requests, target under five minutes, and treat any re-run without a code change as a defect rather than as normal operation.

---

Failure Modes and Common Mistakes

Optimising the wrong number. Total duration is the number that gets reported and the first signal is the number that is felt.

Averaging across branches. Runs on main are not runs anybody is waiting for, and including them hides the number that matters.

Treating flake as separate from speed. An untrusted signal has not arrived, so trust belongs in the latency conversation rather than beside it.

Adding stages without asking what signal they produce. A stage that cannot fail is overhead, and a stage that only fails spuriously is worse than overhead.

Accepting the batch size the pipeline produced. Large changes are usually a symptom of slow feedback rather than a preference, and they make every subsequent failure harder to attribute.