Where the Minutes Actually Go

The dashboard says the average build takes 12 minutes. The engineers say it takes most of an hour. Both are reading real numbers, and the gap between them is the whole problem.

---

The Problem at Scale

A pipeline duration is not one number, it is five numbers added together, and they have almost nothing in common. They have different causes, different owners, different fixes, and wildly different sizes.

  queue     waiting for compute to become available
  setup     checkout, dependency restore, image pull, tool install
  work      compiling, testing, packaging: the part you meant to do
  human     waiting for a review or an approval
  rework    re-runs, flake retries, and push-fix-push cycles

Almost every CI dashboard reports the sum, or worse, reports only work plus setup and calls it the build time. That is why the dashboard and the engineers disagree, and neither is lying.

You cannot fix an aggregate. Halving the wrong bucket is invisible; halving the right one is transformative; and the only way to tell them apart is to measure them separately before touching anything.

---

How It Works

A real decomposition

Here is one change, measured end to end, on a pipeline whose dashboard reports 12 minutes.

push                                              t+0
  queue, waiting for a runner                   4m 12s
  checkout and dependency restore               2m 48s
  compile                                       3m 30s
  unit tests                                    2m 06s
  integration tests (parallel branch)           9m 18s
  ---------------------------------------------------
  first signal (lint, parallel)                 5m 40s
  all required checks green                    21m 54s   <- not 12
  waiting for a reviewer                        6h 12m   <- the real number
  merge queue                                  24m 30s
  deploy                                       11m 00s
  ---------------------------------------------------
  commit to production                        ~7h 10m

Three findings fall out of this and none of them are visible in the aggregate.

Queue plus setup is seven minutes before any useful work begins, which is 32 percent of the time to green. That is a fleet and caching problem, not a test problem.

The dashboard reports 12 minutes and the required checks take 22. The dashboard is measuring the job, not the change.

The human wait is larger than everything else combined by a factor of seventeen. Any conversation about making this pipeline faster that does not mention review latency is a conversation about the small half.

That last point is uncomfortable because review latency is not the platform team's to fix, which is exactly why it stays unmeasured. Measure it anyway. It changes what the organisation chooses to work on.

The five buckets and their fixes

Queue. Jobs waiting for a runner. Caused by fleet capacity, concurrency limits, or a scale-up that is slower than the arrival rate. Fixed in Module 4. Grows nonlinearly: a fleet at 85 percent utilisation queues occasionally, and at 95 percent it queues constantly.

Setup. Everything before the work. Checkout, dependency restore, container image pull, toolchain install. Caused by cold caches, large images and repository size. Fixed in Modules 2 and 4. Its distinctive property is that it is paid per job, so splitting the pipeline multiplies it.

Work. The compiling and testing. Fixed by parallelism, test selection and caching, in Modules 2 and 3. This is the bucket everybody assumes is the problem, and it usually is not the largest one.

Human. Review and approval. Not a compute problem at all. Reduced by smaller changes, clearer ownership and review expectations, and smaller changes are themselves a consequence of a faster pipeline, from lesson one.

Rework. Runs that had to happen again. A flake retry, a fix for something the pipeline could have told you sooner, a re-run of a whole pipeline for one failed job. Fixed in Modules 3 and 7, and often the easiest large win because it is pure waste rather than a tradeoff.

Why the buckets are usually measured wrong

Three specific traps.

Duration from job start. Excludes queue, which is the bucket most likely to be growing.

Averaging over all runs. Runs on the default branch are not runs anyone waits for, and they are usually the majority, so they dominate the average and hide the pull request experience.

Measuring the pipeline instead of the change. A change that took four pushes to go green consumed four pipelines. The engineer experienced one long wait, not four short ones, and the per-run number reports the short one.

The unit that matters is the change, from first push to merge, including every re-run it took.

---

Building and Operating It

Compute queue time explicitly, since nothing gives it to you.

# createdAt is when the run was requested, startedAt when a runner
# picked it up. The difference is queue, and it is the number that
# is missing from every default dashboard.
gh run list --limit 200 --json createdAt,startedAt,conclusion,event \
  | jq -r '.[] | select(.event=="pull_request")
      | ((.startedAt|fromdate) - (.createdAt|fromdate)) as $q
      | "\($q)s queue"' \
  | sort -n | tail -20

Measure per change rather than per run.

# All runs for one head SHA lineage on a pull request branch. Total
# elapsed from the first push to the last green is what the engineer
# actually experienced.
gh run list --branch "$BRANCH" --json createdAt,updatedAt,conclusion \
  | jq -r 'sort_by(.createdAt) |
      "first push: \(.[0].createdAt)   last result: \(.[-1].updatedAt)   runs: \(length)"'

And do not skip the human bucket.

# Time from ready-for-review to first review. Usually the largest
# single number in the whole system, and usually nobody owns it.
gh pr list --state merged --limit 100 \
  --json createdAt,reviews,mergedAt,number \
  | jq -r '.[] | select(.reviews|length > 0)
      | ((.reviews[0].submittedAt|fromdate) - (.createdAt|fromdate))/3600
      | "\(. * 100 | round / 100) h to first review"'

---

Tradeoffs and Decision Framework

Bucket	Typical share	Fix lives in	Tractability
Queue	5 to 30 percent	Module 4, fleet capacity	High, costs money
Setup	10 to 35 percent	Modules 2 and 4, caching and images	High, costs effort
Work	30 to 60 percent	Modules 2 and 3, parallelism and selection	Medium
Human	Often the largest	Not a platform fix	Low, organisational
Rework	5 to 40 percent	Modules 3 and 7, flake and reliability	High, pure waste

Attack in order of size multiplied by tractability, not in order of what is interesting. Rework first whenever it is material, because it is waste rather than a tradeoff and removing it costs nobody anything. Setup and queue next, since both are mechanical. Work third, since it is the one that requires real engineering. Human always gets reported, even though you cannot fix it, because the organisation cannot prioritise what it cannot see.

Default: instrument the five buckets separately per change rather than per run, compute duration from creation rather than from job start, exclude the default branch from the developer-facing percentile, and report the human bucket even though it is not yours.

---

Failure Modes and Common Mistakes

Reporting one duration number. It is a sum of five unrelated things and it hides which one is growing.

Starting the clock at job start. Queue time disappears exactly when the fleet is under pressure and you most need to see it.

Averaging pull requests together with the default branch. The default branch dominates and nobody is waiting for it.

Measuring runs instead of changes. A change that needed four pushes shows up as four fast runs and one long afternoon.

Ignoring the human bucket because it is not a platform problem. It is frequently larger than everything you can fix, and leaving it unmeasured guarantees it stays that way.

Optimising the interesting bucket. The test suite is the fun problem and the re-run rate is usually the bigger one.