Stages, DAGs and Gates

The pipeline has five stages and the slowest stage takes four minutes, so it should finish in about twenty. It takes thirty eight, and every stage looks idle for part of that time.

---

The Problem at Scale

Almost every CI interface draws a pipeline as a row of stages, left to right, each one starting when the previous finishes. That picture is a rendering choice. The execution model underneath is a directed acyclic graph of jobs with dependency edges, and the two are not the same thing.

The gap between them is where the missing minutes live. A stage boundary is a barrier: everything in stage N finishes before anything in stage N+1 starts. The dependency graph almost never requires that. It usually requires that one specific job waits for one specific other job, and the barrier serialises the rest for free.

Nobody adds those barriers deliberately. They arrive because the tool draws stages, so people design in stages, and the drawing becomes the architecture.

---

How It Works

Barriers versus edges

Consider a pipeline with a build, three test suites of very different lengths, and a package step that needs only the build.

STAGE MODEL (what the UI draws, and what people then build)

  build 3m  |  unit 2m  integration 9m  lint 1m  |  package 2m
            |<-------- barrier ------->|         |<- barrier ->|
  total: 3 + 9 + 2 = 14 minutes

DAG MODEL (what the dependencies actually require)

  build 3m ─┬─> unit 2m ──────┐
            ├─> lint 1m ──────┤
            ├─> integration 9m┤ (only merge gate needs these)
            └─> package 2m ───┘
  critical path: build 3m -> integration 9m = 12 minutes
  package finishes at minute 5, not minute 14

Two minutes recovered in a toy example. In a real pipeline with fifteen jobs and four barriers, the same effect routinely accounts for a third of the duration.

The diagnostic question for any barrier is blunt: which job in the later stage actually consumes an output of which job in the earlier one. If you cannot name the pair, the edge does not exist and the barrier is imaginary.

The cost of splitting

Edges are cheap and jobs are not. Every job pays a fixed cost before it does any work:

runner acquisition        10 to 60 s   (Module 4)
repository checkout        5 to 30 s
dependency restore        10 to 90 s
artifact download          variable, and often the surprise

Splitting one job into two adds one copy of all of that, plus the cost of moving any intermediate state between them. A build that uploads a 900 MB artifact so that three downstream jobs can each download it has bought parallelism and paid for four transfers of 900 MB.

This is the tension that makes graph design a real decision rather than a mechanical one. Fine-grained jobs shorten the critical path and lengthen every path, and there is a granularity past which splitting makes the pipeline slower. Module 2 does the arithmetic.

Gates, and what each one serialises

A gate is an edge you added for a reason other than data dependency. There are four kinds and they cost different things.

Required status checks. The merge is blocked until named checks pass. Cost: the merge waits for the slowest required check, so marking a 30 minute job required puts 30 minutes in front of every merge. The question to ask is not whether the check is useful but whether it must be answered before the merge rather than after.

Approvals. A human edge. Its duration is not measured in compute and is frequently the largest single number in the pipeline. It is also the one nobody instruments, which is why Module 1 lesson three measures it separately.

Environment protection. A gate on a deployment target: wait timers, reviewers, branch restrictions. These serialise deploys rather than builds, and they are usually correct.

Concurrency limits. Not a gate on correctness but on parallelism, and they behave like one. A concurrency group of one on a shared environment means every run past the first is queued, and queue time is indistinguishable from slowness to the person waiting.

Fan-in is where duration hides

The last job in a graph, the one that reports the overall verdict, has an edge from everything. Its start time is the maximum of all its predecessors, so it inherits the worst case rather than the typical case.

The consequence is that improving the median job does nothing. If nineteen jobs finish in four minutes and one finishes in fourteen, the pipeline takes fourteen minutes and speeding up the nineteen changes nothing at all. This is unintuitive enough that teams routinely optimise the wrong nineteen.

---

Building and Operating It

Express the real dependencies rather than the drawn ones.

# Every edge here is a data dependency. There are no stages, and
# package does not wait for tests it does not consume.
jobs:
  build:
    runs-on: ubuntu-latest
    steps: [...]

  lint:
    needs: build          # needs the resolved dependencies only
  unit:
    needs: build
  integration:
    needs: build
  package:
    needs: build          # NOT needs: [unit, integration]
    # package produces an artifact; whether to ship it is the
    # merge gate's decision, not this job's.

  verdict:
    needs: [lint, unit, integration, package]
    # the fan-in. its duration is max(), so this is the job to watch.

Find the critical path from real run data rather than from the diagram:

# Per job start and end for one run. The longest chain through these
# is your wall clock; everything else has slack and is not worth
# optimising until the chain moves.
gh api "repos/$REPO/actions/runs/$RUN_ID/jobs" \
  | jq -r '.jobs[] | [.name, .started_at, .completed_at] | @tsv' \
  | sort -k2

# Which required checks gate the merge, and therefore sit in front of
# every change this repository accepts.
gh api "repos/$REPO/branches/main/protection/required_status_checks" \
  | jq -r '.contexts[]'

Then ask, per required check, whether it must be answered before the merge or whether it can run after and page somebody. Most organisations have at least one check that is required out of habit.

---

Tradeoffs and Decision Framework

Choice	Buys	Costs
Fewer, larger jobs	Less fixed overhead, no artifact transfer	Long critical path, coarse failure attribution
More, smaller jobs	Short critical path, precise failures	Fixed overhead per job, transfer volume
Stage barriers	A simple mental model and a tidy UI	Rounds up to the slowest job, repeatedly
A check made required	The failure cannot reach main	Its duration sits in front of every merge
Approval gates	A human decision point	Unbounded latency, usually unmeasured

Three questions size up any pipeline graph. What is the longest path, since that and nothing else is the duration. Which barriers are real edges, which you test by naming the producing and consuming jobs. And what is required versus merely useful, because required is a decision about repository throughput and not just about this change.

Default: model the pipeline as a DAG with only data dependencies as edges, keep the required set small and fast, and optimise the critical path rather than the slowest looking stage.

---

Failure Modes and Common Mistakes

Designing in stages because the UI draws stages. The barriers this creates are invisible and they compound.

needs: listing everything out of caution. Each unnecessary entry is a barrier with a nice syntax.

Splitting jobs until the overhead exceeds the work. Fixed cost per job is real and there is a point of diminishing return.

Passing large artifacts between many jobs. Upload once, download n times, and the transfer becomes the critical path.

Optimising jobs with slack. Anything not on the longest path contributes nothing until the path itself moves.

Making a check required without doing the merge arithmetic. It becomes a limit on how many changes the repository can accept per day.