The Difference Between Senior and Staff Is Not Skill. It Is Scope of Consequence.

The best engineer on the team is often not the staff engineer, and that confuses people for years.

Plenty of senior engineers are better at Kubernetes than the staff engineer sitting beside them. They debug faster, they know the internals more deeply, and they write better code. They also keep getting the same feedback at review time, phrased vaguely enough to be useless: broaden your impact, think bigger picture, operate at more scope.

Nobody says what is actually being measured, usually because they cannot articulate it either.

Here is what is being measured. Not how hard the problem is. How much depends on the decision, over what horizon, how many other people it constrains, and how expensive it is to take back.

That is a property of the decision, not of you, which is why it is teachable and why the ladder feels arbitrary until somebody names it.

Two decisions on the same Tuesday#

The fleet is 64 nodes, 8 H100 SXM each, 512 GPUs on a 400 Gb/s InfiniBand fabric, shared by fourteen teams running a mix of multi-week training jobs and always-on inference replicas. Two decisions land in the same week.

The first is the NCCL transport configuration baked into the training base image: which HCAs collectives are allowed to use, which interface NCCL falls back to, and whether the fabric is actually being reached. Get it wrong and NCCL silently falls back to TCP over the front-end NIC. Nothing errors. Jobs run. All-reduce bandwidth on a 32-GPU job measures 18 GB/s instead of 340 GB/s, and fleet-wide model FLOPs utilization sits at 31 percent where it should be 52. Finding that requires knowing the collective layer well enough to distrust a job that looks healthy, and fixing it nearly doubles the useful output of a fleet that cost about eighteen million dollars.

The second takes an afternoon and produces no measurement at all. It is what a team submits when it wants GPUs. Either they write resources: nvidia.com/gpu: 8 on a Deployment and the default scheduler places pods as capacity appears, or they submit a job object with a gang size, minMember: 32, into a queue that admits the whole thing or nothing.

The first is the one that gets discussed for two weeks and written up afterwards. It is harder, it is more interesting, and it is worth more money this quarter.

The second decides what the platform is.

The four multipliers#

Score both decisions on four axes, one to five, and the reason becomes arithmetic rather than opinion.

Decision: NCCL transport defaults in the training base image
  blast radius        4   every training job silently degrades; fleet MFU 52% -> 31%
  time horizon        2   ends at the next fabric change or NCCL major, ~12 months
  teams constrained   0   any team overrides it with an env var in its own pod spec
  cost of reversal    1   revert the image tag; hours, one person
  verdict: operational. Ship it, measure it, revert if wrong.

Decision: the unit of the scheduling contract (per-pod GPU request vs per-job gang)
  blast radius        4   partial allocation deadlock strands ~96 H100s at peak, ~$240/hr
  time horizon        5   quota, priority, preemption, chargeback all expressed in this unit
  teams constrained   5   all 14: submission path, CI templates, dashboards, on-call runbooks
  cost of reversal    5   rewrite 14 submission paths, drain in-flight jobs: 2 to 3 quarters
  verdict: staff scope. Written decision, named owner, explicit revisit trigger.

Both decisions affect the entire fleet. Only one of them can be taken back.

Each axis has a specific failure of measurement attached to it, and getting the definition slightly wrong is what makes the framework produce comfortable answers instead of useful ones.

Blast radius is measured at the worst moment. The scheduling unit scores 4 rather than 2 because partial allocation only strands GPUs when the fleet is near full, which is exactly when stranded GPUs cost the most and nobody has spare capacity to work around it. Infrastructure failures correlate with load, so an average-case blast radius describes a situation you will never be in.

Time horizon is measured by naming the event that ends it. Not "we will revisit this next year," which is aspirational and never happens, but "this decision dies when we move off InfiniBand." If you cannot name a concrete terminating event, the horizon is longer than you think and should be scored a point higher.

Teams constrained counts consent, not exposure. All fourteen teams are affected by the NCCL defaults, and the axis still scores zero, because any one of them can set the env var in their own pod spec and be free of your decision this afternoon. Nobody has to agree with you for you to change your mind. This axis is the one that turns your reversal cost into a coordination cost, and coordination cost grows faster than headcount.

Cost of reversal is measured in work and calendar time, not in feasibility. Almost everything is technically possible. Score the engineer-weeks and then score the calendar, because a change that is four engineer-weeks and requires fourteen teams to schedule a window is not a month, it is two quarters.

Do not average the four#

This is where most frameworks like this one go wrong, and it is worth being precise about.

Averaging ranks the NCCL decision above half the real staff-scope decisions on the platform, because three high scores and one low one beat two mid scores and two high ones. The composition is not a mean. It is a gate.

Size is the larger of blast radius and teams constrained. Reversal cost is the gate. A decision is staff scope when reversal cost is 4 or higher on its own, or when reversal cost is 3 and teams constrained is 3 or higher. Below that it is operational, no matter how big it looks, and it should be decided quickly by whoever is closest to it.

Reversal cost gates rather than contributes because the other three axes describe how expensive it is to be wrong once, and reversal cost describes how many times you are allowed to be wrong.

Deliberation only creates value where feedback is unavailable. Where you find out in a week and undo it in an afternoon, an hour of debate is a worse instrument than the experiment you could have run instead. Where you find out in eight months and undo it over three quarters, deliberation is the only instrument you have.

And there is a second reason, the one people miss. Blast radius, horizon and teams constrained are all estimable up front from the design. Reversal cost is the only axis that grows silently after the decision is made, as manifests, dashboards, runbooks and habits accumulate on top of it. The per-pod GPU request cost two engineer-days to reverse in month one and two quarters in month fourteen, and nothing about the decision changed in between.

Now four places where this changes the answer, each one a real decision with a competent senior response and a different staff response.

1. The design review: the premise is the most valuable thing to be wrong about#

You have thirty minutes and a fourteen-page document you did not write, called "Multi-Region Failover for the Payments API." Page 1 cites the March 12 incident: payments unavailable for 41 minutes, roughly $380,000 of failed transactions. Pages 6 through 11 propose active-passive across two regions. Page 12 states RTO 15 minutes and RPO "near zero." Pages 13 and 14: two engineers for two quarters, plus $34,000 a month of standby infrastructure.

You have read it once and found nine things wrong.

Nothing in the senior review is wrong and none of it is optional. The shallow health check is the single most common defect in failover designs and the one that turns a rehearsed plan into a manual page at 3am.

The difference is what each answer treats as reviewable. The senior review takes the document's premise as given and reviews the design against it, which is what a review is normally understood to be. The staff review treats the premise as the most valuable thing in the document to be wrong about, because it is the cheapest thing to change and the most expensive thing to get wrong.

Put numbers on it. The nine findings, all fixed, improve a design costing 26 engineer-weeks plus $408,000 a year of standby capacity. The page 1 finding is worth all of it, because the actual remedy for a bad-deploy outage is canary analysis with automated rollback on the payments deploy path, which is about three engineer-weeks. The deployment and rollback patterns that address exactly this were never in the document, because the alternative was never framed.

2. The migration that is correct in principle and wrong in practice#

Six clusters, 214 nodes, Calico in IPIP mode, running NetworkPolicy for twenty-two teams across forty-one namespaces. The proposal is to migrate all six to Cilium.

The pain is real and measured. There are 380 NetworkPolicy objects, because a shared Helm chart applies a default-deny plus one allow per dependency to every service, and policy programming on a node with the full set takes 6 to 9 seconds after a restart. During that window a rescheduled pod is reachable by nothing, which has produced two incidents where a node reboot turned into four minutes of 502s. Nobody has L7 visibility. IPIP costs 20 bytes of MTU, which surfaces once a quarter as somebody's gRPC stream stalling.

Cilium fixes all three. The proposal is careful: eighteen months, 1.5 engineers, cluster by cluster, with a translation script and a rollback plan.

Your platform team has six engineers. The organisation has reorganised twice in three years. Median tenure on this team is 2.1 years.

The staff move is two questions the plan does not contain.

Does this organisation finish eighteen-month projects? Eighteen months is longer than the median tenure on this team and spans, at the historical rate, one reorganisation. The engineer who wrote the plan will probably not be the one who ends it. That is not a reason to refuse. It is a reason to require that value arrives per cluster rather than at completion.

Is the cited pain caused by the thing being replaced? The 6 to 9 second programming time is a function of rule count, and the rule count is a function of that Helm chart. Collapse it to namespace-scoped policies with a tier model and the 380 objects become about 90, and measured programming time drops under 2 seconds. That is three engineer-weeks, and it removes the incident class that justifies the migration.

A migration that takes eighteen months to fix something a policy refactor fixes in three weeks is not a wrong destination. It is a wrong instrument.

Three conditions flip this back, and they occur often. If the current version is out of support in nine months, the migration is not discretionary and the analysis becomes pure sequencing. If each cluster's completion is independently valuable, completion probability matters far less. And if the dependency graph genuinely is that dense, the policy set is irreducible, the cheap alternative evaporates, and the migration returns with a stronger case than it started with.

3. The half-migrated state is where migrations live permanently#

Migration proposals estimate the work and describe the destination. They rarely price the state the project will actually stop in, and that state varies enormously.

Moving between container registries leaves two systems that do not interact. The half state is nearly free and you can sit in it indefinitely. A CNI migration leaves two policy models, where a NetworkPolicy means something slightly different depending on which node a pod landed on, and on-call needs both mental models. That state is tolerable for six weeks and dangerous for six months. Several of the recovery scenarios in the Kubernetes Failure Recovery course exist because somebody stopped halfway.

Score your specific half state rather than assuming it resembles the last one.

Then notice where value lands. In most migrations, everything that depends on uniformity — retiring the old expertise, one runbook, one security model, one thing to certify — arrives only at completion. And the interrupt reliably arrives before completion, because the clusters or services that hurt most were done first, so the remainder has no forcing function. What is left is merely inconsistent, and inconsistent never wins a prioritisation meeting against a launch.

Restructuring a migration so that stopping early is acceptable is usually worth more than accelerating it. Make each unit independently valuable. Better still, apply the new thing to new clusters only and let the existing ones age out on their natural rebuild cycle, which converts an eighteen-month project into a policy about how the next cluster is built. A policy does not need an owner to survive a reorganisation.

4. The trap that feels exactly like importance#

You own the GitOps repository. CODEOWNERS names you on /clusters/** and /apps/**, covering every Argo CD Application, every ApplicationSet generator and every Kustomize overlay for fourteen teams across six clusters.

Last quarter: 61 pull requests a month, median 2.5 days to your first review, roughly eight hours a week. On 9 percent of PRs you caught something real, and two of those would have been incidents.

Here is the part nobody counts.

Four teams have stopped putting certain changes through the repo. There is a Slack thread you were not in where an engineer explains that for anything urgent they apply directly with kubectl and open the PR afterwards, or sometimes not. A drift report found 12 percent of live objects do not match git. That number is the value of GitOps, expressed as a percentage destroyed. And three teams copied a sync-wave pattern from a fourth team's directory, including its bug, because reading an example took ten minutes and asking you took three days.

You are the most informed person about this repository in the company. You are also the reason it does not work.

The obvious fix is queue management: fixed review hours, a published schedule, a second reviewer on CODEOWNERS. All correct, all worth doing, and it takes you from 61 PRs a month to maybe 120 while postponing the real problem by three quarters.

The thing to notice is that the 9 percent you catch is not judgment. It is seven rules — sync waves violating a CRD dependency, selectors matching production when the annotation says staging, prune: true on a path with an open restructure, patches whose target no longer resolves, missing resource limits, Applications targeting a cluster the team does not own, unpinned chart versions. Every one is expressible as a policy check in CI, and a policy check runs in 40 seconds at 03:00 on a Saturday, which no version of you does.

Encode the seven. Move CODEOWNERS to per-team ownership. Keep a personal gate on exactly one thing: cluster-scoped resources and the app-of-apps root, where reversal cost is genuinely high.

Then accept the uncomfortable part. The catch rate drops, probably from 9 percent to about 7, because two of the seven are caught badly by a rule and there is always an eighth nobody has seen yet. You are trading two points of catch rate for 12 percent drift going to near zero and fourteen teams getting a four-hour median.

The question underneath it: is your attention the resource to optimise, or the constraint to remove? The senior answer optimises the resource and roughly doubles throughput. The staff answer removes the constraint from the 91 percent of changes that never needed it, which raises throughput by a factor that does not have you in it at all.

Where to start#

Not with your tasks. With your decisions, which is a much shorter list, usually three to six at any moment.

For each one, write the four numbers and, next to each, the sentence that justifies it. The sentence does the work: "teams constrained: 5, because Helm charts in fourteen repos name this field" is a score. "teams constrained: 5, because it is platform-wide" is a feeling.

Then apply the gate. Most senior engineers doing this honestly find that their highest-scoring decision is not the one absorbing their attention, and that the one absorbing their attention scores high on blast radius and low on everything else. That is not a coincidence: high blast radius with low reversal cost is the exact profile of an incident, and incidents are what gets scheduled for you.

Scope of consequence is not seniority you are granted. It is a property of the decisions you are holding, and you can measure it this afternoon.

---

This post is the opening argument of The Staff Engineer's Judgment, a course that works through real infrastructure decisions with the competent senior answer, then the staff answer, then what the second one saw that the first did not. Thirty-one lessons across scope, deciding under ambiguity, technical judgment, multiplying other engineers, operating across the org, and the four traps that cap a career.

Module 1 is free. Four lessons, no signup wall.