Scope of Consequence, Not Depth of Skill
You are better at this than the staff engineer on your team. You have been for a while. Nobody is arguing with you about that, and it has not moved you an inch.
The Problem
Your platform team runs eleven kubeadm clusters on EC2, about 1,400 nodes between them, with seven engineers. Two pieces of work land in the same quarter.
The first one is yours. On the production cluster, etcd_disk_wal_fsync_duration_seconds p99 climbs to 780ms roughly every four hours across all three etcd members. etcd_server_leader_changes_seen_total moves three or four times a day. API server write latency p99 hits six seconds during the spikes, controllers stall behind it, and Argo CD sync operations start timing out. Nobody can reproduce it on demand, and it has been happening for two months.
It takes you three weeks. The answer is three things stacked. The etcd volumes are gp3 sitting at the 3,000 IOPS baseline. The snapshot cron fires on all three members at the same minute. And --auto-compaction-retention was never set, so the MVCC history had grown to where each compaction was doing real work against the same volume the WAL was trying to fsync to. You stagger the snapshots twenty minutes apart, set hourly auto-compaction, and move etcd to io2 at 8,000 provisioned IOPS. p99 fsync drops from 780ms to 3ms. Zero leader elections in the following ninety days.
That is excellent work. It required knowing that etcd is sensitive to fsync latency rather than to CPU, that a snapshot is a read amplification event landing on the write device, and that compaction and defragmentation are different operations with different costs. Nobody else on that team could have done it, which is why your manager brings it up in your review.
The second piece of work takes forty minutes. Another engineer, in a review of the cluster provisioning Terraform module, changes two defaults. New clusters get controlPlaneEndpoint set to a DNS name in front of an NLB rather than the private IP of the first control plane node. And the PKI tarball written at bootstrap goes to a bucket in a separate AWS account, under a bucket policy that denies s3:DeleteObject to every role the platform team holds.
Neither change required any skill you do not have. The first is a string in a Terraform variable. The second is twelve lines of IAM policy.
Now look at what each of them actually decided.
Your etcd fix repaired one cluster, and you rolled the same settings to the other ten over the next month. Its effects last until something changes the storage layer underneath it, which realistically means an instance family migration or a move to a managed control plane: eighteen months, maybe two years. No other team had to do anything. And every part of it reverses in an afternoon, because a volume type and a compaction flag are both one-line changes.
The controlPlaneEndpoint default decided something you cannot undo cheaply. That address is written into the API server serving certificate SANs, into /etc/kubernetes/kubelet.conf on every node, into admin.conf, controller-manager.conf and scheduler.conf, and into the kubeconfig on every engineer's laptop. Changing it on a live cluster means reissuing the serving certificate with new SANs, rewriting the kubeconfigs on every node, and rolling every kubelet. It has its own lesson in the Kubernetes Failure Recovery course, and it is a coordinated migration rather than a configuration change. Across 1,400 nodes it is weeks of work touching every team that owns a node pool. Set correctly at bootstrap it costs nothing, and it means the control plane can move without the fleet noticing.
The PKI bucket policy decided whether one careless or compromised platform-team credential can destroy the only copy of the material that no procedure regenerates.
Neither of those decisions was hard. Both of them were larger than three weeks of very good debugging. That gap is the subject of this course.
The thing being measured at staff scope is not how difficult the work was for you. It is how much depends on the decision, over what horizon, how many other people it constrains, and what it costs to undo. Depth of skill is a rate multiplier on work you have already been handed. Being twice as fast at etcd forensics halves the time you spend on etcd forensics; it does not change how many etcd incidents exist, because that number is set by the size of the fleet and the quality of the defaults. There is a ceiling on the value of being the best debugger in the building, and the ceiling is the volume of problems that exist. There is no comparable ceiling on choosing a default, because a default is applied by everyone, forever, without anyone deciding again.
Two honest qualifications. This is a model of scope, not a promotion mechanism; organizations vary enormously and promotion is partly circumstance. And in a small enough shop the model inverts, because when you run two clusters and forty services the constraint genuinely is how many hard problems your best engineer resolves per month. The defaults are set, the fleet is not growing, and consequence has nowhere to accumulate. That situation is real and this course will not pretend otherwise.
How It Works
Score a piece of work by consequence rather than by difficulty and the two pieces above separate immediately. The etcd fix scores high on skill and low on everything else. The Terraform default scores near zero on skill and high on the things that persist.
The same quarter, scored two ways
The etcd fsync investigation
Three weeks, high skill, contained consequence
The controlPlaneEndpoint default
Forty minutes, no skill, permanent consequence
The second column is worse to be evaluated on and better to be doing, and that tension is where most of the frustration in this course comes from. High-skill work is legible: two numbers in a sentence and everyone understands something was fixed. High-consequence work is frequently illegible, because its output is an absence. The control plane never had to move, the PKI was never deleted, cluster twelve came up correct. Lesson 6.4 is entirely about doing this work and holding none of the credit for it.
The diagram is not saying that skill and consequence compete. The controlPlaneEndpoint default was only available to someone who had once rebuilt a control plane and watched what happened to the kubelets. Depth of skill is the input to consequence, not its rival. What changes between senior and staff is not how much you know but which question you point it at: the same understanding of the etcd write path that diagnoses an fsync spike in week three decides, in an hour, that etcd storage is a specified tier so no future cluster can be built on baseline gp3 at all.
That second act is the staff-scope version of the same work, it takes a day, and almost nobody does it, because the incident is closed and the graph is green and there is another incident waiting.
A platform team of nine spent most of a quarter on a genuinely hard problem: intermittent five-second DNS timeouts traced to a conntrack race on parallel UDP DNAT insertions, resolved with NodeLocal DNSCache and a single-request-reopen resolver option. Real forensics, real fix, celebrated internally. In the same quarter the same team stood up four new clusters, and because nobody had touched the provisioning module, every one of them got the default --service-cidr=10.96.0.0/12 that overlapped the corporate VPN supernet already routed into the VPC. The DNS work saved perhaps forty engineer-hours a year. The service CIDR overlap surfaced eight months later as unroutable traffic to an on-prem payments gateway, and because a service CIDR cannot be changed on a live cluster, the remedy was rebuilding two of the four clusters and migrating their workloads. The hard problem cost three weeks and bought a year. The trivial problem cost nobody any thought and bought a five-month migration.
Applying It
Audit the last quarter honestly
List every piece of work you did in the last three months, at a granularity of about eight to fifteen items. Against each one write four things: what breaks if this was wrong, how long the effects persist before something else retires them, how many other teams would have to change their own code to change your decision, and how many engineer-weeks it costs to reverse.
Do it with the numbers you actually have. "Weeks" is not an answer; "reissue serving certs plus roll 1,400 kubelets, three weeks with two teams coordinating" is an answer. The exercise fails when you let yourself be vague, because vagueness always scores in favor of whatever you spent the most time on.
The pattern most senior engineers find is not that their work scores low. It is that the two or three highest-scoring items took the least time and are not the ones they talked about in their review.
Find the decision inside the fix
Every incident you resolve contains a decision you could take instead, and the decision is almost always cheaper than the incident was.
The etcd investigation contains one. Having established that a control plane on baseline gp3 will eventually flap, the decision available to you is that the provisioning module refuses to build a cluster whose etcd volume provides less than a stated IOPS floor, and that the bootstrap check fails loudly rather than warns. That is a day of work. It costs less than the third week of the investigation did, and it applies to cluster twelve, which does not exist yet and will therefore never generate the incident.
The test for whether you found the decision: does it apply to a system that has not been built yet? If it only applies to systems that already exist, you found another fix.
Change what you report, not just what you do
"Reduced etcd WAL fsync p99 from 780ms to 3ms" is scored as skill, accurately, because that is what it was. "Etcd storage is now a specified tier in the provisioning module, so the failure cannot be reintroduced by the next cluster" is scored as scope. Both are true of the same quarter, only one usually gets written down, and it is not because the other one was secret.
The fastest way to raise your score on this framework is to make your decisions harder to reverse, and that is a genuinely destructive move dressed as scope. An engineer who introduces a custom CRD and a controller where a ConfigMap would have worked has manufactured a decision that eleven clusters and six teams now depend on, and they have done it by spending other people's reversal budget. Consequence you created is not the same as consequence you were exposed to. Lesson 2.3 splits these carefully and Module 6 is largely a catalogue of people who got this backwards.
Tradeoffs and Decision Framework
| The work | What breaks if wrong | Persists for | Teams that must change | Cost to reverse |
|---|---|---|---|---|
| Diagnosing the etcd fsync spike | Nothing new; the status quo continues | 18 months | 0 | An afternoon |
| Making etcd storage a tier in the provisioning module | Every future cluster inherits a flapping control plane | Life of the module | 0 now, all of them later | A day |
Choosing controlPlaneEndpoint at bootstrap | The control plane is pinned to one node address forever | Life of each cluster | Every team with a node pool | 3 weeks, fleetwide kubelet roll |
| Choosing the service CIDR at bootstrap | Overlaps with corporate routing become unfixable in place | Life of each cluster | Every team in the cluster | 4 to 6 months, cluster rebuild plus migration |
| Putting the PKI backup in a separate account | One bad credential destroys the only irreplaceable material | Life of the platform | 0 | Hours |
| Upgrading one cluster from 1.28 to 1.29 | One cluster's workloads, for one window | One release cycle | 1 to 3 | Rollback or reroll |
| Writing the etcd restore runbook | The next restore is improvised at 3am | Until the procedure changes | 0 | Trivial |
Three questions settle which row a piece of work is really in. First: if this is wrong, does anything break that is not already broken? A fix that returns a system to its intended state has a blast radius of zero, which is why fixes rarely score high no matter how hard they were. Second: would this decision apply to a system nobody has built yet? That is the difference between the etcd fix and the etcd tier, and it is usually one day of extra work. Third: to change this later, does anyone outside your team have to change their own code, their own manifests, or their own on-call procedure? If yes, you are holding a staff-scope decision whether or not anybody handed it to you.
Default: assume the highest-consequence thing you touched this quarter is something you did in under an hour without discussing it. That is the honest base rate. The three-week investigation is visible, satisfying, and almost never the answer, because a fix restores a state and a default creates one.
Common Mistakes
Reading this as a claim that debugging does not matter. It matters enormously, the fleet needs someone who can do it, and at small scale it is genuinely the binding constraint. The claim is narrower: skill saturates as a signal because it multiplies work you were already given, and consequence does not, because it changes what work exists.
Scoring difficulty and calling it consequence. The three-week investigation feels ten times more significant than the forty-minute Terraform review, and difficulty is the only axis on which that is true. Every question in the framework above deliberately asks about the system rather than about you.
Confusing surface area with consequence. Something that touches every cluster and reverses in an afternoon is an operations problem. Something that touches one cluster and cannot be reversed without a rebuild is a staff-scope problem. Reversal cost is doing more work in this model than raw reach, and the next lesson explains why.
Waiting to be given the high-consequence work. The controlPlaneEndpoint default was not assigned to anyone. It appeared as a line in a Terraform module review, was noticed by the person who happened to be reviewing, and could have gone unnoticed forever. Lesson 1.4 is about that specific pattern.
Assuming the model transfers cleanly to your organization. Some companies genuinely promote the best debugger, some promote the person nearest the revenue-critical system, and some do not have a coherent policy at all. Lesson 1.3 is about reading yours from evidence rather than from the ladder document, and it is the only defence against optimizing for the wrong thing for two years.
Treating illegibility as somebody else's problem. Correct irreversible decisions produce no incident to point at, and the absence of an incident does not report itself. Write down which decisions you took or be evaluated entirely on the fixes.
Doing the staff-scope version of the work and then not shipping it. The etcd tier is worth nothing as a design document and the full amount as a merged change with a failing bootstrap check behind it, because the value came entirely from nobody having to decide again.
Assuming a high score on consequence means you were right. The framework measures how much depends on a decision, not whether it was a good one. That is exactly how a platform ends up with an unroutable service CIDR.
Over one quarter you (a) spend three weeks diagnosing an etcd fsync problem that had been degrading one cluster for two months, and (b) spend one day adding a check to the cluster provisioning module that fails the build if the etcd volume provides under 5,000 IOPS. Your team runs eleven clusters and provisions roughly four more each year. Which of these is the staff-scope work, and why?