The Decisions You Are Not Being Given
Nobody is going to hand you a staff-scope problem. The problems at that scope are the ones with no obvious owner, which is exactly why they are still unsolved.
The Problem
Over five quarters your organization writes four postmortems, from three different teams, and every one of them has the same sentence buried in the timeline: outbound calls to a third-party API showed a p99 latency of almost exactly five seconds.
That number is a fingerprint. It is the glibc resolver timeout, and it means a UDP DNS query was dropped rather than answered slowly. The mechanism is well understood and the Kubernetes Debugging for SREs course walks through it properly: glibc fires the A and AAAA queries in parallel from one socket, both packets race to insert a conntrack entry for the same tuple on the way through DNAT, the kernel drops one, and nothing retries for five seconds.
Underneath it is a volume problem that makes the race far more likely. The kubelet writes a /etc/resolv.conf with four search domains and ndots: 5. Any hostname with fewer than five dots gets tried against every search domain before it is tried as written, so resolving a two-dot external hostname costs four NXDOMAIN round trips for A and four for AAAA before the real answer, and your CoreDNS pair is serving 46,000 queries per second of which 71 percent are NXDOMAIN.
Four teams hit it. Four teams fixed it.
The payments team added dnsConfig with ndots set to 1 on their deployments. The search team rewrote every external hostname in their configuration as a fully qualified name with a trailing dot. The data team set dnsPolicy: Default to bypass cluster DNS entirely, which broke their in-cluster service discovery, so they hardcoded the ClusterIPs of the four services they call. And the platform team, seeing CoreDNS CPU-saturated, scaled it from 2 replicas to 8 and put an HPA in front of it.
Every one of those is a defensible local decision made by a competent engineer under incident pressure. Three of them work. Together they are four incompatible resolution behaviors in one cluster, one of which is a hardcoded ClusterIP that will produce a total outage for the data team the first time somebody recreates a Service, and none of which touched the search path that generates 71 percent of the load.
Nobody made a bad decision here. The bad outcome came from a decision nobody made. What the cluster's DNS resolution contract is, what ndots should be, whether external names are written fully qualified, and whether the conntrack race is mitigated at the node, is a real decision with a real answer. It is not in anybody's job description, because the pod spec belongs to the app teams, the CoreDNS Deployment belongs to the platform team, and --cluster-dns plus the resolver template belong to whoever built the node image eighteen months ago and has since left.
The decisions with the highest scores on the four multipliers systematically end up unowned, and the reason is structural rather than cultural. Ownership in an engineering organization is assigned by component, and the expensive decisions are the ones whose consequences span components. The DNS resolution contract spans three ownership boundaries, so it belongs to none of them, and it stays unowned no matter how many postmortems it generates, because every postmortem is written by a team that owns only their own third of it. Look for the decision that would require three owners to agree, and you have found the one nobody is making.
How It Works
Unowned decisions announce themselves in three ways, and all three are things you can go and check today rather than intuitions you have to develop.
The first is the recurring incident with rotating owners. Search your incident tracker by cause instead of by service, which almost nobody does because the tracker is organized by service. Four postmortems from three teams sharing a mechanism is not four incidents; it is one unowned decision producing incidents on a schedule.
The second is the same workaround appearing independently in more than two repositories. This one is literally greppable. Search every deployment manifest in the organization for dnsConfig, or for dnsPolicy: Default, or for a hardcoded address in the 10.96.0.0/12 range, and count the distinct teams. Two teams solving the same problem privately is a coincidence. Four is a missing decision, and the grep takes ten minutes.
The third is the decision that has already been made implicitly by whoever moved first. The data team's hardcoded ClusterIPs are now the de facto answer to "how does this service reach that one," and nobody approved it, reviewed it, or knows it exists outside that team. An unowned decision does not stay unmade. It gets made by the person under the most time pressure, in the narrowest possible scope, and then it becomes precedent.
Where the DNS decision lives, and who owns each piece of it
Hover components for details
Taking the decision is not the dramatic part, and it is worth being specific about how small the act actually is. You write down what the contract should be: NodeLocal DNSCache as a DaemonSet so pod queries hit a local listener and reach CoreDNS over TCP, which removes the conntrack race entirely; ndots set to 2 as the cluster default; external hostnames written fully qualified as a documented rule; and the four existing workarounds removed once the first two are in. Then you send it to the platform lead and the three team leads with one sentence at the top: I am going to take this unless it is already yours.
That sentence is the whole technique, and it is what separates filling a vacuum from stepping on somebody. It is cheap for anyone to reclaim, it is explicit rather than a fait accompli, and it converts an ambiguous situation into one where silence is consent. In practice the reply you get is relief, because the DNS contract has been generating incidents for three of the four recipients and none of them believed it was theirs.
An engineer noticed that three teams had independently written retry-with-backoff wrappers around calls to the internal auth service, each with different budgets, and that the wrappers were the reason a fifteen-second auth degradation had turned into a nine-minute cascading failure the previous month. Nobody owned the retry contract; the auth team owned the service, each caller owned its client. They wrote a two-page document defining the retry budget, jitter, and circuit-breaking behavior expected of any caller, sent it with the sentence about taking it unless it was already someone else, and got no objection in a week. Eight months later that document was the thing three separate design reviews referenced when someone proposed a new caller. The document was two pages and took a day. The engineer did not write any of the client code, and the decision was more consequential than any code they shipped that year.
There is a harder case, which is when the decision genuinely does belong to someone and they are getting it wrong. Scaling CoreDNS to 8 replicas was the platform team's call to make, it was inside their scope, and it treated a symptom. Taking that decision away from them is overstepping, and it will cost you far more than the DNS fix is worth.
The move there is different: you supply the artifact and let them decide. Bring the query breakdown showing 71 percent NXDOMAIN, the four divergent workarounds with their repository paths, and the specific claim that replica count does not touch the cause. That is a decision made legible rather than a decision taken, and it is the higher-leverage act in this situation because the fix has to live in their component either way. Lesson 4.2 is about writing artifacts that move decisions you do not own, and lesson 5.4 covers what to do when they still say no.
Applying It
Run the three searches this week
Search the incident tracker by mechanism rather than by service, over the last four quarters. Grep every deployment manifest in the organization for the same workaround appearing in more than two teams. And list the decisions that would require three named owners to agree, because that list is almost exactly the list of decisions nobody is making.
For each candidate, apply the gate from lesson 1.2. An unowned decision that reverses in an afternoon is not worth the political cost of claiming it; leave it to whoever moves first, because whoever moves first can be corrected cheaply. The ones worth taking are unowned and score 4 or higher on cost of reversal, which is why they have been quietly generating incidents for five quarters.
Use the sentence
"I am going to take this unless it is already yours" is genuinely the mechanism, and the reason it works is that it is cheap to refuse. It gives the real owner a costless way to reclaim the decision, it gives everyone else a costless way to consent by doing nothing, and it means that if you proceed you can point at the moment you asked.
Send it to everyone who owns a component the decision touches, not just to your manager. In the DNS case that is four people. Give it a week. The failure mode is sending it to one person, because one person cannot tell you whether it is unowned; they can only tell you it is not theirs.
Write the contract, not the fix
The artifact that claims an unowned decision is a statement of what the behavior should be, not a pull request that implements it. The two pages defining ndots, fully qualified names, and the NodeLocal DNSCache mitigation are the thing that persists and the thing that gets cited in reviews eight months later. The DaemonSet is a day of work that anyone can do.
Taking an unowned decision by shipping the change and telling people afterwards works exactly once and costs more than it returns. It reads as competence to you and as territorial behavior to the three people whose components you touched, and the cost is that the next unowned decision, which is larger, will be defended against you. The order matters more than the content: ask, wait a week, then implement. If a week is genuinely unaffordable because the incident is live, ship the mitigation and open the contract question separately, and be honest that you are doing two different things.
Tradeoffs and Decision Framework
| Situation | Who actually owns it | Correct move | What it costs you if you get it wrong |
|---|---|---|---|
| Decision spans three components, no named owner | Nobody | Write the contract, send the sentence, wait a week | Almost nothing; the sentence is cheap to refuse |
| Owner exists, is slower than you, is not wrong | Them | Offer to draft it for them and let them own the outcome | Looks like impatience, and it is |
| Owner exists and is treating a symptom | Them | Supply the artifact and the specific claim, let them decide | Taking it instead buys a DNS fix and spends a working relationship |
| Owner left the company, component unclaimed | Nobody, formally | Claim it explicitly with your manager in the loop | Ambiguity persists and the next incident has the same shape |
| Already decided implicitly by whoever moved first | The team under the most time pressure | Surface the precedent before proposing anything | Reopening it without naming it feels like an attack on that team |
| Genuinely political, two directors disagree | Neither, in practice | This is lesson 2.5, not this lesson | Substantial; do not treat a political vacuum as a technical one |
Three questions settle whether to take it. Can you name a specific person who would be surprised or annoyed to learn you had made this decision, and if you can, the answer is that it is theirs and this is a different problem? Does the decision gate as staff scope on cost of reversal, or is it cheap enough that letting whoever moves first decide is genuinely fine? And has it already been decided implicitly, in which case your first job is making the existing decision visible rather than proposing a new one?
Default: if it is unowned, expensive to reverse, and you can send the sentence to everyone it touches, take it. The base rate strongly favors taking. Unowned expensive decisions are the highest-value work available to a senior engineer and the supply is large, because organizations assign ownership by component and the expensive decisions are the ones that span components.
Common Mistakes
Waiting to be assigned a staff-scope problem. By construction these problems have no owner, which means there is no one to assign them. The DNS contract had generated four postmortems and there was still nobody whose job it was to fix it.
Confusing an unowned decision with an unowned component. Nobody owning the DNS contract is an opportunity. Nobody owning a service is a staffing problem, and adopting an orphaned service is how you become the bottleneck described in lesson 6.2.
Shipping first and asking afterwards. It works once, it reads as territorial to everyone whose component you touched, and it makes the next and larger unowned decision harder to take. Ask, wait a week, then implement.
Sending the sentence to one person. One person can only tell you that it is not theirs. Send it to every owner the decision touches, which in the DNS case is four people and in most cases is three to five.
Taking a decision away from someone who is merely slower than you. Being slower is not the same as being wrong, and a working relationship with the platform lead is worth considerably more than getting NodeLocal DNSCache deployed six weeks earlier.
Treating a political vacuum as a technical one. When the decision is unmade because two directors disagree, the sentence does not help and claiming it puts you in the middle of something that is not about DNS at all. Lesson 2.5 covers that case specifically.
Claiming decisions faster than you can hold them. Every contract you write is something you are now expected to maintain, arbitrate, and defend in design reviews. Two or three at a time is sustainable and eight is how a person becomes the thing every decision waits on.
Writing the fix instead of the contract. The DaemonSet is a day of work anybody can do. The two pages defining what resolution behavior the cluster guarantees are what gets cited in a design review eight months later, and they are the part that scores on teams constrained.
You find that four teams have independently worked around cluster DNS behavior in four incompatible ways, and one of those workarounds hardcodes ClusterIPs and will cause an outage the next time a Service is recreated. The platform team owns CoreDNS and has responded by scaling it from 2 replicas to 8, which does not address the 71 percent NXDOMAIN load. What is the highest-value move?