Desired State and Reconciliation

At 02:00 an engineer scales a deployment from 3 replicas to 12 to stop a paging storm, and goes back to bed. By 02:04 it is running 3 replicas again and paging resumes. Nobody touched it. Understanding why, and deciding whether that was the right outcome, is what this lesson is for.

---

The Concept Explained

A reconciler is a loop that never ends:

forever:
    desired = read the declaration          (Git)
    actual  = observe the real world        (the cluster)
    if desired != actual:
        act to close the gap
    wait

That is the whole idea, and it is not new. It is exactly how Kubernetes itself works: the Deployment controller does not "create pods when you run kubectl." It continuously compares how many pods should exist against how many do, and acts on the difference. Delete a pod by hand and one appears, not because your deletion triggered a handler, but because the next comparison found a shortfall.

GitOps applies that same loop one level up. Kubernetes reconciles the cluster toward the objects in etcd. Argo CD reconciles the objects in etcd toward the manifests in Git.

---

How It Works

Refresh and sync are two different operations

In Argo CD, the loop splits into two steps that people conflate, and separating them explains most confusing behavior:

Refresh answers "are we different?" It fetches the repository, generates manifests (running Helm or Kustomize if needed), compares the result against live cluster state, and updates the Application's sync status. Refresh never changes the cluster.

Sync answers "make it the same." It applies the generated manifests to the cluster.

refresh    Git ──> render manifests ──> diff against live state ──> status: Synced | OutOfSync
sync       apply the rendered manifests to the cluster

An Application with automated sync disabled still refreshes: it will tell you it is OutOfSync forever and do nothing about it. That is a perfectly valid mode, and it is exactly the "level 2" resting point from Lesson 1.1: detect drift, let humans decide.

What actually triggers a pass

Three things, and knowing all three prevents a lot of confusion:

• Polling. By default the controller reconciles every 3 minutes (timeout.reconciliation in the argocd-cm ConfigMap, default 180s). This is the safety net that makes the system level-triggered rather than event-driven.
• Webhooks. A push notification from your Git provider triggers an immediate refresh, which is how you get seconds instead of minutes without polling aggressively.
• Cluster events. The controller watches cluster resources, so a change to a managed object can trigger comparison much faster than the polling interval.

Polling is the guarantee; webhooks are the optimisation. A team that relies purely on webhooks and disables polling has quietly rebuilt an edge-triggered system, and will eventually lose a delivery and never converge.

Drift, and where it actually comes from

Drift is any difference between declared and live state. The interesting part is that most of it is not humans running kubectl:

Source	Example	Should it be corrected?
Manual change	An engineer edits a Deployment during an incident	Usually yes, after the incident
Another controller	HPA changes replicas	No. The HPA owns that field
Mutating webhook	A sidecar injector adds containers	No. Exclude it from comparison
Defaulting	The API server fills in unspecified fields	No. Not real drift
Deletion	A namespace cleanup removes a resource	Yes, this is what self-heal is for
Failed apply	A partially applied sync	Yes, and this is the recovery case

The rows that say "no" are why naive self-heal causes trouble, and the HPA case is the one everybody hits.

The fight nobody wins

Two controllers that both believe they own the same field will fight forever:

HPA:      load is high, set replicas = 12
Argo CD:  Git says replicas = 3, this is drift, set replicas = 3
HPA:      load is still high, set replicas = 12
Argo CD:  drift again, set replicas = 3
          ...every reconciliation, indefinitely

Both are behaving exactly as designed. The mistake is upstream of both: the manifest declares a field that a different controller is authoritative for.

There are two correct resolutions, and choosing between them is a real design decision:

1. Stop declaring the field. Omit replicas from the manifest and let the HPA own it entirely.
2. Declare it and exclude it from comparison. Keep it as a starting value and tell Argo CD to ignore differences on that path, so it is used at creation and never enforced afterwards.

Lesson 4.3 covers the mechanics of ignoreDifferences and the diffing rules. The principle to carry now: for every field, exactly one system is authoritative. Drift is not always an error. Sometimes it is a sign that two things believe they are in charge.

Convergence is eventual, and can fail

The loop is a best effort, not a guarantee. A sync can fail because a manifest is invalid, a CRD does not exist yet, a webhook rejects the object, a namespace is terminating, or a resource is stuck on a finalizer.

When that happens the Application sits OutOfSync, the controller keeps trying, and the system does not converge. This is the failure mode that matters operationally, because it is quiet: nothing crashes, no pods restart, and production simply stops receiving changes while every dashboard stays green.

Which is why the sync failure alert (Lesson 8.1) is the single most valuable alert in a GitOps setup. A stalled reconciler looks exactly like a healthy one from the outside.

---

In Production

• Reconciliation is continuous work, and it costs. Every pass fetches, renders, and diffs. At thousands of Applications this is real CPU in the repo server and real load on Git. Lesson 2.5 covers where that time goes and 8.4 covers tuning it.
• Shorter intervals are not free. Dropping to 30 seconds multiplies fetch and render load sixfold for a benefit webhooks already provide. Use webhooks for speed and leave polling as the backstop.
• Self-heal is a policy choice, not a default. It guarantees convergence and removes manual intervention as an option. Decide it per Application: application workloads yes, cluster-critical infrastructure often no.
• Drift signal is only useful if it is quiet. If every Application shows permanent drift from defaulted fields and injected sidecars, nobody will notice the one that matters. Tuning comparison rules is what makes the signal usable.
• The loop reveals problems the pipeline hid. Teams adopting GitOps often discover their manifests were never actually applied cleanly. The reconciler does not fail differently, it fails visibly and repeatedly, which is uncomfortable and correct.

---

Tradeoffs and Decision Framework

Detect against correct. Detection alone (self-heal off) keeps humans in the loop and lets drift persist. Correction guarantees the declared state and will revert an emergency change without asking. Start with detection, move to correction once you know what legitimately drifts.

Polling frequency against load. Faster convergence, more Git and CPU pressure. Webhooks give the responsiveness without the cost, so tune polling as the safety net rather than as the mechanism.

Declaring a field against ceding it. Every field is owned by exactly one system. Declaring something another controller manages guarantees a fight, so decide ownership explicitly rather than discovering it in a reconciliation loop.

Strict comparison against practical comparison. Comparing everything catches real drift and drowns you in noise from defaults and injected fields. Excluding too much hides genuine problems. This is a tuning exercise, not a one-time setting.

---

Common Mistakes

Assuming sync is instant. Default polling is three minutes. Without a webhook, a commit is not live the moment it merges.

Disabling polling because webhooks work. One dropped webhook and that Application never converges. Polling is the property that makes the system level-triggered.

Declaring replicas alongside an HPA. The most common self-inflicted reconciliation loop in production.

Treating all drift as error. Defaulted fields, injected sidecars, and controller-owned values are expected differences, not incidents.

Ignoring persistent OutOfSync. A permanently unsynced Application means delivery has silently stopped for that workload.

No alert on sync failure. A stalled reconciler is invisible: nothing is broken, nothing is changing, and everything looks fine.

Turning on self-heal without telling anyone. Somebody's 02:00 fix will be reverted at 02:03, and the confusion costs more than the drift did.