What GitOps Actually Is
A team tells you they do GitOps. Their manifests live in Git, a pipeline runs
kubectl applyon merge to main, and deployments are reproducible. It is a genuinely good setup. It is also not GitOps, and the difference is not pedantry: it is the difference between a system that deploys and a system that stays deployed.
The Concept Explained
GitOps has a real definition. The OpenGitOps project publishes four principles, and a system either satisfies them or it does not:
- Declarative. The desired state is expressed declaratively.
- Versioned and immutable. That state is stored so it is immutable, versioned, and retains a complete history.
- Pulled automatically. Software agents pull the desired state from the source on their own.
- Continuously reconciled. Agents continuously observe actual state and work to apply the desired state.
Most teams that say "we do GitOps" satisfy the first two comfortably and neither of the last two. Manifests in Git plus a pipeline that applies them gives you declarative and versioned. It gives you nothing that pulls, and nothing that reconciles.
Here is the one-question test. Someone runs kubectl scale deployment/api --replicas=10 directly against production. What happens?
If the answer is "nothing, until somebody notices," you have version-controlled deployment, which is good and is not GitOps. If the answer is "within a few minutes it goes back to what Git says, and an alert fires," you have GitOps. Continuous reconciliation is the property that separates the two, and it is the one people skip.
How It Works
Git as source of truth, not Git as storage
The phrase "Git is the source of truth" gets repeated so often that it stops meaning anything. It has a precise meaning worth recovering.
Git as storage: the manifests happen to live in Git. A pipeline reads them and pushes them at the cluster. The cluster's actual state and the repository can diverge freely afterwards, and nothing detects it. Git records what you intended at deploy time.
Git as source of truth: the repository is continuously compared against the cluster, and disagreement is a defect the system works to remove. Git describes what the cluster is, and if that is ever untrue, something is wrong and will be corrected or reported.
That distinction is why the fourth principle carries most of the value. Without it, your repository is documentation that gradually becomes fiction.
Why each principle earns its place
Declarative means you describe the end state, not the steps to reach it. replicas: 3 rather than "scale up by one, three times." This is what makes the state comparable: you cannot diff a procedure, but you can diff a description. Every property below depends on it.
Versioned and immutable gives you three things at once, and they are the ones auditors ask about: every change has an author, a timestamp, and a review; any prior state can be named exactly; and history cannot be quietly rewritten. "Who changed the production memory limit and when" becomes git log rather than an investigation.
Pulled automatically moves the agent inside the cluster. Nothing outside needs credentials to the cluster's API, because nobody outside is pushing anything. This is the security argument, and Lesson 1.2 is entirely about it.
Continuously reconciled is what makes the system self-correcting rather than merely repeatable. A pipeline is a one-shot event: it runs, it succeeds, it ends, and whatever happens to the cluster afterwards is not its concern. A reconciler is a loop with no end that keeps asking whether reality matches the declaration.
Where CI stops and CD begins
GitOps forces a boundary that most pipelines blur, and drawing it clearly is the first structural change teams make.
CI: source code -> build -> test -> push image -> commit new tag to the config repo
│
(CI stops here)
│
CD: agent in cluster <- pulls config repo <- reconciles <- appliesCI produces artifacts and updates a declaration. It never touches the cluster. CD is a controller inside the cluster that notices the declaration changed and converges toward it.
The immediate consequence: your CI system no longer needs cluster credentials. It needs push access to a Git repository, which is a much smaller thing to lose.
The clearest signal that a team has internalised GitOps is that their CI pipeline has no kubectl, no kubeconfig, and no cluster credentials at all. Its final step is a commit. If a pipeline still needs cluster access for "just the deploy step," the boundary has not moved yet.
What you get, concretely
Beyond the philosophy, four practical properties follow directly:
- A complete audit trail for free. Every production change is a reviewed commit. Not a policy anyone has to enforce, a structural consequence.
- Disaster recovery becomes a bootstrap. Rebuild an empty cluster, point an agent at the repository, and it converges toward the declared state. Lesson 8.2 covers what this does and does not restore.
- Drift becomes visible. Manual changes stop being invisible; they become a diff that shows up on a dashboard.
- Rollback is a revert. Usually. Lesson 1.4 is about the cases where it very much is not.
In Production
Reality is a spectrum, not a binary, and being honest about where you sit is more useful than claiming the label.
Level 0. Manifests in a repo, humans run kubectl apply. Declarative only.
Level 1. Pipeline applies on merge. Declarative and versioned. This is where most teams are, and it is where most teams believe they are already done.
Level 2. An agent pulls and reconciles, self-heal off. Drift is detected and reported, humans decide. A good deliberate resting point, especially early, because you learn what actually drifts before you let a controller act on it.
Level 3. Reconciled with self-heal on. The cluster actively converges. This is full GitOps, and it means accepting that manual changes will be reverted, which is a cultural change more than a technical one.
Most organisations run a mix, and that is fine when it is chosen rather than accidental: level 3 for application workloads, level 2 for cluster infrastructure where a surprise revert is expensive.
A platform team turned on automated sync with self-heal for every Application on the same afternoon, including the ingress controller. That evening an engineer applied an urgent annotation directly to the ingress to work around a certificate issue, went to dinner, and came back to a production outage. Argo CD had reverted the annotation within three minutes, exactly as instructed. Nobody had done anything wrong, and nobody had agreed on what self-heal meant for emergency changes. The technical fix was minutes of configuration. The real fix was deciding, in advance, which parts of the system are allowed to be touched by hand and what you do instead when they are not.
Tradeoffs and Decision Framework
Reconciliation latency against control. A push pipeline applies the moment it runs. A puller notices on its next cycle, three minutes by default, or immediately with a webhook. You trade a little immediacy for a system that keeps holding the state rather than setting it once.
Self-heal against operator freedom. Self-heal guarantees the cluster matches Git and removes the ability to make a quick manual change. The correct answer is not "always on," it is "on, with an agreed process for the cases where you need to bypass it, and a way to see when someone did."
Everything in Git against practical exceptions. The purist position is that all state is declared. In practice secrets, some cluster-scoped resources, and anything with a bootstrap dependency need special handling (Lesson 1.4). Decide the exceptions deliberately and write them down, because the alternative is discovering them one incident at a time.
Git as a workflow against Git as a bottleneck. Requiring a reviewed commit for every change is excellent governance and real friction. Teams that skip the friction with an emergency kubectl path, and never reconcile it afterwards, end up with the worst of both: the process cost and the drift.
Common Mistakes
Calling a kubectl apply pipeline GitOps. It satisfies two of four principles and misses the two that provide self-correction.
Adopting the tooling without the reconciliation. Installing Argo CD and running every Application with manual sync gives you a nicer dashboard for a push workflow.
Treating drift alerts as noise. Drift means reality and your declared state disagree. If that alert is routinely ignored, the repository is no longer the source of truth and everything built on that assumption, including disaster recovery, is unsound.
Keeping cluster credentials in CI "just in case." The security benefit comes from those credentials not existing outside the cluster. Retaining them keeps the entire attack path you were removing.
Committing secrets to make everything declarative. Encoded is not encrypted. Module 5 covers the actual options.
Turning self-heal on everywhere without agreeing what it means. It is a policy decision about how humans work, not a checkbox, and it will revert somebody's emergency fix.