All posts
GitOps

Your Manifests Are in Git. That Is Not GitOps.

Manifests in a repository and a pipeline that runs kubectl apply satisfies two of the four GitOps principles and misses the two that matter. There is a one-command test that tells you which side of the line you are on, and the answer changes how your delivery system fails.

By Sharon Sahadevan··9 min read

Here is a setup most teams would call GitOps, and it is a genuinely good one.

Manifests live in a repository. Changes go through pull requests. A pipeline runs on merge to main and applies them to the cluster. Deployments are reproducible, every change is reviewed, and the whole thing is version controlled.

It is also not GitOps, and the difference is not vocabulary. It changes how the system behaves when something goes wrong, where your production credentials live, and whether the repository still describes reality six months from now.

The one-command test#

Run this against production:

kubectl scale deployment/api --replicas=10

Now wait five minutes and look again.

If it is still at 10 replicas, you have version-controlled deployment. Good practice, real benefits, not GitOps.

If it went back to what the repository says, and something told you it happened, you have GitOps.

That is the whole distinction. Everything below is why it matters.

The four principles, and the two everyone skips#

GitOps has an actual definition. The OpenGitOps project publishes four principles, and a system either satisfies them or it does not:

  1. Declarative. The desired state is expressed declaratively.
  2. Versioned and immutable. That state is stored immutably, versioned, with complete history.
  3. Pulled automatically. Software agents pull the desired state from the source on their own.
  4. Continuously reconciled. Agents continuously observe actual state and work to apply the desired state.

Manifests in Git plus a pipeline gives you the first two comfortably. It gives you nothing that pulls, and nothing that reconciles.

The gap is not academic. Those last two principles are what make the system self-correcting rather than merely repeatable, and they are the ones that change your architecture.

Git as storage against Git as source of truth#

The phrase "Git is the source of truth" gets repeated until it stops meaning anything. It has a precise meaning worth recovering.

Git as storage: the manifests happen to live in a repository. A pipeline reads them and pushes them at the cluster. Afterwards, the cluster and the repository can diverge freely and nothing detects it. Git records what you intended at deploy time.

Git as source of truth: the repository is continuously compared against the cluster, and any disagreement is a defect the system works to remove. Git describes what the cluster is, and if that is ever untrue, something is wrong and will be corrected or reported.

Without the fourth principle, your repository is documentation that gradually becomes fiction. Every hotfix applied by hand at 2am, every kubectl edit during an incident, every change made by a controller nobody accounted for: each one widens the gap, and nothing measures it.

The question worth asking of your own estate is not "are our manifests in Git." It is "if production stopped matching the repository right now, how long before anyone found out?"

Edge-triggered against level-triggered#

There is a deeper reason a pipeline cannot do this job, and it is the same idea Kubernetes itself is built on.

A pipeline is edge-triggered. It responds to an event: a merge happened, run these steps. It executes, succeeds, and exits. Its relationship with the cluster ends at that moment. If the deployment is later changed, deleted, or partially applied, the pipeline neither knows nor cares. It already succeeded.

A reconciler is level-triggered. It compares desired state against actual state on every pass, forever. A missed event costs nothing, because the next comparison finds the same difference and acts on it.

push     commit -> apply -> done.  cluster drifts.  nothing notices.
pull     commit -> apply -> compare -> compare -> compare -> ...
                                  drift detected and corrected

This is exactly how the Deployment controller works. It does not create pods "when you run kubectl." It continuously compares how many pods should exist against how many do. Delete one by hand and another appears, not because your deletion fired a handler, but because the next comparison found a shortfall.

GitOps applies that same loop one level up. Kubernetes reconciles the cluster toward the objects in etcd. Argo CD reconciles the objects in etcd toward the manifests in Git.

The practical consequence: a pipeline can tell you a deployment succeeded. Only a reconciler can tell you the cluster is currently correct.

The part that should interest your security team#

The third principle, "pulled automatically," sounds like an implementation detail. It is the one with the largest blast radius.

In a push model, the deployer lives outside the cluster and reaches in, which means your CI system holds standing administrative credentials to production. It has to, because the deploy step runs kubectl or helm.

So ask the uncomfortable question: what is the blast radius of one compromised pipeline dependency? Not the pipeline you wrote, the dozens of actions and container images it pulls in to do its job.

Now do the arithmetic across an estate:

PUSH, 8 teams x 4 environments
  32 pipelines, each holding credentials to a cluster
  every credential needs rotation, scoping, storage, and audit
  every CI runner is a path to production
  compromise of the CI platform reaches everything

PULL, same estate
  32 agents, each holding read-only access to a repository
  zero standing cluster credentials outside the clusters
  compromise of CI produces a commit, which is reviewable and revertible

Pull does not make CI safe to compromise. It changes what a compromise gets you: the ability to propose a change to a repository, subject to branch protection and review, instead of direct authenticated access to the production API server.

This is the same shape as mounting the Docker socket into a build container. In both cases a pipeline is handed a credential far broader than its task requires, because the task was described as "deploy the app" rather than "hold permanent administrative access to the cluster." And the fix has the same shape too: stop needing the credential rather than protecting it better.

There is a network consequence as well. Push requires your CI system to reach the cluster's API server, which means opening inbound access to production from wherever your runners live. Pull inverts it: the agent makes an outbound connection to Git and nothing needs to reach the API server at all. For private clusters, on-premises estates, and strict security reviews, that difference often decides the design on its own.

Where CI stops and CD begins#

Adopting the pull model forces a boundary that most pipelines blur:

CI:  source -> build -> test -> push image -> commit new tag to the config repo
                                                          |
                                                    (CI stops here)
                                                          |
CD:  agent in cluster <- pulls config repo <- reconciles <- applies

CI produces artifacts and updates a declaration. It never touches the cluster. CD is a controller inside the cluster that notices the declaration changed and converges toward it.

The clearest signal that a team has actually made this shift: their CI pipeline has no kubeconfig at all. Its final step is a commit. If a pipeline still needs cluster access for "just the deploy step," the boundary has not moved yet.

What you honestly give up#

Pull is not strictly better, and pretending otherwise is how teams get surprised.

Immediacy and certainty. A push pipeline applies now and reports what happened. Anyone watching knows within seconds whether the deploy worked. With pull, the pipeline finishes at the commit and the deployment result surfaces somewhere else, on its own schedule. "The pipeline is green, is it live?" becomes a real question that needs a real answer, usually notifications and a dashboard you have to build.

Cross-system orchestration. Some deployments genuinely span systems: run a migration, deploy, invalidate a CDN, notify a partner. A pipeline sequences that trivially because it is a script. A reconciler has no natural way to express "and then, elsewhere."

Simplicity for small estates. One team, one cluster, a kubectl apply step: the operational cost of running and upgrading a reconciler may genuinely exceed the benefit. GitOps pays off with more clusters, more teams, and stricter audit requirements.

The adoption levels#

Most teams are somewhere on this ladder rather than at one end, and being honest about which rung is more useful than claiming the label:

Level 0. Manifests in a repo, humans run kubectl apply. Declarative only.

Level 1. A pipeline applies on merge. Declarative and versioned. This is where most teams are, and where most believe they are already finished.

Level 2. An agent pulls and reconciles, self-heal off. Drift is detected and reported, humans decide. A perfectly good deliberate resting point, especially early, because you learn what legitimately drifts before letting a controller act on it.

Level 3. Reconciled with self-heal on. The cluster actively converges. This is full GitOps, and it means accepting that manual changes get reverted, which is a cultural change more than a technical one.

That last part is not a footnote. Turning on self-heal without agreeing what it means for emergency changes is how a team discovers, at 02:04, that the fix someone applied at 02:00 has been reverted by a controller doing exactly what it was told.

So, the test#

Run the command. Wait five minutes.

If nothing happened, you have version-controlled deployment. That is worth having, and it is worth knowing what it does not give you: no drift detection, no self-correction, and a repository whose accuracy decays quietly from the day you stop looking.

If it snapped back and something told you why, you have the thing the four principles describe. The repository is not a record of intent any more. It is a description of reality, continuously proven.


This is the opening argument of GitOps with Argo CD, a course on running GitOps in production: the reconciliation model, repository and environment structure, sync mechanics and their failure modes, secrets, multi-cluster topologies and tenancy, progressive delivery with Argo Rollouts, and operating Argo CD as critical infrastructure. Module 1, which covers everything in this post plus what GitOps deliberately does not solve, is free. Related reading: Mounting the Docker Socket Is Root on Your Host for the same credential mistake in a build pipeline, and A Timeout Tells You Nothing for why retries and reconciliation both assume repeating an operation is safe.