What GitOps Does Not Solve
Six weeks after adopting GitOps, a bad release goes out. Someone reverts the commit, the reconciler applies it within three minutes, and the incident continues anyway. The deployment rolled back perfectly. The database schema did not.
The Concept Explained
The previous three lessons made the case for GitOps. This one draws its boundary, because the failure mode of a good idea is applying it where it does not fit.
GitOps reconciles declarative state that a controller can observe and change. That is a genuinely large surface, and it excludes more than people expect:
- Anything that cannot safely be written down in a repository
- Anything that is not idempotent to apply
- Anything whose current state the reconciler cannot see
- Anything where "make it match" is not a safe instruction
Every hard problem in this course lives in one of those four categories.
The load-bearing assumption of GitOps is that applying the declared state repeatedly is always safe. Reconciliation depends on it: the loop will apply the same thing over and over, forever. Any operation where that is untrue, a migration, a one-off job, a destructive change, does not belong in the reconciliation path, and forcing it there is how GitOps adoptions produce outages that push pipelines never did.
How It Works
Secrets: the thing you cannot commit
The first wall everyone hits. A Kubernetes Secret is base64-encoded, not encrypted, so committing one publishes it to everyone with repository read access, plus every fork, clone, CI cache, and backup, permanently, because Git history does not forget.
The options split into two families:
- Encrypt in Git. Sealed Secrets and SOPS. The ciphertext is committed, and something in the cluster decrypts it. Fully declarative, and rotation means a commit, and losing the decryption key is catastrophic.
- Reference from Git. External Secrets Operator. The repository holds a pointer to a secret store, never the value. Rotation happens outside Git entirely, at the cost of a runtime dependency on that store.
Module 5 works through all three properly. The point here is structural: the fourth GitOps principle cannot fully apply to secrets, because a reconciler comparing declared and actual state for a secret would have to read the secret to compare it. Every solution is a way of working around that.
Stateful data and migrations
Deployments are idempotent. Databases are not.
Deployment apply the same manifest 500 times -> identical outcome
DB migration run the same migration twice -> error, or corruptionA schema migration is a procedure, not a state. It has ordering, preconditions, and consequences that cannot be undone by applying an earlier version. Nothing about "make reality match this file" describes it.
Migrations therefore live at the edge of the model, usually as a PreSync hook (Lesson 4.2) so they run once before the new code, with the reconciler carefully not treating them as state to converge on. The uncomfortable part is that the hook runs inside a system built on the assumption of repeatability, and hooks can be retried.
This is the asymmetry that turns a routine rollback into an incident. Reverting a commit restores the previous manifests, so the old application version comes back. It does not un-run the migration, restore deleted columns, or recover rows a data backfill rewrote. Your application code is now the previous version and your data is not, which is a state that was never tested. Every rollback plan needs to say explicitly whether the schema change was backward compatible, and if it was not, what the actual recovery path is.
The durable practice is to make migrations expand and contract: add the new column, deploy code that writes both, migrate readers, then remove the old column in a later release. Each step is independently revertible. It is more work and it is what makes "revert the commit" a real rollback rather than half of one.
Revert is not always rollback
Beyond data, three more cases where reverting the declaration does not restore the previous world:
- Mutable image tags. If the manifest says
:latestor:v2, reverting the manifest may pull a different image than the one that ran before, because the tag moved. Digest pinning is what makes a revert deterministic. - External side effects. The release sent emails, published events, charged customers, called partner APIs. Reverting the deployment does not recall any of it.
- Resources deleted by the revert. If the change added a resource and the revert prunes it, that resource is destroyed. For a PersistentVolumeClaim, so is the data (Lesson 4.4).
The promotion problem
GitOps stores desired state. It does not tell you what should be promoted, when, or by whom.
"This version passed staging, put it in production" is a workflow question. Git gives you the mechanism for the change (a commit) and none of the process around it: no notion of a release candidate, no record of what was validated, no gate. Teams build that themselves out of directories, branches, overlays, image updaters, and pull request automation, and it is one of the least standardised parts of the whole ecosystem. Lesson 3.5 is entirely about the options.
A related gap: Git tells you what should be deployed, not what is deployed. Those differ whenever a sync is failing, pending, or partially applied. Answering "what is in production right now" means asking the cluster or the reconciler, not the repository. Teams that skip this build dashboards that confidently display the wrong answer.
Ordering across systems
A reconciler converges resources within its scope. It has no view of the world beyond it:
Terraform provisions a database <- not Argo CD
Argo CD deploys the app that needs it <- needs the database to exist firstSync waves (Lesson 4.2) order resources inside an Application. They cannot wait for a Terraform run, a DNS propagation, or a partner's change window. Cross-system sequencing needs either an orchestrator above both, or a design where each side tolerates the other being late, which is usually the better answer.
Bootstrap, and who deploys the deployer
A genuine chicken-and-egg problem: Argo CD manages your cluster's state, and something has to install Argo CD. It can manage itself afterwards, and the initial install cannot come from the thing being installed.
That first step is always imperative: a Helm install from a laptop, a Terraform module, a cluster bootstrap script. Which means every GitOps estate has at least one component outside GitOps, and pretending otherwise leads to disaster recovery plans with a missing first step. Lesson 8.2 makes this concrete, and 8.3 covers why Argo CD cannot safely deploy its own upgrade.
Operational actions that are not state
Some things you need to do in production are verbs, not nouns:
- Restart a deployment to clear a stuck connection pool
- Run a one-off backfill job
- Flush a cache
- Drain a node
- Scale up temporarily for a launch, then back down
None are desired state. Each is an action, and a reconciler models the world as nouns. GitOps needs an agreed answer for these, and the honest one is usually "we do them imperatively, and here is the audit trail and the process for reconciling afterwards."
In Production
What mature GitOps estates deliberately keep outside the reconciliation path:
- Secret values. References in Git, values in a secret store.
- Schema migrations. Hooks or a separate release process, with an explicit backward-compatibility policy.
- The bootstrap. Argo CD itself, installed by something else and documented as step one of DR.
- Emergency actions. A break-glass path, audited, with an agreed rule that changes made outside Git are reconciled back or reverted deliberately.
- Cluster provisioning. Usually Terraform or a managed control plane, with GitOps starting once the cluster exists.
- Data. Backups, restores, and anything stateful, handled by systems designed for it.
The pattern is that this list is written down. The failure is discovering each item during an incident.
Tradeoffs and Decision Framework
Purity against practicality. Chasing "everything in Git" pushes non-declarative operations into a model that cannot express them safely. Naming the exceptions explicitly is better engineering than hiding them.
Hooks against a separate process. Migrations as sync hooks keep one workflow and put a non-idempotent operation inside an idempotent loop. A separate migration pipeline is more moving parts and clearer semantics. Both are defensible; drifting between them is not.
Revert-as-rollback against a real rollback plan. Reverting is fast, familiar, and sufficient only when changes are backward compatible. Anything touching schemas or external systems needs a rollback plan that says what revert does not cover.
Where promotion lives. Directories, branches, overlays, or automation all work. What matters is that it is explicit and consistent, because this is where most GitOps setups accumulate their worst complexity.
Common Mistakes
Committing secrets to complete the model. Base64 is not encryption, and Git history is permanent.
Assuming revert equals rollback. It restores manifests, not data, not consumed side effects, not a moved image tag.
Putting non-idempotent operations in the sync path. Reconciliation may retry them, and the loop assumes repeating is safe.
Reading Git to answer "what is running." Git holds intent. Ask the cluster for reality.
No bootstrap documentation. A DR plan that begins with an already-working Argo CD is missing its first and hardest step.
Expecting sync waves to order external systems. They order resources in one Application, nothing beyond it.
Leaving emergency procedure undefined. People will act during incidents. Undefined means undocumented, unaudited, and never reconciled back.