GitOps Best Practices for Production

GitOps is easy to demo and easy to get wrong. Installing Argo CD or Flux takes an afternoon; the failure modes show up months later — secrets in Git history, a prod branch that drifted from staging a year ago, an auto-heal loop fighting a database operator at 3 a.m. These are the GitOps best practices we hold teams to when their Kubernetes clusters actually carry production traffic. None of them are exotic. All of them get skipped.

1. Git is the only source of truth — the cluster pulls, humans don't push

The core contract of GitOps: the desired state of every cluster lives in Git, and a controller inside the cluster (Argo CD or Flux) continuously reconciles live state to match it. Nobody runs kubectl apply against production. Not for hotfixes, not "just this once", not during an incident.

The moment someone hand-applies a change, you have two sources of truth, and the next sync will either silently revert the fix or silently keep unreviewed state alive. Both are worse than the outage. If production is on fire, the fast path is still a Git commit — merged by an admin, synced in seconds. If your process makes that slower than SSH-ing in, fix the process, not the contract.

This also means cluster credentials disappear from CI. The pipeline builds, tests, and commits a manifest change; it never holds a kubeconfig. That single property removes the most common blast radius in a compromised CI system — a leaked admin token with a direct line to production.

2. Separate app code from config, and use directories for environments

Keep the application source repo and the deployment config repo separate (or, in a monorepo, a strictly separated deploy/ tree with its own CODEOWNERS). The reasons are practical: different review policies, different release cadences, and your CI pipeline can update manifests without triggering an app rebuild — or an infinite commit loop when an image-update automation writes back to the repo it watches.

For environments, use directories with overlays, not long-lived branches. A branch-per-environment model sounds tidy and rots fast: merge conflicts on generated manifests, cherry-picks that skip staging, and a prod branch whose diff against staging nobody can explain. Kustomize overlays keep the delta between environments visible in one tree:

config-repo/
├── apps/
│   └── payments/
│       ├── base/
│       │   ├── deployment.yaml
│       │   ├── service.yaml
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── staging/
│           │   └── kustomization.yaml   # replicas: 2, staging ingress
│           └── prod/
│               └── kustomization.yaml   # replicas: 6, PDB, prod ingress
└── clusters/
    ├── staging/
    └── prod/

The base holds what is common; each overlay holds only what differs. Reviewing a promotion means reading a small, honest diff. Helm users get the same shape with a shared chart and per-environment values files; the principle is identical — one lineage of config, explicit per-environment deltas, no divergent histories to reconcile by hand.

3. Manage apps declaratively — app-of-apps, ApplicationSets, Kustomizations

If engineers click "New App" in the Argo CD UI, your app inventory is itself unmanaged state — the exact problem GitOps exists to solve. The controller's own configuration belongs in Git too:

Argo CD: use the app-of-apps pattern or, better for fleets, ApplicationSet with a Git generator — every directory under apps/ becomes an Application automatically. Adding a service to a cluster is a PR, not a click.
Flux: compose Kustomization and HelmRelease resources, with a top-level Kustomization per cluster that pulls in the rest and dependsOn ordering between them.

The test: could you rebuild a cluster from an empty control plane with one bootstrap command and a Git URL? If not, something lives outside Git that shouldn't.

4. Promote by PR, and pin images by digest

Environment promotion should be boring: CI builds an image once, staging runs it, and after it passes, the same artifact flows to prod as a pull request that edits one line in the prod overlay. That PR is your promotion gate — reviewable, auditable, revertable — and it replaces every bespoke "deploy to prod" pipeline stage with a merge button plus branch protection. The reviewer sees exactly what changes: one digest, nothing else.

# overlays/prod/kustomization.yaml
images:
  - name: registry.example.com/payments
    newName: registry.example.com/payments
    digest: sha256:9f2c8a41d6e0b3f7...   # the digest that passed staging

Never deploy :latest, and treat mutable tags with suspicion generally. A mutable tag means "what runs in prod" depends on when the node happened to pull, Git no longer describes reality, and rollback becomes guesswork. Pinning by sha256 digest guarantees that the exact bytes you tested in staging are the bytes prod runs. Image-update automation (Argo CD Image Updater, Flux image automation) is fine — as long as it works by writing a commit, so the audit trail survives.

5. Secrets: never plaintext in Git, no exceptions

A secret committed to Git lives in history forever; deleting the file changes nothing. A leaked config repo becomes a leak of every credential in it. You have three sane options:

Sealed Secrets: encrypt client-side against the cluster's public key; only the in-cluster controller can decrypt. Simple, but ties ciphertext to one cluster.
SOPS (with age or a cloud KMS): encrypt values in-place in YAML. Native in Flux; works with Argo CD via plugins. Diffs stay reviewable because keys remain plaintext.
External Secrets Operator: Git holds only a reference; the operator pulls the value from Vault, AWS Secrets Manager, or similar at runtime. Best fit when a central secrets store already exists — rotation happens in one place.

Pick one, enforce it with a pre-commit hook and a scanner (gitleaks) in CI, and treat any plaintext secret that ever touched a commit as compromised: rotate it.

6. Auto-sync and self-heal — with care around stateful workloads

Enable automated sync, self-heal, and pruning for stateless services; drift detection without automatic correction is just a dashboard people ignore. But be deliberate where reconciliation can fight another controller or a human mid-incident:

Databases and anything operator-managed: operators mutate their own resources, and a naive self-heal will revert them in a loop. Use ignoreDifferences for operator-owned fields, or leave auto-heal off and alert on drift instead.
Use sync waves (Argo CD) or dependsOn (Flux) so CRDs, namespaces, and databases reconcile before the apps that need them.
Define health checks — including custom Lua checks for CRs — so "synced" means "actually healthy", and gate later waves on them.
Enable pruning, but protect crown jewels with Prune=false annotations on resources whose accidental deletion would be unrecoverable.

7. Rollback is git revert — so keep changes small

The payoff for all this discipline: recovering from a bad release is git revert plus one reconcile loop. No pipeline re-run, no hunting for the previous Helm values. But that only holds if commits are small and atomic — one logical change per commit. A commit that bumps three services and rewrites an ingress cannot be reverted surgically. If your rollback plan involves anything other than reverting a commit, your GitOps setup has a hole in it.

Practice the revert before you need it. Run a game day: revert a real (harmless) prod commit, time the reconcile, and confirm the app returns to the prior digest. A rollback path you have never exercised is a rollback theory.

One caveat: reverting manifests does not revert data. If a release ran a schema migration, the Git revert restores the old code against the new schema. Keep migrations backward-compatible for at least one release (expand-and-contract), so a manifest revert is always safe on its own.

8. Least privilege for the controller, boundaries between tenants

The GitOps controller is the most powerful workload in your cluster — often cluster-admin by default. Treat it accordingly:

Scope permissions down. In Argo CD, use AppProject to restrict which repos, destination namespaces, and resource kinds each team's apps may touch. In Flux, run per-tenant Kustomizations under distinct service accounts with namespace-scoped RBAC.
Separate tenants so team A's repo cannot deploy into team B's namespace — or into kube-system.
Give the controller read-only deploy keys per repo, not an org-wide token. Lock down who can merge to the paths that feed prod with CODEOWNERS and branch protection — that merge button is now your production deploy button.

9. Monitor the controller like the production system it is

Once deploys flow through reconciliation, a silently failing controller means changes stop shipping — including your incident fix — and drift accumulates unseen. Minimum viable observability:

Alert on apps stuck in a failed or degraded sync state beyond a threshold.
Alert on persistent drift (OutOfSync that self-heal is not correcting).
Alert on the controller's own health and its ability to reach Git.
Scrape the metrics both tools export (argocd_app_info, Flux's gotk_reconcile_condition) and wire notifications into the channel engineers actually read.

Track sync-failure duration as a real SLO. Teams routinely discover their controller had been failing to reconcile for days — an expired deploy key, a renamed branch — only when someone asks why a merged change never shipped. By then the diff between Git and the cluster is large enough that the eventual sync is itself a risky event.

Anti-patterns that should fail review

Every one of these looks like a shortcut and costs more than it saves. If any of them describes your setup today, that is the highest-leverage thing to fix:

Plaintext secrets in the repo — including "temporary" ones.
:latest or any mutable tag in a prod manifest.
Long-lived branch-per-environment with cherry-pick promotion.
Manual kubectl hotfixes that leave prod drifted from Git.
Direct pushes to the prod path with no PR review.
Apps created by hand in the UI instead of declaratively.
Auto-heal blindly enabled on operator-managed stateful resources.
No alerting on sync failures — discovering a dead controller during an incident.

Where to start

If you are adopting GitOps now: structure the config repo with overlays first, wire up digest-pinned promotion by PR second, and solve secrets before the first real credential needs a home. Retrofitting any of these onto a live setup is far more painful than starting with them. If you already run GitOps, audit against the anti-pattern list above — most production incidents we get called into trace back to one of those eight.

If you want an experienced pair of hands, our GitOps consulting practice designs and hardens exactly these workflows, and our Kubernetes platform work covers the cluster underneath them.

GitOps Best Practices for Production Kubernetes