GitOps Explained: Pull-Based Deploys, Drift Repair and Rollback by Revert

Your cluster slowly stops matching what anyone thinks is running. GitOps fixes this by making git the only place that says what should run, and letting the cluster pull from it.

Picture a common setup. A food delivery app in Hyderabad runs its orders-api service on Kubernetes, the system that runs containers across many machines. The setup is made up, but each failure below is common.

Their CI pipeline, the automation that builds and tests each change, ends with a deploy step. That step runs kubectl apply, the command that sends YAML files to the cluster. To do that, CI holds an admin key to production.

During a Friday night sale, an on-call engineer runs kubectl scale to raise orders-api from 3 copies to 10. Nobody writes it down. Monday's deploy quietly sets it back to 3, and checkout slows at lunch.

• Nobody can say what is running without logging in to the cluster and looking.
• The admin key sits in the CI system, so anyone who breaks into CI can change production.
• A rollback means finding and re-running an old pipeline, and hoping nothing else changed since.
• Manual fixes survive until the next deploy, then vanish without warning.

Each problem has the same root. The pipeline pushes changes in at one moment and never checks again. GitOps turns that around.

You need six words. Each one is tied to the orders-api example.

• Declarative config: a file that says what should exist, such as three copies of orders-api version 1.5.0, rather than the steps to get there.
• Desired state: the full set of declarative files in git for a system. For orders-api, its Deployment, Service and ConfigMap YAML files.
• Live state: what the cluster is actually running right now, as reported by the Kubernetes API.
• Drift: any difference between desired and live state, such as 10 copies running when git says 3.
• Reconcile: compare desired with live state and act to close the gap. A loop that does this forever is a reconciliation loop.
• Sync: one run of applying the desired state to the cluster. Argo CD shows an app as Synced or OutOfSync.

GitOps is a set of four principles for running software, published by the OpenGitOps project as version 1.0.0. In GitOps, desired state is declarative, versioned and immutable, pulled automatically and continuously reconciled. In practice, that versioned store is almost always git.

In plain words: you describe the system in files and keep every version in git. A program inside the cluster fetches those files and keeps fixing the cluster to match. People change production by merging a pull request, not by running commands.

• Declarative: the system's desired state is written as files that say what, not how.
• Versioned and immutable: git keeps every version, and a commit never changes after it is made.
• Pulled automatically: software agents fetch the desired state from git themselves.
• Continuously reconciled: agents keep watching live state and apply the desired state when they differ.

For orders-api, the team keeps a separate config repo that holds only deployment files. CI still builds and tests the code, and it pushes the container image, the packaged app, to a registry. Then it stops.

To release, CI or a person opens a pull request that changes one line in the config repo: the image tag. Once it merges, Argo CD or Flux, running inside the cluster, notices the new commit and applies it.

An everyday picture helps. Think of a restaurant kitchen with the menu card pinned on the wall. The cooks check each plate against the card, not against what the last waiter shouted.

The pinned card is git. Changing a dish means printing a new card, which leaves the old one in the file. The head cook who keeps checking plates is the agent. Where the picture breaks: a real agent can only fix things the card describes.

Because only a puller can check again later. The OpenGitOps glossary says agents must be able to read the desired state at any time, not only when a change triggers a push.

A push pipeline acts once per change. If someone edits the cluster afterwards, the pipeline never knows. A puller compares again on a timer and on events, so it notices drift caused by people or scripts.

Pull is not free. The agent needs read access to git and write access to the cluster, so you protect the agent and the config repo instead. The win is that you have one place to protect.

It repeats three steps forever: read the desired state, read the live state, and act on the difference. Kubernetes itself is built from such loops, which its docs call controllers.

The Kubernetes docs use a thermostat. You set the temperature you want, the room has a real temperature, and the thermostat switches heating on or off to close the gap. A GitOps agent does the same for YAML files.

Here is a toy loop in Python that shows the whole idea. It is not how Argo CD or Flux are written; it only shows the rule each one follows. We ran it on Python 3.13.

# A toy reconcile loop: desired state comes from git, live state from the cluster.
desired = {"orders-api": {"image": "orders-api:1.5.0", "replicas": 3}}
live = {"orders-api": {"image": "orders-api:1.5.0", "replicas": 10},  # kubectl scale drift
        "old-flags": {"NEW_CHECKOUT": "off"}}                        # removed from git


def reconcile(desired, live, fix_drift=True, prune=True):
    actions = []
    for name, spec in desired.items():
        if live.get(name) != spec and fix_drift:
            live[name] = dict(spec)
            actions.append(f"apply {name}")
    for name in [n for n in live if n not in desired]:
        if prune:
            del live[name]
            actions.append(f"prune {name}")
    return actions or ["in sync"]


print(reconcile(desired, live))
print(reconcile(desired, live))

What just happened? The first call prints ['apply orders-api', 'prune old-flags'], and the second prints ['in sync']. The first pass fixed the drifted copy count and deleted an object that git no longer lists. The second found nothing to do.

Not every difference is drift you want to undo. If an autoscaler, a controller that changes the copy count with load, owns the replicas field, keeping replicas in git makes the two fight. Argo CD's best practices say to leave replicas out of git then.

You revert the bad commit in the config repo, and the agent applies the result like any other change. The rollback is itself a reviewed, timestamped commit, so the history shows both the mistake and the fix.

git log --oneline -3
# 51e210c orders-api 1.5.0
# 12dd1b1 orders-api 1.4.0
git revert --no-edit 51e210c
git push origin main
# The agent pulls the new commit and sets the image back to orders-api:1.4.0

We tested this flow with real git 2.43 and a small simulated loop. After the revert, the loop applied the new commit and the live image went back to orders-api:1.4.0. Commit hashes change on every run.

Avoid the older habits. Running kubectl rollout undo by hand is drift, so an agent with self-heal on will put the bad version back. In Argo CD 3.5.6, argocd app rollback is refused while auto-sync is on.

The refusal message in the source reads: rollback cannot be initiated when auto-sync is enabled. That is deliberate: git, not the tool's own history, decides what should run.

Beginners can stop here: you know what GitOps is, why it pulls, how drift is found and fixed, and how rollback works. The rest covers the real defaults in Argo CD and Flux, and where GitOps hurts.

Argo CD is cautious by default and Flux is strict. Argo CD only reports drift until you turn on auto-sync, self-heal and prune. Flux corrects drift on every interval and makes you choose prune explicitly.

Argo CD: the Application and its sync policy

In Argo CD, an Application is a Kubernetes object that says which repo path to deploy to which cluster and namespace. This one turns on auto-sync with self-heal and prune.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/deploy-config.git
    targetRevision: main
    path: apps/orders-api/production
  destination:
    server: https://kubernetes.default.svc
    namespace: orders
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

We checked this file against the Application schema shipped in Argo CD v3.5.6, without a live cluster. The CLI equivalent is argocd app set orders-api --sync-policy automated --auto-prune --self-heal.

The auto-sync docs list rules worth knowing. Auto-sync runs only when the app is OutOfSync. It tries each commit once, and does not try a commit again after its sync failed.

Argo CD checks git every 3 minutes by default: 120 seconds plus up to 60 seconds of random delay, called jitter, so all apps do not poll at once. A webhook, a call your git host makes on each push, removes that wait.

Live changes are faster. In the v3.5.6 source, a change to any object an app manages triggers a refresh of that app. So a kubectl scale shows as OutOfSync without waiting for the 3-minute poll.

The self-heal timing the auto-sync page gets wrong

The auto-sync page says self-heal retries after a 5-second timeout. The v3.5.6 source does something different. The --self-heal-timeout-seconds flag defaults to 0, and an exponential backoff is on by default instead. Its flags are listed in the controller's command reference, but the auto-sync page still says 5 seconds.

Its defaults are a 2-second start, a factor of 3 and a 300-second cap. We copied the backoff code from the v3.5.6 source into a small Go program and ran it.

Pause and predict: if something undoes Argo CD's fix again and again, how often will Argo CD fix it? The docs suggest every 5 seconds. Here are the real waits between repeated self-heals of one commit.

• First heal: no wait.
• Then 2 s, 6 s, 18 s, 54 s and 2 minutes 42 seconds.
• After that, 5 minutes each time, the cap.
• A new commit starts a fresh sync, so the count starts again.

This matters when two things fight. If another controller keeps changing a field Argo CD owns, self-heal does not re-apply every 5 seconds. It backs off, and the app can sit OutOfSync for up to 5 minutes between fixes.

A second detail is also only in the source. An automated sync with no retry block still gets a retry limit of 5. The default retry wait starts at 5 seconds, doubles and stops growing at 3 minutes. These retries happen inside one sync; once it finally fails, auto-sync leaves that commit alone.

Flux: GitRepository plus Kustomization

Flux splits the job in two. A GitRepository object fetches the repo on its interval. A Kustomization object, Flux's apply unit and not the same as a kustomization.yaml file, builds the files and applies them.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: deploy-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://git.example.com/platform/deploy-config.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: orders-api
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: deploy-config
  path: ./apps/orders-api/production
  prune: true
  targetNamespace: orders

Both objects pass the schemas shipped with Flux v2.10.0. When we removed the prune line, the Kustomization failed the check with prune is a required property, so Flux forces you to decide.

The Flux docs say each Kustomization interval runs a server-side apply dry run to detect and correct drift. They suggest at least 60 seconds. Under the default apply policy, kubectl edits to Flux-managed objects are reverted.

To skip waiting, run flux reconcile kustomization orders-api --with-source, which fetches the repo and applies it now. To stop corrections while you debug, run flux suspend kustomization orders-api, then flux resume kustomization orders-api when done.

Three places hurt most: secrets, ordering between resources, and promotion across environments. Each one comes from the same rule that makes GitOps work: everything must be a file in git.

Secrets cannot sit in git as plain text

A Kubernetes Secret is only base64 encoded, a reversible text encoding and not encryption. The Flux docs call plain or base64 Secrets in git unsafe, whatever the repo's access rules.

• Let a controller in the target cluster create the real Secret, so the agent never handles the plain value. Argo CD's docs strongly recommend this style.
• Argo CD's docs name External Secrets Operator, which copies values from a vault, and Sealed Secrets, which keeps encrypted values in git that only the cluster can decrypt.
• Flux can also decrypt secrets itself. It supports SOPS, its only decryption provider, with age, OpenPGP or cloud key services.
• Avoid injecting secrets while Argo CD renders manifests. Its docs warn that generated manifests sit in plain text in its Redis cache.

Ordering: some objects must exist first

A file in git has no notion of first. A custom resource fails if its definition, the CRD, is not installed yet. A database migration Job should finish before the new app version starts.

Argo CD uses sync waves: an annotation with a number, default 0, applied lowest first, with a 2-second pause between waves. Hooks such as PreSync run Jobs before the main apply. Pruning runs in reverse wave order.

Flux uses dependsOn: a Kustomization waits until the ones it lists report Ready. The Flux docs give the classic case of applying cert-manager's CRDs before the certificates that use them.

Promotion: moving a version from staging to production

Git has no built-in idea of environments, so you build one. The common layout keeps a base folder plus one overlay folder per environment, all on one branch, as the Flux repository structure guide shows.

Promotion is then a pull request that copies the tested image tag from the staging folder to the production folder. It is reviewable and revertible, but someone or something must open that request after staging passes.

• Avoid one long-lived branch per environment. The Flux guide keeps environments in folders on one branch with short-lived feature branches.
• Use a separate production repo when production config must be hidden. Git cannot limit who reads single files.
• Pin remote bases and charts to a tag or commit SHA. Argo CD's docs warn that a remote base at HEAD can change meaning without any commit of yours.

What is coming: Argo CD's source hydrator, beta since v3.5.0 and off by default as of 27 October 2026, writes rendered manifests to git. It can write to a staging branch so promotion becomes a pull request, though Argo CD does not open that request for you.

Start with one service, detection only, and turn on automatic fixing once the repo is trusted. These steps use Argo CD names. Flux applies from the start, so with Flux begin step 3 with prune: false instead.

1. 1

   Create a config repo

   Put orders-api's YAML under apps/orders-api with base and per-environment folders. Keep it separate from the code repo so CI commits do not trigger new builds.

2. 2

   Remove the cluster key from CI

   CI builds, tests and pushes the image, then opens a pull request that changes the image tag. Delete the kubeconfig from CI secrets.

3. 3

   Install the agent with manual sync

   Create the Application without syncPolicy.automated. Read the diff in the UI or with argocd app diff orders-api, and fix files until it shows Synced.

4. 4

   Turn on auto-sync, then self-heal and prune

   Enable automated sync first. Add selfHeal and prune after a week without surprise diffs, because prune deletes whatever git no longer lists.

5. 5

   Add a webhook and alerts

   Point a git webhook at Argo CD to skip the 3-minute poll. Alert when an app stays OutOfSync or Degraded longer than the 5-minute self-heal cap, say 10 minutes.

6. 6

   Practise a rollback

   Merge a harmless bad change in staging, then git revert it. Time how long the cluster takes to return to the old image.

Lead with the inversion: CI stops pushing, and an agent in the cluster pulls desired state from git. Then show you know the defaults, the failure modes and the limits.

1. 1

   State the need

   Many services, several clusters, and a need to know exactly what runs, who changed it and how to undo it.

2. 2

   Draw the flow

   CI builds an image and opens a config pull request. An agent in each cluster pulls the config repo and applies it.

3. 3

   Explain the loop

   The agent compares desired and live state on a timer and on change events. Drift is reported, and fixed if self-heal is on.

4. 4

   Cover rollback and safety

   Rollback is git revert. Prune deletes objects, so Argo CD makes it opt-in and Flux makes it a required field.

5. 5

   Name the trade-offs

   Secrets need an operator or SOPS. Ordering needs sync waves or dependsOn. Promotion is a pull request between environment folders.