Containers

ArgoCD ApplicationSet: Multi-Cluster Deployment Patterns

An Argo CD ApplicationSet turns one template into many Applications, one per cluster, directory or config file. Adding a cluster to the fleet stops meaning another hand-written manifest. The generators are easy to read and easy to get wrong: a typo in a goTemplate key produces zero Applications, two ApplicationSets that render the same name fight over ownership, and the wrong delete command takes your workloads with it.

Original content from computingforgeeks.com - post 166929

We ran every generator (list, cluster, Git directory, Git file, matrix and merge) against two registered spoke clusters, then broke each one on purpose to capture the real error, and timed a RollingSync rollout from Git push to Healthy. Every command and output below came from Argo CD 3.6.1 on three kind clusters (Kubernetes 1.37) in November 2026.

Map the hub and spoke layout

Argo CD runs on one cluster only, the hub. The prod and staging clusters run nothing from Argo CD: they are registered as cluster secrets on the hub, and the application controller talks to their API servers on port 6443. The ApplicationSet controller sits next to it and does one job, which is rendering Application objects from generators and keeping them in step with Git.

Argo CD ApplicationSet architecture diagram: Gitea fleet.git feeds the ApplicationSet controller on the hub cluster, which creates Applications that the application-controller syncs to webapp and addons on the prod and staging clusters over port 6443

Keep that split in mind when something fails. If no Application appears, the problem is in the ApplicationSet (template, generator, ownership). If the Application exists but won’t sync, the problem is between the hub and the spoke (credentials, DNS, the manifests in Git), and the ApplicationSet has already done its part.

Build the lab: hub, prod and staging

The lab needs three clusters and a Git server the hub can reach, and nothing else. We used kind on one Ubuntu 24.04 VM with 8 vCPUs and 32 GB of RAM, which is a comfortable floor for three single-node clusters plus Gitea. Production spokes are whatever you already run; the hub only needs network access to each spoke’s API server. Create the clusters:

for c in hub prod staging; do kind create cluster --name $c --wait 120s; done

That took 2 minutes 12 seconds for all three. Each cluster runs as a Docker container on the shared kind network, which matters later when Argo CD needs to resolve the spoke API servers:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

All three control planes should be up on the same node image:

NAMES                   IMAGE                  STATUS
staging-control-plane   kindest/node:v1.37.0   Up 31 seconds
prod-control-plane      kindest/node:v1.37.0   Up About a minute
hub-control-plane       kindest/node:v1.37.0   Up About a minute

For Git we ran Gitea in a container attached to the same network, so pods on the hub reach it as http://gitea:3000. Any Git server works here; the generators only care that the repo server can clone the URL. Start it:

docker run -d --name gitea --network kind -p 127.0.0.1:3000:3000 \
 -e GITEA__security__INSTALL_LOCK=true -e GITEA__server__ROOT_URL=http://gitea:3000/ -e GITEA__database__DB_TYPE=sqlite3 \
 gitea/gitea:latest

With INSTALL_LOCK set there is no web installer, so create the first user from inside the container. Wait for the health endpoint first, because the first start generates the config and migrates the SQLite database. Pick your own password and keep it in a shell variable, because the repo API call and the Git remote both need it:

for i in $(seq 40); do curl -fs http://127.0.0.1:3000/api/healthz >/dev/null && break; sleep 3; done
export GITEA_PASS='Use-A-Strong-Pass-1'
docker exec -u git gitea gitea admin user create --username josphat --password "$GITEA_PASS" --email [email protected] --admin --must-change-password=false

Then create an empty public repository called fleet through the Gitea API. Public matters here, since it lets Argo CD clone without a repository credential:

curl -s -u josphat:"$GITEA_PASS" -H 'Content-Type: application/json' -d '{"name":"fleet","private":false,"default_branch":"main"}' http://127.0.0.1:3000/api/v1/user/repos | jq -r .clone_url

The clone URL it prints, http://gitea:3000/josphat/fleet.git, is the address every ApplicationSet below uses. Now build the repo content locally. The workload is a plain nginx Deployment with a Service, laid out as a Kustomize base:

mkdir -p ~/fleet/apps/webapp/base ~/fleet/apps/webapp/overlays/{prod,staging} ~/fleet/addons/{logging,metrics} ~/fleet/clusters/{prod,staging}
vim ~/fleet/apps/webapp/base/deployment.yaml

Leave the base at one replica, because the overlays decide the real count per environment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80

Add the Service in ~/fleet/apps/webapp/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  selector:
    app: webapp
  ports:
    - port: 80
      targetPort: 80

Next, ~/fleet/apps/webapp/base/kustomization.yaml just lists both files:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

Each overlay patches the replica count. This is ~/fleet/apps/webapp/overlays/prod/kustomization.yaml; the staging copy in overlays/staging is identical except for value: 2:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - target:
      kind: Deployment
      name: webapp
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 3

The add-ons are deliberately tiny, one ConfigMap each, so the generator behaviour is easy to see. This is ~/fleet/addons/logging/configmap.yaml; the metrics one only changes the name to metrics-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logging-config
data:
  enabled: "true"

The two clusters/*/config.json files are one line each and are shown in the Git file generator section; create them now so the first commit is complete. Then commit everything and push it to Gitea. The remote URL stores the password in ~/fleet/.git/config, which is acceptable for a throwaway lab and not for anything else:

cd ~/fleet
git init -q -b main; git config user.name josphat; git config user.email [email protected]
git add -A; git commit -qm "initial fleet repo"
git remote add origin "http://josphat:${GITEA_PASS}@127.0.0.1:3000/josphat/fleet.git"
git push -u origin main

The repo now has this layout. The apps/webapp tree is the Kustomize base with one overlay per environment (3 replicas in prod, 2 in staging), addons holds one directory per add-on, and clusters holds one JSON file per cluster:

tree ~/fleet

Nine files in total:

/home/josphat/fleet
├── addons
│   ├── logging
│   │   └── configmap.yaml
│   └── metrics
│       └── configmap.yaml
├── apps
│   └── webapp
│       ├── base
│       │   ├── deployment.yaml
│       │   ├── kustomization.yaml
│       │   └── service.yaml
│       └── overlays
│           ├── prod
│           │   └── kustomization.yaml
│           └── staging
│               └── kustomization.yaml
└── clusters
    ├── prod
    │   └── config.json
    └── staging
        └── config.json

13 directories, 9 files

Install Argo CD on the hub. The ApplicationSet controller ships in the standard install manifest, so there is nothing extra to enable for the basic generators. Our Argo CD install guide covers ingress and TLS; for this lab the plain manifest is enough:

kubectl config use-context kind-hub
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd wait --for=condition=Ready pods --all --timeout=600s

Every pod was Ready 56 seconds after the apply. Check that argocd-applicationset-controller is among them:

kubectl -n argocd get pods

Seven pods, all Running:

NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          53s
argocd-applicationset-controller-794bb95c9d-mgqq6   1/1     Running   0          54s
argocd-dex-server-66cf64bfcf-g5l5r                  1/1     Running   0          54s
argocd-notifications-controller-5878c8d644-4rgqh    1/1     Running   0          54s
argocd-redis-5869b7547-6brzg                        1/1     Running   0          54s
argocd-repo-server-7d55cc466b-lzhzm                 1/1     Running   0          54s
argocd-server-85b849c688-6zbcb                      1/1     Running   0          53s

The rest of the guide uses the argocd CLI for cluster registration and argocd appset commands. If it isn’t on your workstation yet, install the Argo CD CLI first, then port-forward the API server and log in as admin (the CLI login guide covers SSO and tokens for anything beyond a lab):

nohup kubectl port-forward svc/argocd-server -n argocd 8080:443 --address 127.0.0.1 >~/pf.log 2>&1 &
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d > ~/argocd-admin-pass.txt; chmod 600 ~/argocd-admin-pass.txt
argocd login 127.0.0.1:8080 --username admin --password "$(cat ~/argocd-admin-pass.txt)" --insecure

Once you’re logged in, argocd version reports both the client and the server build, which is the quickest way to confirm the CLI is talking to the hub and not some other context.

Register the spoke clusters

Register each spoke with argocd cluster add and label it, because every generator that targets clusters reads Argo CD’s cluster secrets and their labels. argocd cluster add logs into the spoke with your kubeconfig, creates an argocd-manager ServiceAccount there and stores its token on the hub. Export each spoke’s kubeconfig with its internal address, because the hub reaches the spokes over the kind network and not through localhost:

for c in prod staging; do kind get kubeconfig --name $c --internal > ~/$c.kubeconfig; done
argocd cluster add kind-prod --name prod --kubeconfig ~/prod.kubeconfig --yes

Our first attempt failed. The CLI also uses that kubeconfig from the VM itself, and the VM’s resolver has never heard of prod-control-plane:

{"level":"fatal","msg":"failed to create service account \"argocd-manager\" in namespace \"kube-system\": Post \"https://prod-control-plane:6443/api/v1/namespaces/kube-system/serviceaccounts\": dial tcp: lookup prod-control-plane on 127.0.0.53:53: server misbehaving","time":"2026-11-13T09:05:22Z"}

The fix is to make the name resolvable on the machine running the CLI, while keeping the URL Argo CD stores the same one the hub can resolve. On a kind lab that means adding each control plane’s container IP to /etc/hosts. On real clusters the same error means your kubeconfig points at a name only the cluster network knows, and you should use the API server’s routable DNS name instead:

for c in prod staging; do ip=$(docker inspect -f '{{(index .NetworkSettings.Networks "kind").IPAddress}}' $c-control-plane); echo "$ip $c-control-plane" | sudo tee -a /etc/hosts; done
argocd cluster add kind-prod --name prod --kubeconfig ~/prod.kubeconfig --yes
argocd cluster add kind-staging --name staging --kubeconfig ~/staging.kubeconfig --yes

This time both spokes register, and each one reports the ServiceAccount, ClusterRole and token it created:

{"level":"info","msg":"ServiceAccount \"argocd-manager\" created in namespace \"kube-system\"","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"ClusterRole \"argocd-manager-role\" created","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"ClusterRoleBinding \"argocd-manager-role-binding\" created","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"Created bearer token secret \"argocd-manager-long-lived-token\" for ServiceAccount \"argocd-manager\"","time":"2026-11-13T09:05:27Z"}
Cluster 'https://prod-control-plane:6443' added

Now label them. The cluster generator selects on these labels and exposes them to templates, so treat them as the fleet’s inventory: env picks the overlay and tier marks how careful a rollout has to be.

argocd cluster set prod --label env=prod --label tier=critical
argocd cluster set staging --label env=staging --label tier=standard
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster --show-labels

Labels land on the cluster secrets themselves, which is what the generator reads:

NAME                                       TYPE     DATA   AGE   LABELS
cluster-prod-control-plane-3648912337      Opaque   3      6s    argocd.argoproj.io/secret-type=cluster,env=prod,tier=critical
cluster-staging-control-plane-1443425697   Opaque   3      5s    argocd.argoproj.io/secret-type=cluster,env=staging,tier=standard

Don’t worry if argocd cluster list shows prod as Unknown with “Cluster has no applications and is not being monitored.” A cluster may sit in that state until an Application targets it. In our lab staging already read Successful at that point while prod read Unknown, and every sync to prod afterwards worked, so this state alone is not a registration failure.

Choose a generator

Pick the generator by where the list of targets already lives, because that source becomes the thing you edit to change the fleet:

  • A short fixed list in your head: the list generator
  • Argo CD’s cluster registry: the cluster generator
  • The repo layout: a Git directory or file generator
  • Two of those at once: matrix or merge

These are the times we measured from kubectl apply to every generated Application reporting Synced and Healthy, so you know what normal looks like before something goes wrong:

GeneratorSource of targetsApps in our labApply to Healthy
listElements written in the ApplicationSet210 s
clustersCluster secrets matching a label selector22 s
git directoriesDirectories matching a path glob2, then 33 s (new dir: 357 s)
git filesJSON or YAML files matching a glob23 s
matrixEvery combination of two generators612 s
mergeA base generator with per-key overrides2not timed (we were checking the tier labels, not health)

One number stands out: 357 seconds. A Git directory generator without a webhook only notices a new directory on its next poll, and that delay is covered in the Git directories section below. Everything else was ready within seconds, because the generator output was already known when the ApplicationSet was applied.

Deploy with the list generator

The list generator is the right choice for a handful of fixed targets, and it’s the easiest one to debug because every parameter is visible in the manifest. Each element becomes one Application, and its keys become template variables. We keep all the manifests in ~/lab/manifests on the workstation, so create that directory and the first manifest:

mkdir -p ~/lab/manifests
vim ~/lab/manifests/webapp-list.yaml

Every ApplicationSet in this guide sets goTemplate: true with missingkey=error. Go templates give you default, get and other Sprig functions, and missingkey=error turns a typo into a visible error instead of an Application named webapp-<no value>:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: webapp-list
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - cluster: prod
            url: https://prod-control-plane:6443
            path: apps/webapp/overlays/prod
          - cluster: staging
            url: https://staging-control-plane:6443
            path: apps/webapp/overlays/staging
  template:
    metadata:
      name: 'webapp-{{.cluster}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: '{{.path}}'
      destination:
        server: '{{.url}}'
        namespace: webapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Apply it on the hub, then ask Argo CD what it generated:

kubectl apply -f ~/lab/manifests/webapp-list.yaml
argocd appset get webapp-list
kubectl -n argocd get applications

Read the three conditions first. ParametersGenerated True means the generator produced its parameter sets, and ResourcesUpToDate True means the Applications match the template. The {{.url}} and {{.path}} shown in the Server and Path fields are normal: appset get prints the template, not the rendered values.

argocd appset get webapp-list showing Healthy with all conditions passing, and kubectl get applications listing webapp-prod and webapp-staging Synced and Healthy

Both spokes got their overlay within 10 seconds of the apply: webapp ran 3/3 replicas on prod and 2/2 on staging. Note one more thing on the generated Application before moving on, because it decides what happens when you delete the ApplicationSet:

kubectl -n argocd get application webapp-prod -o jsonpath='{.metadata.finalizers}'

By default, the Application carries the resources finalizer:

["resources-finalizer.argocd.argoproj.io"]

With that finalizer present, kubectl -n argocd delete appset webapp-list removed both Applications and the webapp Deployment with them; on prod only the empty namespace was left behind. That’s the right default for a test, and a nasty surprise in production, so the deletion section further down shows how to keep workloads alive.

Target clusters by label with the cluster generator

The cluster generator removes the URL bookkeeping. It walks the cluster secrets on the hub, keeps the ones matching the selector, and hands each match to the template as .name, .server and .metadata.labels. Register a new cluster with env=prod and it gets the workload with no change in Git:

vim ~/lab/manifests/webapp-clusters.yaml

The overlay path is built from the cluster’s env label, so the label and the directory name have to agree:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: webapp-clusters
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:
        selector:
          matchExpressions:
            - key: env
              operator: In
              values: [prod, staging]
  template:
    metadata:
      name: 'webapp-{{.name}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: 'apps/webapp/overlays/{{.metadata.labels.env}}'
      destination:
        server: '{{.server}}'
        namespace: webapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Apply it and check what the generator produced:

kubectl apply -f ~/lab/manifests/webapp-clusters.yaml
argocd appset get webapp-clusters
kubectl -n argocd get applications

The template fields still show {{.server}} and the label path, while all three conditions report success:

Name:               argocd/webapp-clusters
Project:            default
Server:             {{.server}}
Namespace:          webapp
Health Status:      Healthy
Source:
- Repo:             http://gitea:3000/josphat/fleet.git
  Target:           main
  Path:             apps/webapp/overlays/{{.metadata.labels.env}}
SyncPolicy:         Automated (Prune)

CONDITION            STATUS  MESSAGE                                            LAST TRANSITION
ErrorOccurred        False   All applications have been generated successfully  2026-11-13 09:06:41 +0000 UTC
ParametersGenerated  True    All parameters have been generated successfully    2026-11-13 09:06:41 +0000 UTC
ResourcesUpToDate    True    All applications have been generated successfully  2026-11-13 09:06:41 +0000 UTC

Both labelled spokes got an Application, Synced and Healthy 2 seconds after the apply:

NAME             SYNC STATUS   HEALTH STATUS
webapp-prod      Synced        Healthy
webapp-staging   Synced        Healthy

Keep the selector. We rendered a copy with clusters: {} and no selector through argocd appset generate, which previews the output without creating anything:

argocd appset generate ~/lab/manifests/webapp-clusters-noselector.yaml -o json | jq -r '.[].metadata.name'

Notice the hub itself showing up as a third target:

webapp-prod
webapp-staging
webapp-in-cluster

An empty cluster generator includes in-cluster, the hub Argo CD runs on. Applied for real, that would have deployed the prod overlay onto the management cluster. Run argocd appset generate before every new ApplicationSet; it costs nothing and catches exactly this kind of surprise.

Generate apps from Git directories

The Git directory generator creates one Application per directory matching a glob, which suits platform add-ons: drop a new directory under addons/ and it deploys. In goTemplate mode the path comes through as an object, so you write {{.path.path}} for the full path and {{.path.basename}} for the last segment:

vim ~/lab/manifests/addons-git-dir.yaml

To keep the example short, this one deploys to the hub (https://kubernetes.default.svc). The matrix section sends the same add-ons to the spokes:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: addons-git-dir
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: http://gitea:3000/josphat/fleet.git
        revision: main
        directories:
          - path: addons/*
  template:
    metadata:
      name: 'addons-{{.path.basename}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'addons-{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Applying it created addons-logging and addons-metrics in 3 seconds. Then we pushed a third add-on, addons/tracing/configmap.yaml, a copy of the logging ConfigMap named tracing-config, and waited. The matrix section later relies on it too:

cd ~/fleet; git add -A; git commit -qm "add tracing addon"; git push -q origin main

addons-tracing appeared 357 seconds after the push. The ApplicationSet controller polls Git every 3 minutes by default, and the Git generator also reads through the repo server’s revision cache, so without a webhook a new directory can sit unseen for more than one poll.

If six minutes is too slow, note that the Argo CD API server’s webhook does not help here. Per the Git generator docs, the ApplicationSet controller runs its own webhook server (the argocd-applicationset-controller Service, port 7000), which you expose with a separate Ingress and point GitHub or GitLab at its /api/webhook. We left the lab on polling, since Gitea is not one of the documented senders; requeueAfterSeconds on the generator is the documented knob for a shorter poll. To skip a directory instead, add a second path entry with exclude: true:

        directories:
          - path: addons/*
          - path: addons/logging
            exclude: true

Excluding a directory is a delete, not a pause. Applying the excluded version removed addons-logging and, because of the resources finalizer, its ConfigMap went with it:

kubectl -n argocd get applications

Only two add-ons remain:

NAME             SYNC STATUS   HEALTH STATUS
addons-metrics   Synced        Healthy
addons-tracing   Synced        Healthy

Use exclude when you really want an add-on gone from every cluster. If you only want to stop syncing it for a while, set preserveResourcesOnDeletion on the ApplicationSet first.

Read per-cluster values with the Git file generator

The Git file generator reads JSON or YAML files and turns every top-level key into a template variable. It’s the cleanest way to keep per-cluster settings in Git, reviewed like any other change, instead of in cluster secret labels. Each cluster gets one small file:

cat ~/fleet/clusters/prod/config.json ~/fleet/clusters/staging/config.json

Each file holds the cluster name, its API URL and a few settings:

{"cluster":"prod","url":"https://prod-control-plane:6443","replicas":"3","tier":"critical"}
{"cluster":"staging","url":"https://staging-control-plane:6443","replicas":"2","tier":"standard"}

Point the generator at the glob and use the keys directly. Create the manifest:

vim ~/lab/manifests/webapp-git-files.yaml

Here {{.cluster}} and {{.url}} come straight from the JSON:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: webapp-git-files
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: http://gitea:3000/josphat/fleet.git
        revision: main
        files:
          - path: clusters/*/config.json
  template:
    metadata:
      name: 'webapp-{{.cluster}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: 'apps/webapp/overlays/{{.cluster}}'
      destination:
        server: '{{.url}}'
        namespace: webapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Apply it, then check the ApplicationSet, the Applications and the Deployment on each spoke:

kubectl apply -f ~/lab/manifests/webapp-git-files.yaml
argocd appset get webapp-git-files
kubectl -n argocd get applications
kubectl --context kind-prod -n webapp get deploy
kubectl --context kind-staging -n webapp get deploy

Here Server reads {{.url}}, the key from the JSON files, and the conditions are clean:

Name:               argocd/webapp-git-files
Project:            default
Server:             {{.url}}
Namespace:          webapp
Health Status:      Healthy
Source:
- Repo:             http://gitea:3000/josphat/fleet.git
  Target:           main
  Path:             apps/webapp/overlays/{{.cluster}}
SyncPolicy:         Automated (Prune)

CONDITION            STATUS  MESSAGE                                            LAST TRANSITION
ErrorOccurred        False   All applications have been generated successfully  2026-11-13 09:13:52 +0000 UTC
ParametersGenerated  True    All parameters have been generated successfully    2026-11-13 09:13:52 +0000 UTC
ResourcesUpToDate    True    All applications have been generated successfully  2026-11-13 09:13:52 +0000 UTC

Both Applications were Synced and Healthy 3 seconds after the apply, and each overlay landed with its own replica count, 3/3 on prod and 2/2 on staging:

NAME             SYNC STATUS   HEALTH STATUS
webapp-prod      Synced        Healthy
webapp-staging   Synced        Healthy

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
webapp   3/3     3            3           2s

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
webapp   2/2     2            2           2s

The trade-off against the cluster generator is that the URL now lives in two places, the cluster secret and the JSON file, so a re-registered cluster with a new address needs a commit too.

Combine generators with matrix

Matrix produces every combination of two child generators. Git directories times clusters is the classic use: every add-on on every cluster, so three add-ons on two clusters give six Applications. Parameters from both children are available, which is why the name below mixes .path.basename from Git with .name from the cluster:

vim ~/lab/manifests/addons-matrix.yaml

Paste the matrix definition:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: addons-matrix
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: http://gitea:3000/josphat/fleet.git
              revision: main
              directories:
                - path: addons/*
          - clusters:
              selector:
                matchExpressions:
                  - key: env
                    operator: In
                    values: [prod, staging]
  template:
    metadata:
      name: '{{.path.basename}}-{{.name}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: '{{.server}}'
        namespace: addons
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Apply it and list the result:

kubectl apply -f ~/lab/manifests/addons-matrix.yaml
kubectl -n argocd get applications

Six Applications, all Synced and Healthy 12 seconds after the apply:

NAME              SYNC STATUS   HEALTH STATUS
logging-prod      Synced        Healthy
logging-staging   Synced        Healthy
metrics-prod      Synced        Healthy
metrics-staging   Synced        Healthy
tracing-prod      Synced        Healthy
tracing-staging   Synced        Healthy

In the web UI, the Applications page shows the same six tiles, each pointing at its own add-on path and destination cluster:

Argo CD Applications tiles for the matrix generator: logging, metrics and tracing on prod and staging, all Healthy and Synced

Watch the multiplication. Matrix output grows as the product of its children, so ten add-ons on twenty clusters is two hundred Applications, and every one of them is reconciled by the same application controller. Check the count with argocd appset generate before you apply a matrix to a real fleet.

Override one cluster with the merge generator

Merge takes a base generator and lets later generators override parameters for entries that share a merge key. The usual job is “every cluster gets the default, prod gets something different”. It is also the generator where we hit the most surprises, so here is what worked and what didn’t. The version that works keeps the override as a plain top-level key:

vim ~/lab/manifests/webapp-merge-toplevel.yaml

The list element matches the prod cluster on server and adds tier: critical. Clusters without an override fall back to standard through get and default, which also keeps missingkey=error happy:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: webapp-merge-toplevel
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - merge:
        mergeKeys:
          - server
        generators:
          - clusters:
              selector:
                matchExpressions:
                  - key: env
                    operator: In
                    values: [prod, staging]
          - list:
              elements:
                - server: https://prod-control-plane:6443
                  tier: critical
  template:
    metadata:
      name: 'webapp-{{.name}}'
      labels:
        tier: '{{ get . "tier" | default "standard" }}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: 'apps/webapp/overlays/{{.metadata.labels.env}}'
      destination:
        server: '{{.server}}'
        namespace: webapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Apply it and check which tier label each Application received:

kubectl apply -f ~/lab/manifests/webapp-merge-toplevel.yaml
kubectl -n argocd get applications -o custom-columns=NAME:.metadata.name,TIER:.metadata.labels.tier,REV:.spec.source.targetRevision,PATH:.spec.source.path,SYNC:.status.sync.status,HEALTH:.status.health.status

Prod picked up the override and staging kept the default:

NAME             TIER       REV    PATH                           SYNC     HEALTH
webapp-prod      critical   main   apps/webapp/overlays/prod      Synced   Healthy
webapp-staging   standard   main   apps/webapp/overlays/staging   Synced   Healthy

Two variants we tried first did not work. Putting the cluster defaults under the cluster generator’s values: block and overriding them with flat values.tier: critical keys in the list (the form the official merge example uses) applied cleanly but changed nothing: both Applications came out tier=standard. Nesting a values: map inside the list element was worse, and the merge section of the troubleshooting list below shows the controller panic it caused.

Roll out in stages with RollingSync

Every generator so far syncs all clusters at once, so a bad commit reaches prod and staging together. RollingSync, part of the Progressive Syncs feature, changes that: it groups the generated Applications into ordered steps by label and only syncs the next step once the previous one is Healthy. The upstream Progressive Syncs documentation marks it as a Beta feature, and it is off by default. Turn it on in argocd-cmd-params-cm and restart the controller:

kubectl -n argocd patch cm argocd-cmd-params-cm --type merge -p '{"data":{"applicationsetcontroller.enable.progressive.syncs":"true"}}'
kubectl -n argocd rollout restart deploy argocd-applicationset-controller
kubectl -n argocd rollout status deploy argocd-applicationset-controller --timeout=120s

The ConfigMap key only takes effect through an environment variable the controller reads at startup, which is why the restart matters. Confirm the running pod picked it up:

kubectl -n argocd exec deploy/argocd-applicationset-controller -- env | grep PROGRESSIVE

Expect true:

ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS=true

Now define the steps. Each generated Application gets an envLabel label from the list element, and the two steps select staging first, then prod. Create the manifest:

vim ~/lab/manifests/webapp-rolling.yaml

We left automated in the template on purpose to see what RollingSync does with it:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: webapp-rolling
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - cluster: staging
            envLabel: staging
            url: https://staging-control-plane:6443
            path: apps/webapp/overlays/staging
          - cluster: prod
            envLabel: prod
            url: https://prod-control-plane:6443
            path: apps/webapp/overlays/prod
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: envLabel
              operator: In
              values:
                - staging
        - matchExpressions:
            - key: envLabel
              operator: In
              values:
                - prod
  template:
    metadata:
      name: 'webapp-{{.cluster}}'
      labels:
        envLabel: '{{.envLabel}}'
    spec:
      project: default
      source:
        repoURL: http://gitea:3000/josphat/fleet.git
        targetRevision: main
        path: '{{.path}}'
      destination:
        server: '{{.url}}'
        namespace: webapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

RollingSync overrode it. Both generated Applications came out with automated: map[enabled:false prune:true selfHeal:true], so the controller owns every sync from then on. The docs say the controller logs a warning when the template asks for autosync; we grepped the controller log and found none, so don’t count on a warning to tell you. The Application page shows it plainly instead:

Argo CD webapp-prod Application generated by a RollingSync ApplicationSet with Auto sync is not enabled, the Progressive Sync panel at step 2 Healthy and the owning ApplicationSet webapp-rolling marked
  1. 1Auto sync is not enabled
  2. 2Progressive Sync step
  3. 3Owning ApplicationSet

The Sync Status panel reads “Auto sync is not enabled” (1), the Progressive Sync step panel (2) shows which step this Application belongs to and whether it finished, and the Owning ApplicationSet badge in the resource tree (3) links back to webapp-rolling. That badge is the quickest way to find which ApplicationSet to edit when someone asks why a manual change to an Application keeps reverting.

A health gate only means something if health takes a moment to arrive, so before the timed run we added a readiness probe to the container in apps/webapp/base/deployment.yaml, right under its ports list, and pushed it:

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 15

To time a real rollout, we then pushed a commit at 09:28:04 that bumped the image from nginx:1.27-alpine to nginx:1.28-alpine, and polled the Applications every 2 seconds, logging each state change. These are the logged changes:

Time (UTC)staging (step 1)prod (step 2)Image on prod
09:28:05Healthy, SyncedHealthy, Syncednginx:1.27-alpine
09:29:27Healthy, SyncedWaiting, OutOfSyncnginx:1.27-alpine
09:29:58Progressing, sync runningWaiting, OutOfSyncnginx:1.27-alpine
09:30:44HealthyProgressing, Syncednginx:1.28-alpine
09:31:50HealthyHealthynginx:1.28-alpine

Read the table as the controller’s gate at work. Prod had gone OutOfSync by the 09:29:27 sample and then sat in Waiting while staging synced; the rollout moved to step 2 at 09:30:42, the moment staging reported Healthy. The whole rollout took 226 seconds from push to prod Healthy, and the 15 second readiness delay we had added to the Deployment is part of that, because the gate waits for health, not for the sync to finish. The ApplicationSet records each step itself:

RollingSync applicationStatus showing staging Healthy at step 1 and prod Healthy at step 2, with sync start times 45 seconds apart after the image bump commit

While a rollout runs, the RolloutProgressing condition reads “ApplicationSet is performing rollout of step 1”, then step 2. When it ends you get “ApplicationSet Rollout has completed”, the same text the ApplicationSet page shows under its health:

Argo CD ApplicationSet details tree for webapp-rolling showing ApplicationSet Rollout has completed and both Applications Healthy

Two more settings are worth knowing. maxUpdate on a step caps how many Applications in that step sync at once and defaults to 100%, which is fine for one cluster per step and too aggressive for a step of twenty. deletionOrder: Reverse under strategy deletes in the opposite order of the steps; we tested it on a copy of this ApplicationSet and the prod Application was removed at 09:33:36, staging at 09:33:37, with “completed reverse deletion for ApplicationSet webapp-rolling-reverse” in the log. If you want the same staged pattern inside a single cluster at the pod level, that’s what Argo Rollouts canary deployments are for; RollingSync stages clusters, not replicas.

Delete an ApplicationSet without deleting workloads

Set preserveResourcesOnDeletion: true before you need it. As the Application deletion docs explain, generated Applications get the resources finalizer unless that flag is set, and the finalizer is what tells Argo CD to delete everything the Application deployed. The flag sits under the ApplicationSet’s own syncPolicy, not the template’s:

spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]

We tested three ways out, starting each time from the same two-cluster list ApplicationSet:

  1. Flag added to a running ApplicationSet: finalizers removed from both Applications, workloads survived
  2. Flag set from the first apply: no finalizer ever added, workloads survived
  3. No flag, deleted with --cascade=orphan: Applications kept, finalizer kept

The first two behave the way you’d hope, so adding the flag to an existing ApplicationSet is safe and immediate. The third is the trap. --cascade=orphan looks like the safe delete, and at first it is: the Applications stay and the workloads keep running. But the orphaned Applications still carry the finalizer, so the next person who deletes one of them deletes its workload too:

kubectl delete appset with cascade orphan leaves Applications with no owner but still carrying the resources finalizer, so deleting webapp-staging removes its deployment

Deleting webapp-staging removed the staging Deployment, while prod kept running only because nobody had touched its Application yet. Re-applying the same ApplicationSet adopted the orphans again with the finalizer still in place, and it also recreated webapp-staging and redeployed it. So if you’ve orphaned Applications by accident, re-adopt them and add preserveResourcesOnDeletion: true before the next delete. We ran those two halves separately (the first test shows the flag dropping finalizers on a live ApplicationSet), not as one sequence, so check the finalizers with kubectl -n argocd get applications -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers before you delete anything.

Fix common ApplicationSet errors

Every error below came from a manifest we broke on purpose. In most cases the ApplicationSet showed the reason in its conditions and created nothing new (the panic case is the exception, with no conditions at all), so kubectl -n argocd get appset <name> -o jsonpath='{.status.conditions}' | jq . is the first command to run when nothing appears.

map has no entry for key

The template referenced {{.nonexistent}}, a key the list element doesn’t define. With missingkey=error the controller refuses to render, marks the ApplicationSet Degraded and reports reason RenderTemplateParamsError:

failed to execute go template webapp-{{.nonexistent}}: template: base:1:9: executing "base" at <.nonexistent>: map has no entry for key "nonexistent"

Fix the key name, or wrap optional keys as {{ get . "key" | default "x" }} like the merge example does. The corrected manifest generated webapp-prod and the conditions flipped back to “All applications have been generated successfully”.

function “path” not defined

This one comes from copying an old fasttemplate example into a goTemplate ApplicationSet. {{path.basename}} without the leading dot is fasttemplate syntax, and Go templates read path as a function call:

failed to parse template addons-{{path.basename}}: template: base:1: function "path" not defined

Add the dot and use the object form: {{.path.basename}} for the directory name and {{.path.path}} for the full path. Plain {{path}} also has to become {{.path.path}}, since .path on its own is now an object.

contains applications with duplicate name

A template with a fixed name (name: webapp) and two list elements renders two Applications with the same name. The controller logs “generated 2 applications”, then rejects the whole set with reason ApplicationValidationError and creates neither:

ApplicationSet dup-names contains applications with duplicate name: webapp

Put a per-target parameter in the name, such as webapp-{{.cluster}}. With a matrix, include a parameter from each child generator, or two combinations will collide.

is already owned by another ApplicationSet controller

Two different ApplicationSets that render the same Application name don’t merge or overwrite each other. The first one keeps it and the second one goes into error with UpdateApplicationError:

argocd appset get dup-b showing the error Object argocd/webapp-prod is already owned by another ApplicationSet controller dup-a

When we deleted dup-a, dup-b adopted webapp-prod on its next reconcile. That handover can surprise you during a migration from one ApplicationSet to another: the new one silently takes over, with its own sync policy, as soon as the old one is gone. Give each ApplicationSet a distinct name prefix and the error never happens.

Recovered from panic: reflect.Value.SetMapIndex

This came from the merge generator with a nested values: map inside a list element. The ApplicationSet showed an empty health status and generated nothing, while the controller log filled with the same panic, 1,634 times by the time we counted:

kubectl -n argocd logs deploy/argocd-applicationset-controller | grep webapp-merge-nested | grep -o '"msg":"Recovered from panic: [^\\]*' | sort | uniq -c

A count that high means the controller is retrying in a tight loop:

   1634 "msg":"Recovered from panic: reflect.Value.SetMapIndex: value of type interface {} is not assignable to type string

Delete that ApplicationSet straight away, because the loop burns controller CPU for every other ApplicationSet too, and move the override to a top-level key as shown in the merge section.

lookup on 127.0.0.53:53: server misbehaving

This is the argocd cluster add failure from the registration step. The machine running the CLI can’t resolve the API server name in the kubeconfig. Make the name resolvable there, or regenerate the kubeconfig with an address both the CLI and the hub can reach, because the URL you register is the one Argo CD uses for every sync afterwards.

Harden it before the fleet grows

Put these six guardrails in place before a second team writes an ApplicationSet. Everything above ran as admin in the default project, which is fine for a lab and wrong for a shared hub:

  • goTemplate: true with missingkey=error on every ApplicationSet
  • preserveResourcesOnDeletion: true on anything serving traffic
  • A selector on every cluster generator, so in-cluster is never a target by accident
  • A webhook to the ApplicationSet controller, not argocd-server, for Git generators
  • One AppProject per team, with allowed destinations limited to their clusters
  • RollingSync with maxUpdate set on any step that spans many clusters

Once several generators are live, the ApplicationSets page is the first sanity check after any generator change. Each tile shows the health and the number of Applications it owns:

Argo CD ApplicationSets page listing addons-matrix with 6 applications and webapp-rolling with 2, both Healthy

The AppProject point matters most, because an ApplicationSet can render any destination its template allows, and the project is the only thing stopping a typo from targeting the wrong cluster. The Argo CD CLI reference covers argocd proj for setting that up, and if you’re still deciding whether Argo CD is the right hub at all, our Flux vs Argo CD multi-cluster comparison runs the same kind of fleet on both. Our verdict for most teams: start with the cluster generator plus labels, add matrix for add-ons, and switch to RollingSync the day prod and staging stop being allowed to change together.

Keep reading

Best UI Applications for Managing Docker Containers Containers Best UI Applications for Managing Docker Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Containers Install Docker and Run Containers on Ubuntu 24.04|22.04 Install UniFi OS Server on Ubuntu 24.04 LTS Containers Install UniFi OS Server on Ubuntu 24.04 LTS OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Cloud OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Kiri Container Runtime on Linux: Install and Benchmark vs Docker Containers Kiri Container Runtime on Linux: Install and Benchmark vs Docker Install Flux CD on Kubernetes (Bootstrap with GitHub) Containers Install Flux CD on Kubernetes (Bootstrap with GitHub)

Leave a Comment

Press ESC to close